A side effect of the last change was that tokens for both codes for operators, functions and commands and codes with operands (constants, variables, arrays, defined functions and user functions) were using the same token constructor. This token constructor searched through alternate codes for the code with the appropriate return data type.
This was unnecessary for operator, function and command codes. The table new token function called for these codes passed in the return data type from the table entry of the code. The token constructor then called the new table set token code function. Since the data type matched the return data type (which was just passed in), no alternates were checked and the code, type and data type of the token was set. This was extra unnecessary work.
A new token constructor was added for operator, function and command codes, which only required arguments for the code, column, length and string. The string argument is only used for the REM and REM operator codes. This constructor replaces the table new token function. This constructor calls the table set code function which just sets the code, type and data type of the token from the table entry of the code. For consistency the code argument was put first in the other token constructor for codes with operands.
While looking at the creation of tokens, I decided that using the standard unique pointer within the parser was unnecessary. The parser can just allocate a token and return its pointer. The translator then can put the allocated tokens into a standard shared pointer. The parser was changed to use plain token pointers. The translator routines were changed to use the new token constructor directly via the standard make shared function. The translator get operand was changed to use the reset function to set the token member since shared pointers cannot be assigned directly to a pointer.
[branch table commit 1bfb76ae0a]
Friday, January 2, 2015
Parser – Codes With Operands
The last token type not being set fully in the parser were codes with operands (constants, variables, arrays, defined functions and user functions). Constant tokens were corrected with the last change. Arrays, defined functions and user functions are not fully implemented and so did not need to be changed. Variables however, were only partially set in the parser (only to the base Variable or Variable Reference code) and weren't set for the data type of the variable until the translator.
The parser get identifier function was modified to set the data type to Double if the word obtained from the input does not have a type. This applies to all identifiers not found in the table. The token constructor for codes is used for commands, operators, functions and codes with operands. The type argument was unnecessary since that is set from the table entry. However, an issue was found with how codes were found in the table.
For operators and functions, the [return] data type of the token is set from the table entry. (This issue doesn't affect commands since command don't have a return data type.) For codes with operands, the data type of the identifier is used to find the appropriate table entry (for example, Variable, Variable Integer, or Variable String) by looking at the data types of alternate codes. The current table set token code function did not work correctly because it searches alternate codes by operand data type. For this instance, the alternate codes need to be search by return data type.
A new set token code function was added without an operand index argument to search by return data type. If the data type (of the identifier) does match the code passed, then the alternate codes are searched for a matching return data type. If there are no alternates or none were found, then the code passed is set in the token along with the token type of the code. The data type is set to the data type of the identifier and not from the table entry (which may not match for codes like arrays that are not fully implemented yet).
The type argument was removed from the token constructor for codes. The type from the table entry of the code was passed (and the new set token code now does this). A call to the new set token code was added to the body of the constructor (previously empty).
Since codes for constants, variable, and variable references were found in the table incorrectly by operand data type, these table entries contained operand data types so that it would work. These codes do not have operands (in the sense that operands and functions do within expressions; not to be confused that in the program, these codes do have an operand index). These table entries were corrected with expression info instances containing no operands.
The translator get operand previously set the default data type of the token just obtained (set to Double if None and not a function). This was removed since the parser now does this. The token set default data type function called to do this was removed. The call to set the code for a no parentheses (variable) token was also no longer needed. With the parser now setting the default data type to Double, the expected results to the parser tests (#2, #3 and #5) needed to be updated.
[branch table commit acc37f0650]
The parser get identifier function was modified to set the data type to Double if the word obtained from the input does not have a type. This applies to all identifiers not found in the table. The token constructor for codes is used for commands, operators, functions and codes with operands. The type argument was unnecessary since that is set from the table entry. However, an issue was found with how codes were found in the table.
For operators and functions, the [return] data type of the token is set from the table entry. (This issue doesn't affect commands since command don't have a return data type.) For codes with operands, the data type of the identifier is used to find the appropriate table entry (for example, Variable, Variable Integer, or Variable String) by looking at the data types of alternate codes. The current table set token code function did not work correctly because it searches alternate codes by operand data type. For this instance, the alternate codes need to be search by return data type.
A new set token code function was added without an operand index argument to search by return data type. If the data type (of the identifier) does match the code passed, then the alternate codes are searched for a matching return data type. If there are no alternates or none were found, then the code passed is set in the token along with the token type of the code. The data type is set to the data type of the identifier and not from the table entry (which may not match for codes like arrays that are not fully implemented yet).
The type argument was removed from the token constructor for codes. The type from the table entry of the code was passed (and the new set token code now does this). A call to the new set token code was added to the body of the constructor (previously empty).
Since codes for constants, variable, and variable references were found in the table incorrectly by operand data type, these table entries contained operand data types so that it would work. These codes do not have operands (in the sense that operands and functions do within expressions; not to be confused that in the program, these codes do have an operand index). These table entries were corrected with expression info instances containing no operands.
The translator get operand previously set the default data type of the token just obtained (set to Double if None and not a function). This was removed since the parser now does this. The token set default data type function called to do this was removed. The call to set the code for a no parentheses (variable) token was also no longer needed. With the parser now setting the default data type to Double, the expected results to the parser tests (#2, #3 and #5) needed to be updated.
[branch table commit acc37f0650]
Wednesday, December 31, 2014
Constant Token Codes
The code and token type enumerations will be combined into a single code type enumeration. Before proceeding, the parser needs to return all tokens assigned to a code. This has been mostly accomplished, though one exception is constant tokens, which were still being assigned codes in the translator. This is complicated because the type of numerical constants may not be known when the constant is parsed. Consider these statements:
Instead of passing whether a number is allowed flag to the parser, the requested data type is now passed. If the data type is integer or double, the code of a constant token is fixed (the Integer Constant sub-code is not needed and is cleared if set). For other requested data types, either the Constant or Constant Integer code is set is described above with the Integer Constant sub-code set for small doubles. The parser also now sets the Constant String code for string constants. The parser makes no attempt to report any errors for data type mismatches.
The decimal flag argument of the token constructor for double constants was removed as the data type is set to Double and the Integer Constant sub-code is set if the value is within the integer range. The convert code was cleaned up by making the desired data type the primary switch and there was no need for secondary switches on the token data type since only one of two data types need to be checked for each desired data type. A convert constant helper function was added to handle changing constant token codes.
The table find code function was simplified due to the change on how constants are represented. The first argument of the set token and set token code functions were changed from a standard shared token pointer reference to a straight token pointer so that they can be called from the parser (with a standard unique pointer), translator (with a standard shared pointer) or token member function (with just a pointer). This simply required calling the get access function of the unique or shared pointers.
The translator get operand function no longer sets the code for constants. The get expression and process internal function functions no longer need to look for and set the codes for constants (the later needs to clear the Integer Constant sub-code for functions taking both number argument types, specifically ABS, SGN and STR$). And the get token function now only needs to pass the data type to the parser. The token convert and table find code functions are used by the translator and will finalize constants not set by the parser once the final data type is known.
[branch table commit 3099f8850f]
A = B + 5For numerical constants, both the integer and double representations of the constant is stored in the constant dictionary except for the case where a double constant does not fit into a 32-bit signed integer. Optimally the representation required is used without a hidden conversion code to unnecessarily convert the constant. For the first statement above, the double value of the constant is used. With the other two statements, the integer value of the constant is used. Number constant tokens have three states:
A% = B% + 5
A% = B% + 5.4
- An integer (no decimal point or exponent; fits into 32 bits)
- A small double (has a decimal point or an exponent; fits into 32 bits when converted)
- A large double (does not fit into 32 bits; cannot be converted)
Instead of passing whether a number is allowed flag to the parser, the requested data type is now passed. If the data type is integer or double, the code of a constant token is fixed (the Integer Constant sub-code is not needed and is cleared if set). For other requested data types, either the Constant or Constant Integer code is set is described above with the Integer Constant sub-code set for small doubles. The parser also now sets the Constant String code for string constants. The parser makes no attempt to report any errors for data type mismatches.
The decimal flag argument of the token constructor for double constants was removed as the data type is set to Double and the Integer Constant sub-code is set if the value is within the integer range. The convert code was cleaned up by making the desired data type the primary switch and there was no need for secondary switches on the token data type since only one of two data types need to be checked for each desired data type. A convert constant helper function was added to handle changing constant token codes.
The table find code function was simplified due to the change on how constants are represented. The first argument of the set token and set token code functions were changed from a standard shared token pointer reference to a straight token pointer so that they can be called from the parser (with a standard unique pointer), translator (with a standard shared pointer) or token member function (with just a pointer). This simply required calling the get access function of the unique or shared pointers.
The translator get operand function no longer sets the code for constants. The get expression and process internal function functions no longer need to look for and set the codes for constants (the later needs to clear the Integer Constant sub-code for functions taking both number argument types, specifically ABS, SGN and STR$). And the get token function now only needs to pass the data type to the parser. The token convert and table find code functions are used by the translator and will finalize constants not set by the parser once the final data type is known.
[branch table commit 3099f8850f]
Saturday, December 27, 2014
Table – More Alternate Codes
Since it will be ideal to reduce the number of unique code type enumerators needed, several uses of the current code enumerators was eliminated. Most of these were accomplished by assigning the codes needed as alternate codes to commands that need them in the Alternate Info initializer list and obtaining the codes using the alternate map:
The Input code enumerator was used in the common INPUT translate routine to determine if the routine was called for the INPUT or INPUT PROMPT command. This test was changed to checking if the second name of the command token code is empty, which is empty for INPUT and not empty for INPUT PROMPT.
The Constant String code enumerator was used in the token equality operator function to determine if token string comparison should be case sensitive (for REM, REM operator and string constants) or case insensitive (variables, arrays, etc.). This test was changed to checking if the token type is a constant and the data type is a string.
The Print code enumerator was used in the print recreate function instead of using the name from the table entry. This was done because the print semicolon recreate function called this function after appending a semicolon. If the table entry was used, a semicolon would have been output for the command instead of PRINT. The function was changed to create a temporary RPN item with a temporary token with the Print code, but not the code enumerator using the alternate code. This set of statements is convoluted, but will be far simpler once the new table model is implemented.
Unrelated to removing the use of code enumerators, it was noticed that the print comma recreate and print semicolon recreate functions were using the strings for a comma and semicolon directly but technically should have been using the name from the table entry. Both of these were changed to use the table entry name.
[branch table commit b6850c7c7f]
Assign → Let (alternate 0)Since the command code is available in the translate function needing them (LET, INPUT, or PRINT) , it can be used to find the alternate code needed. The Print code is assigned as an alternate to the Semicolon code for recreation (see below). For a PRINT statement ending with a semicolon, no PRINT command code is stored in the program code so the semicolon recreate needs to also recreate the PRINT keyword.
Input Begin → Input (alternate 0)
Input Begin String → Input Prompt (alternate 0)
Input Assign → Input (alternate 1)
Input Assign → Input Prompt (alternate 1)
Print Double → Print (alternate 0)
Print → Semicolon (alternate 0)
Variable Reference → Variable (alternate 1)
The Input code enumerator was used in the common INPUT translate routine to determine if the routine was called for the INPUT or INPUT PROMPT command. This test was changed to checking if the second name of the command token code is empty, which is empty for INPUT and not empty for INPUT PROMPT.
The Constant String code enumerator was used in the token equality operator function to determine if token string comparison should be case sensitive (for REM, REM operator and string constants) or case insensitive (variables, arrays, etc.). This test was changed to checking if the token type is a constant and the data type is a string.
The Print code enumerator was used in the print recreate function instead of using the name from the table entry. This was done because the print semicolon recreate function called this function after appending a semicolon. If the table entry was used, a semicolon would have been output for the command instead of PRINT. The function was changed to create a temporary RPN item with a temporary token with the Print code, but not the code enumerator using the alternate code. This set of statements is convoluted, but will be far simpler once the new table model is implemented.
Unrelated to removing the use of code enumerators, it was noticed that the print comma recreate and print semicolon recreate functions were using the strings for a comma and semicolon directly but technically should have been using the name from the table entry. Both of these were changed to use the table entry name.
[branch table commit b6850c7c7f]
Table – New Code Type Enumeration
With the current table model, each entry contains a token type that identifies the type of token that is created for the table entry. Each entry has a code enumerator, which is simply used as an index to the entry. The code (index) is put into the program code. The code enumeration was originally automatically generated from comments next to the table entries. This avoided mismatches between the code enumerators and the table entries (though this was a poor design choice). Only some entries were identified using these code enumerators.
For the new table model, each table entry will be a unique class with a unique instance. Each table entry instance will be assigned a unique index by the base table entry constructor. Some table entries will still need to be found be some means by the parser and translator. This could be done directly by referencing the table entry instance, but this would require exposing the derived table entry classes. The plan is only to expose the base table entry class definition.
To find these small number of table entries, an enumeration will still be used. This enumeration will be similar to the token type enumeration currently given to each table entry. Therefore a single Code Type enumeration will be defined and will also replace the token type enumeration. Unlike the current Code enumeration, table entries could have the same Code Type enumerator (for example, all of the six variable codes will be assigned the Variable code type). Only the first table entry assigned to a code type will be returned for a code type enumerator; the others will be assigned as alternate codes of the first.
Code types will generally not be assigned to table entries for commands, operators and functions (which will no longer be referred to as internal functions) except for a few cases (for example, the LET command and the equal operator). New table flags will be assigned instead. There is already a Command table flag, and there will be Operator and Function table flags. Each of the codes with operands (variables, arrays, constants, defined functions, user functions, and subroutines) will have a code type. There will not be separate enumerators for tokens with and without parentheses (more on this later).
For the new table model, each table entry will be a unique class with a unique instance. Each table entry instance will be assigned a unique index by the base table entry constructor. Some table entries will still need to be found be some means by the parser and translator. This could be done directly by referencing the table entry instance, but this would require exposing the derived table entry classes. The plan is only to expose the base table entry class definition.
To find these small number of table entries, an enumeration will still be used. This enumeration will be similar to the token type enumeration currently given to each table entry. Therefore a single Code Type enumeration will be defined and will also replace the token type enumeration. Unlike the current Code enumeration, table entries could have the same Code Type enumerator (for example, all of the six variable codes will be assigned the Variable code type). Only the first table entry assigned to a code type will be returned for a code type enumerator; the others will be assigned as alternate codes of the first.
Code types will generally not be assigned to table entries for commands, operators and functions (which will no longer be referred to as internal functions) except for a few cases (for example, the LET command and the equal operator). New table flags will be assigned instead. There is already a Command table flag, and there will be Operator and Function table flags. Each of the codes with operands (variables, arrays, constants, defined functions, user functions, and subroutines) will have a code type. There will not be separate enumerators for tokens with and without parentheses (more on this later).
Friday, December 26, 2014
Sub-Code Enumeration Refactoring
When the Double sub-code enumerator value was changed to support double identifiers with the double data type character (#), it was noticed that the Parentheses and Colon sub-codes would never be used with the same codes. The Parentheses sub-code will only be set for operators and operands (variable, constants, etc.), and the Colon sub-code will only be set for commands (with some exceptions).
The available sub-code bits are limited - there are only six. Several sub-codes were already combined with the Option sub-code ('LET' for assignment codes, 'Question' for the Input Begin String code, and 'Keep' for the INPUT and INPUT PROMPT command codes). The same thing was done for the Parentheses and Colon sub-codes, but instead of using the same sub-code enumerator for both (creating an appropriate name was impossible), both current enumerators were simply given the same value.
There was one problem with this scheme. Two codes that could have the Colon sub-code are the Comma and Semicolon codes, which are only present in a PRINT statement and will be commands at run-time. These codes are defined as operators. The assignment codes are defined as no types, but are commands and could have the Colon sub-code. To indicate that these are commands, a new Command table flag was added and their table entries were given this flag.
Therefore, a code has the Colon sub-code if it is a command type or has the Command table flag, otherwise the code has the Parentheses sub-code. Checks for the Command table flag were added to the recreator function operator, the stream insert operator for program words, and the test stream insert operator for tokens. A few other minor sub-code related changes were made, click Continue... for details.
The available sub-code bits are limited - there are only six. Several sub-codes were already combined with the Option sub-code ('LET' for assignment codes, 'Question' for the Input Begin String code, and 'Keep' for the INPUT and INPUT PROMPT command codes). The same thing was done for the Parentheses and Colon sub-codes, but instead of using the same sub-code enumerator for both (creating an appropriate name was impossible), both current enumerators were simply given the same value.
There was one problem with this scheme. Two codes that could have the Colon sub-code are the Comma and Semicolon codes, which are only present in a PRINT statement and will be commands at run-time. These codes are defined as operators. The assignment codes are defined as no types, but are commands and could have the Colon sub-code. To indicate that these are commands, a new Command table flag was added and their table entries were given this flag.
Therefore, a code has the Colon sub-code if it is a command type or has the Command table flag, otherwise the code has the Parentheses sub-code. Checks for the Command table flag were added to the recreator function operator, the stream insert operator for program words, and the test stream insert operator for tokens. A few other minor sub-code related changes were made, click Continue... for details.
Thursday, December 25, 2014
Double Identifier Problem
An existing problem was discovered when the parser was modified to not store the data type character of identifiers. The issue was with double identifiers when using the optional # data type character. The identifiers Variable and Variable# were incorrectly added to the dictionary as separate entries when they should have been the same entry.
The parser get identifier function was modified to not store the data type character in the token. This caused a problem when recreating double identifiers where the # character entered would disappear. Recreating all double identifiers with a # character was also not desirable. This was corrected by adding the Double sub-code to the token. A sub-code argument was added to the token constructor for identifiers. This sub-code is encoded into the program code so that the # character is recreated when it is entered.
The Double sub-code was only being used for constants. When the value of a constant is within the integer range, its data type is set to integer, and if a decimal point is present, the Double sub-code is set. The translator uses this sub-code to determine if a constant can be used as a double even though the data type is integer (see post from October 28 for details). This sub-code does not survive past the translator (not put into the program code).
A new string with data type access function was added to the token to add the data type character (#, % for integers, and $ for strings) to the token string returned. A # character is only added if the Double sub-code is set. This function replaced the string access function in the test token stream insert operator, tester print token, and several recreate functions.
A sub-code argument was added to the table entry operand text functions. The variable operand text functions were modified to add the data type character to the variable name. For double variables, the character is only added if the Double sub-code is set. An Ignore sub-code enumerator was added, and when passed to the operand text function, no data type character is added to the variable name. This option was needed for the program model decode function that uses the operand text function to set the string of the token (since tokens no longer store the data type character).
The value of the Double sub-code was changed so that its bit value was within the range of the sub-code bits (not necessary before since this sub-code was not used in the program code). The return type of program code instruction sub-code access function was changed to the Sub-Code enumeration type (from an integer). The expected encoder test results were updated, specifically the dictionaries output since the data type characters are no longer present in the entries.
[branch table commit e97057efca]
The parser get identifier function was modified to not store the data type character in the token. This caused a problem when recreating double identifiers where the # character entered would disappear. Recreating all double identifiers with a # character was also not desirable. This was corrected by adding the Double sub-code to the token. A sub-code argument was added to the token constructor for identifiers. This sub-code is encoded into the program code so that the # character is recreated when it is entered.
The Double sub-code was only being used for constants. When the value of a constant is within the integer range, its data type is set to integer, and if a decimal point is present, the Double sub-code is set. The translator uses this sub-code to determine if a constant can be used as a double even though the data type is integer (see post from October 28 for details). This sub-code does not survive past the translator (not put into the program code).
A new string with data type access function was added to the token to add the data type character (#, % for integers, and $ for strings) to the token string returned. A # character is only added if the Double sub-code is set. This function replaced the string access function in the test token stream insert operator, tester print token, and several recreate functions.
A sub-code argument was added to the table entry operand text functions. The variable operand text functions were modified to add the data type character to the variable name. For double variables, the character is only added if the Double sub-code is set. An Ignore sub-code enumerator was added, and when passed to the operand text function, no data type character is added to the variable name. This option was needed for the program model decode function that uses the operand text function to set the string of the token (since tokens no longer store the data type character).
The value of the Double sub-code was changed so that its bit value was within the range of the sub-code bits (not necessary before since this sub-code was not used in the program code). The return type of program code instruction sub-code access function was changed to the Sub-Code enumeration type (from an integer). The expected encoder test results were updated, specifically the dictionaries output since the data type characters are no longer present in the entries.
[branch table commit e97057efca]
Parser – Identifier Codes
The parser previously set the code for an identifier token only when the word was found in the table (command, operator or function). The codes for other identifiers were set in the translator: defined functions with no parentheses and variables (get operand); arrays, functions, and defined functions with parentheses (process parentheses tokens). This was changed to set all codes in the parser.
To do this in the parser, the parser needed to know if a reference operand was being requested. For now identifiers with no parentheses are set to variables, and with parentheses are set to arrays unless they start with an F (temporary check for testing). Defined functions are identifiers that start with an FN. Eventually the parser will need access to the program dictionaries to fully determine which code to assign to an identifier token.
The get identifier function was modified to set the code as described above for identifiers not found in the table. A reference argument was added, which was also added to the parser function operator. (The Reference enumeration was moved from the translator class to the main header file so that its enumerators are accessible.) The token constructor for codes and identifiers were combined to a single constructor with default arguments for the string and reference members.
For variables, the reference argument is used to determine if the code is a variable or a variable reference. Only the base code is set as the translator changed the code for the data type. In the case of a variable reference, the reference member of the token is not set (the translator did not previously set it either).
Several token type cases in the translator get operand function was modified. For defined functions with no parentheses, the token reference and code members no longer need to be set. For no parentheses tokens (variables), the code is still updated for the data type. The parser will do this once the new table model is implemented. For parentheses tokens (arrays), the token reference member no longer needs to be set.
The translator process parentheses token function no longer does the check for functions (temporarily identifiers starting with F), or set the code of the token. For determining an array (to set the expected expression types to integer for the subscripts), the Array code is checked for. This check will need to be modified when arrays are implemented since there will be different array codes for each data type, which will be set by the parser.
[branch table commit 69dff18e26]
To do this in the parser, the parser needed to know if a reference operand was being requested. For now identifiers with no parentheses are set to variables, and with parentheses are set to arrays unless they start with an F (temporary check for testing). Defined functions are identifiers that start with an FN. Eventually the parser will need access to the program dictionaries to fully determine which code to assign to an identifier token.
The get identifier function was modified to set the code as described above for identifiers not found in the table. A reference argument was added, which was also added to the parser function operator. (The Reference enumeration was moved from the translator class to the main header file so that its enumerators are accessible.) The token constructor for codes and identifiers were combined to a single constructor with default arguments for the string and reference members.
For variables, the reference argument is used to determine if the code is a variable or a variable reference. Only the base code is set as the translator changed the code for the data type. In the case of a variable reference, the reference member of the token is not set (the translator did not previously set it either).
Several token type cases in the translator get operand function was modified. For defined functions with no parentheses, the token reference and code members no longer need to be set. For no parentheses tokens (variables), the code is still updated for the data type. The parser will do this once the new table model is implemented. For parentheses tokens (arrays), the token reference member no longer needs to be set.
The translator process parentheses token function no longer does the check for functions (temporarily identifiers starting with F), or set the code of the token. For determining an array (to set the expected expression types to integer for the subscripts), the Array code is checked for. This check will need to be modified when arrays are implemented since there will be different array codes for each data type, which will be set by the parser.
[branch table commit 69dff18e26]
Sunday, December 21, 2014
Table – Entry Pointers
The new table implementation will have a single Table class that will represent a table entry for a single code and will serve as the base class to all of the derived table entry classes. Global table information (name to entry map, alternate code map, etc.) will be static members of this class. This is equivalent to the current since table instance.
Currently the Table class and Table Entry structure are separate definitions, but eventually the table entry members will be members of the new base table class. The current table class has many access functions where their first argument is a code enumerator (which is currently used as an index). There are also many access functions that have a token pointer as their first argument, but this is mostly used to get the code from the token.
The new table model will access the table entries by a pointer to the entry instead of be a code enumerator used as an index. The code argument access functions will become simply access functions to the table entry. The next step in this transition to the new table model will be to use pointers to table entries instead of code enumerators. The token code member will become a table entry pointer.
To start this transition, the Table Entry structure was moved from the table source file to the table header file. The members were also renamed with the member (m_) prefix except for the function pointer members (which will be replaced with virtual functions in the new table model).
[branch table commit 94decad26c]
Currently the Table class and Table Entry structure are separate definitions, but eventually the table entry members will be members of the new base table class. The current table class has many access functions where their first argument is a code enumerator (which is currently used as an index). There are also many access functions that have a token pointer as their first argument, but this is mostly used to get the code from the token.
The new table model will access the table entries by a pointer to the entry instead of be a code enumerator used as an index. The code argument access functions will become simply access functions to the table entry. The next step in this transition to the new table model will be to use pointers to table entries instead of code enumerators. The token code member will become a table entry pointer.
To start this transition, the Table Entry structure was moved from the table source file to the table header file. The members were also renamed with the member (m_) prefix except for the function pointer members (which will be replaced with virtual functions in the new table model).
[branch table commit 94decad26c]
Table – Internal Code Token Types
There were several internal code table entries (null, assignment, print item, input assign and input parse) that were assigned to either an Operator or Internal Function token type. These internal codes do not require a token type because they are not produced by the parser (the token type is only used for tokens from the parser). These table entries were changed to the default token type (in other words, no type).
The reason for this change will become more evident when the new table class hierarchy is implemented. One of the goals of which is to reduce the amount of unnecessary initialization values. It may even turn out that the token type member of all table entries will be unnecessary, but this is not clear yet.
These changes did cause a minor issue in test output. By default the test output stream insert operator for a token outputs nothing for a token without a type, which cause the above modified codes to not produce their debug name. This function was modified to output the debug name in the default case instead of doing nothing. Since the internal function types also only output the debug name, these cases were removed to let the default case handle these types.
[branch table commit 232a97f6d6]
The reason for this change will become more evident when the new table class hierarchy is implemented. One of the goals of which is to reduce the amount of unnecessary initialization values. It may even turn out that the token type member of all table entries will be unnecessary, but this is not clear yet.
These changes did cause a minor issue in test output. By default the test output stream insert operator for a token outputs nothing for a token without a type, which cause the above modified codes to not produce their debug name. This function was modified to output the debug name in the default case instead of doing nothing. Since the internal function types also only output the debug name, these cases were removed to let the default case handle these types.
[branch table commit 232a97f6d6]
Table – New Token Consolidation
The next major change to the table class will be to start transitioning to using table entry pointers instead of code enumerators and to remove the use of code enumerators as indexes. The code and index values will be separate members of table entries. All table entries will have an index (which is put into the program code), but only a few table entries will have a code (only those the require specific lookup like some of the special symbols including comma, parentheses, colon, etc.). Before proceeding with this, a small simplification was made first.
There were two new token functions in the table class, one taking a single code argument and the other taking column, length and code arguments. The single argument version relied on the default token constructor. This was one of three uses of the default token constructor. The default token constructor contained optional column and length arguments (default to -1 indicating unset), but there were no callers of the default token constructor that used these arguments.
The single code argument new token function was removed along with the default token constructor. The code argument was made the first argument of the other version of the new token function with default -1 values provided for the optional column and length arguments. This second version does not use the default token constructor. All callers of this function were in the parser and were modified for the reordering of the arguments.
The second use of the default token constructor was by the decode function in the program model class to create a default token, which it then used the token set code access function to set the code. This function was changed to use the new token function.
The third use of the default token constructor was in the INPUT translate function where a new token is needed for an input assign code and another token (comma or semicolon) is not available for reuse. This was changed to use the new token function with a Null code.
[branch table commit b5dd96c272]
There were two new token functions in the table class, one taking a single code argument and the other taking column, length and code arguments. The single argument version relied on the default token constructor. This was one of three uses of the default token constructor. The default token constructor contained optional column and length arguments (default to -1 indicating unset), but there were no callers of the default token constructor that used these arguments.
The single code argument new token function was removed along with the default token constructor. The code argument was made the first argument of the other version of the new token function with default -1 values provided for the optional column and length arguments. This second version does not use the default token constructor. All callers of this function were in the parser and were modified for the reordering of the arguments.
The second use of the default token constructor was by the decode function in the program model class to create a default token, which it then used the token set code access function to set the code. This function was changed to use the new token function.
The third use of the default token constructor was in the INPUT translate function where a new token is needed for an input assign code and another token (comma or semicolon) is not available for reuse. This was changed to use the new token function with a Null code.
[branch table commit b5dd96c272]
Saturday, December 20, 2014
Table – Operand Arrays
There was a macro used for generating two arguments to the Expression Info constructor, which took an argument identifying the array less its suffix. This macro was removed (the last such macro) and the predefined operand data type arrays were replaced with standard initializer lists. With an initializer list, the size of the list is available.
The operand count and operand data type array pointer arguments of the Expression Info constructor were replaced with a standard initializer list of data types (with a default of a blank list). The operand count member is initialized to the size of the operands list. A standard initializer list is implemented as an array internally. The begin access function is used to access the beginning of the initializer list to initialize the operand data type array. The arguments of the expression info instances were modified to the initializer lists.
Another set of related changes were also made. A null expression info instance was added (with no return value or operands). The table entries that previously had their expression info pointer member initialized to a null pointer were changed to point to this null expression info instance. This allowed the removal of the check for a null expression info pointer member in several of the access functions and from the add function.
[branch table commit fe5801c227]
The operand count and operand data type array pointer arguments of the Expression Info constructor were replaced with a standard initializer list of data types (with a default of a blank list). The operand count member is initialized to the size of the operands list. A standard initializer list is implemented as an array internally. The begin access function is used to access the beginning of the initializer list to initialize the operand data type array. The arguments of the expression info instances were modified to the initializer lists.
Another set of related changes were also made. A null expression info instance was added (with no return value or operands). The table entries that previously had their expression info pointer member initialized to a null pointer were changed to point to this null expression info instance. This allowed the removal of the check for a null expression info pointer member in several of the access functions and from the add function.
[branch table commit fe5801c227]
Table – Associated Code Removal
Now that use of the associated code variables and access functions has been replaced with the alternate code map, the associated code members could be removed from the Expression Info structure and their access functions removed from the table class. The predefined associated code arrays and the associated code macros were also removed.
Table entries that created their own expression info instance using one of the associated code macros were replaced with the appropriate pointer to a predefined expression info instance. A few additional predefined expression info instances were needed.
[branch table commit 18cb29c185]
Table entries that created their own expression info instance using one of the associated code macros were replaced with the appropriate pointer to a predefined expression info instance. A few additional predefined expression info instances were needed.
[branch table commit 18cb29c185]
Friday, December 19, 2014
Table – Expected Data Type
The expected data type table entry member was recently moved from the Expression Info structure (because codes using the same return and operand data types could have different expected data types). There were several issues with the expected data types initialization implementation (which were initialized automatically to prevent programming mistakes):
The expected data type access function was modified to use the new map. The expected data type member was removed from the Table Entry structure, and its initializer values were removed from the table entries. The separate iteration loop in the table constructor to initialize the expected data types was removed.
A problem was found in the set token code function (used to set the code in a token, possibly an alternate code, depending on its data type) where it could incorrectly add a new blank element to the alternate code map. This did not appear to cause a problem, but was corrected by checking if the code is present before iterating over alternate codes. This issue was that for a standard map, the bracket operator adds blank elements if the key does not exist.
[branch table commit 1374657d7e]
- Every table entry had an expected data type even it is was never used (for example, commands). Even when it was in the Expression Info structure, it was not used for many codes (no argument functions, assignment codes, etc.).
- A separate iteration loop was needed to initialize the expected data types since the alternate code information was used, the alternate code map needed to be initialized first.
- The expected data type initialization took into account if a code had the possibility of having all three data types (Double, Integer, and String) where the expected data type would be set to Any even though there were actually such codes.
- A new entry is added to the name to entry map (a new primary code).
- An entry is replaced in the name to entry map (a new primary when the operand count is less than the current primary; and in this case the old entry is removed for an internal function).
- A new secondary primary is added to the alternate code map (a binary operator to a unary operator).
- A replacement alternate primary is found (one that has the same operands, see last post; the old alternate primary entry is removed).
- A new alternate primary is found (the alternate is added for the first operand of a binary operator, otherwise the current entry of the primary is modified).
The expected data type access function was modified to use the new map. The expected data type member was removed from the Table Entry structure, and its initializer values were removed from the table entries. The separate iteration loop in the table constructor to initialize the expected data types was removed.
A problem was found in the set token code function (used to set the code in a token, possibly an alternate code, depending on its data type) where it could incorrectly add a new blank element to the alternate code map. This did not appear to cause a problem, but was corrected by checking if the code is present before iterating over alternate codes. This issue was that for a standard map, the bracket operator adds blank elements if the key does not exist.
[branch table commit 1374657d7e]
Thursday, December 18, 2014
Table – Other Alternate Codes
There were many alternate codes that couldn't be initialized automatically like with operators and internal functions. These codes include the assignment, sub-string assignment, internal command (for INPUT and PRINT) and codes with operators. For now these other alternate codes need to be initialized manually.
An Alternate Information structure was added containing the primary code, the array index for the alternate codes and a initializer list of alternate codes. An initializer list of these structures was added containing the information for all of these other alternate codes. After iterating through the list of entries, the constructor iterates through this initializer list to add these other alternate codes to the alternate code map.
The rest of the uses of the associated code arrays were changed to use the alternate code map including the set token code function (used to set the code in a token, possibly an alternate code, depending on its data type), the LET translate function (for setting a sub-string assignment, string keep assignment and list assignment codes), and the INPUT translate function (for setting an input parse code).
The check in the constructor for validating the second associated code index was removed. The section for setting the expected data type of an operator or internal function was modified to use the alternate code map. This required a separate entry iteration loop since the alternate code map needs to be initialized completely before looking at the alternate codes.
There was problem with the automatic alternate code map initialization because of the current order of entries. The issue was that the binary operators with two integer operands was being made an alternate code of the operator with first integer and second double operand. This was different than how the associated code arrays were initialized. Instead of moving all of these entries (and their enumerators), a check was added to the alternate map initialization to check for with situation and to swap the entries.
Two additional checks were added to the alternate map initialization to make sure the primary binary operator code has operands with the same data type, throwing an error if not. In the check if a multiple internal function code with the greater number of operands is before the code with less operands had to also set the Multiple flag. The code with less operands was already being made the primary code.
[branch table commit d38f79b8a1]
An Alternate Information structure was added containing the primary code, the array index for the alternate codes and a initializer list of alternate codes. An initializer list of these structures was added containing the information for all of these other alternate codes. After iterating through the list of entries, the constructor iterates through this initializer list to add these other alternate codes to the alternate code map.
The rest of the uses of the associated code arrays were changed to use the alternate code map including the set token code function (used to set the code in a token, possibly an alternate code, depending on its data type), the LET translate function (for setting a sub-string assignment, string keep assignment and list assignment codes), and the INPUT translate function (for setting an input parse code).
The check in the constructor for validating the second associated code index was removed. The section for setting the expected data type of an operator or internal function was modified to use the alternate code map. This required a separate entry iteration loop since the alternate code map needs to be initialized completely before looking at the alternate codes.
There was problem with the automatic alternate code map initialization because of the current order of entries. The issue was that the binary operators with two integer operands was being made an alternate code of the operator with first integer and second double operand. This was different than how the associated code arrays were initialized. Instead of moving all of these entries (and their enumerators), a check was added to the alternate map initialization to check for with situation and to swap the entries.
Two additional checks were added to the alternate map initialization to make sure the primary binary operator code has operands with the same data type, throwing an error if not. In the check if a multiple internal function code with the greater number of operands is before the code with less operands had to also set the Multiple flag. The code with less operands was already being made the primary code.
[branch table commit d38f79b8a1]
Sunday, December 14, 2014
Table Alternate Codes – Operators/Functions (Use)
With the new alternate codes map implemented and partially filled with all the alternate codes for operators and internal functions, the translator was modified to start using this map instead of the associated codes array.
Two access functions were added to the table class, which included the alternate code and alternate code count functions. Both take code enumerator and operand index arguments. These functions have temporary implementations. When the new table model is fully implemented, these functions won't need the code argument as the this pointer will be used as the key to the map. They will also return a entry pointer instead of a code enumerator.
The binary operator check for a unary operator in the translator get expression routine was modified to use the new access functions. In the process internal function routine, the new access function is used to get the alternate code for a function for an operand of a different data type as the primary function code, and when an extra argument is found for a function with multiple forms.
There was an issue with the subtract code table entries. The current associated code arrays are still being used to process operands of operators because the find code routine is still being used to get associated codes for codes that don't have entries in the new alternate map yet, so couldn't be modified to use the new access functions.
The problem was caused by the change to make the main binary subtract code (two double operands) the second associated code of the negate code, which was made the primary code for minus operator. This subtract code was moved to after the subtract code with the first integer operand. Since this code was first, it was made the main binary code and the primary binary alternate to the negate code. This code did not have the correct associated codes on the current associated code array, so hidden conversion operators were incorrectly added to the output list.
This order of the alternates in the table does not matter with the alternate generation, but for the moment, the new alternate map and the associated code arrays need to agree. This problem was corrected by moving the main subtract code to before the subtract with first integer operand. Since the code enumeration is not automatic, the subtract enumerator also had to be moved to match the table. This is a temporary situation.
[branch table commit 01db8002ba]
Two access functions were added to the table class, which included the alternate code and alternate code count functions. Both take code enumerator and operand index arguments. These functions have temporary implementations. When the new table model is fully implemented, these functions won't need the code argument as the this pointer will be used as the key to the map. They will also return a entry pointer instead of a code enumerator.
The binary operator check for a unary operator in the translator get expression routine was modified to use the new access functions. In the process internal function routine, the new access function is used to get the alternate code for a function for an operand of a different data type as the primary function code, and when an extra argument is found for a function with multiple forms.
There was an issue with the subtract code table entries. The current associated code arrays are still being used to process operands of operators because the find code routine is still being used to get associated codes for codes that don't have entries in the new alternate map yet, so couldn't be modified to use the new access functions.
The problem was caused by the change to make the main binary subtract code (two double operands) the second associated code of the negate code, which was made the primary code for minus operator. This subtract code was moved to after the subtract code with the first integer operand. Since this code was first, it was made the main binary code and the primary binary alternate to the negate code. This code did not have the correct associated codes on the current associated code array, so hidden conversion operators were incorrectly added to the output list.
This order of the alternates in the table does not matter with the alternate generation, but for the moment, the new alternate map and the associated code arrays need to agree. This problem was corrected by moving the main subtract code to before the subtract with first integer operand. Since the code enumeration is not automatic, the subtract enumerator also had to be moved to match the table. This is a temporary situation.
[branch table commit 01db8002ba]
Subscribe to:
Posts (Atom)