The token constructors each contain calls to the table static instance function so that the set token or set token code function can be called. These two table functions set the code, type and data type of the token. Now that tokens always contain a code (and later a table entry pointer), the type and data type members will be the same as the table entry (with one exception for data type). So it is not necessary for the token to contain these members.
The type member was removed from the token class (the data type member will have to wait until that one exception is eliminated). The type and is type access functions were modified to read from the table using a new table access function (which for now uses the code to get the table entry pointer). The set type access function was removed as it was only used when the token was created. A type access function was added to the table entry class.
These changes required some refactoring of header files. The token header file include statement in the table file needed to be removed since the token header file needs to access the table entry. The table header file include statement was removed from the token header file. This required the Type enumeration to be moved from the token class to the table header file. This enumeration was not renamed since it will soon be replaced with the new code type enumeration.
[branch table commit 68593c2dde]
Sunday, January 4, 2015
Parser – Table Instance Member
From the last change, most uses of the table instance reference member of the parser class was removed; replaced with use of static table functions. The last two uses were for setting of the code (plus type and data type) for constant tokens. For constant string tokens, the setting of the token code was moved to the token constructor for string constants.
The other use of the table instance was for number constants also in the main function operator function of the parser, but with some statements to determine from the desired data type what the data type of the number constant should be and whether to set the Integer Constant sub-code. These statements and the setting of the code of the token were moved to the token constructors for integer and double constants.
For integer constant tokens, the data type is set to integer unless the desired data type is double. There is no need to for the Integer Constant sub-code since the constant is an integer. For double constant tokens, if the value is within the integer range, the Integer Constant is only set if the desired data type is not an integer or double. If the desired data type is indeterminate (Number or Any) then the data type of the token is set to Double. For values outside the integer range, the data type is set to Double.
The table instance reference member was removed since it was no longer being used. The remaining parser constructor was only initializing the input string stream member, so it was moved to and made in-line in the header file.
[branch table commit b89a1b663b]
The other use of the table instance was for number constants also in the main function operator function of the parser, but with some statements to determine from the desired data type what the data type of the number constant should be and whether to set the Integer Constant sub-code. These statements and the setting of the code of the token were moved to the token constructors for integer and double constants.
For integer constant tokens, the data type is set to integer unless the desired data type is double. There is no need to for the Integer Constant sub-code since the constant is an integer. For double constant tokens, if the value is within the integer range, the Integer Constant is only set if the desired data type is not an integer or double. If the desired data type is indeterminate (Number or Any) then the data type of the token is set to Double. For values outside the integer range, the data type is set to Double.
The table instance reference member was removed since it was no longer being used. The remaining parser constructor was only initializing the input string stream member, so it was moved to and made in-line in the header file.
[branch table commit b89a1b663b]
Parser – Table Entry Pointers
Before the code enumeration (and token type enumeration) can be replaced with the new code type enumeration, uses of the code enumeration type need to be replaced with the use of table entry pointers. This will be done on different parts at a time adding table entry class access functions as needed.
This process was started by changing the table find functions from returning a code enumerator to a table entry pointer (returning a null pointer to indicate no table entry was found). Only the parser routines were using the find functions, so the get identifier and get operator functions were updated to use table entry pointers instead of a code enumerator.
A few table entry access functions were added to support the parser changes including the code, is code, name, has flag and alternate functions. For now the code function simply returns an index value of the entry by subtracting the base of the table entries array. This function is temporary. The is code function checks if the table entry is for a particular code. For now it compares to the code function return value, but eventually will compare to the code member that will be added to the table entry. The alternate function is similar to the alternate code function but returns a table entry pointer.
The code argument of the two token constructors for codes were changed to table entry pointers. For now they just access the code function of the table entry. The immediate goal is to change the interfaces and later to change the underlying code when the token code member is replaced with a table entry pointer. The token constructor taking a code was temporarily left (though the unneeded arguments were removed) for use by the translator routines.
The table entry class was made a friend class of the table class (specifically so the alternate member function can access the static alternate member of the table class). Eventually, the table entry and table classes will be combined into a single class.
[branch table commit 78f0b39780]
This process was started by changing the table find functions from returning a code enumerator to a table entry pointer (returning a null pointer to indicate no table entry was found). Only the parser routines were using the find functions, so the get identifier and get operator functions were updated to use table entry pointers instead of a code enumerator.
A few table entry access functions were added to support the parser changes including the code, is code, name, has flag and alternate functions. For now the code function simply returns an index value of the entry by subtracting the base of the table entries array. This function is temporary. The is code function checks if the table entry is for a particular code. For now it compares to the code function return value, but eventually will compare to the code member that will be added to the table entry. The alternate function is similar to the alternate code function but returns a table entry pointer.
The code argument of the two token constructors for codes were changed to table entry pointers. For now they just access the code function of the table entry. The immediate goal is to change the interfaces and later to change the underlying code when the token code member is replaced with a table entry pointer. The token constructor taking a code was temporarily left (though the unneeded arguments were removed) for use by the translator routines.
The table entry class was made a friend class of the table class (specifically so the alternate member function can access the static alternate member of the table class). Eventually, the table entry and table classes will be combined into a single class.
[branch table commit 78f0b39780]
Saturday, January 3, 2015
Parser – Parentheses Token Handling
With the forthcoming change from the token type and code enumerations to the code type enumeration, it will be advantageous if all similar codes have the same code type. Table flags will be used command, operator and functions codes since some will need their own code type (for example the LET command and equal operator codes).
The codes being discussed are the codes with operands: constants, variables, arrays, defined functions and user functions. Each have separate codes for each of the three data types, and each (except constants) have an additional three of each data type for reference codes. Currently only constants and variables are fully implemented. For variables, there will be a single Variable code type for each of its six codes. The translator will only need to check this single code type for a variable code.
There was a distinction between token types with parentheses and without (internal functions, defined functions, and generic tokens). For defined and internal functions, this distinction was removed (the token type enumerators were combined for each). For internal functions, instead of checking the token type to determine if there are parentheses, the number of operands is now checked. Eventually defined functions (and later user functions) will have a similar check as the number of operands will be stored in their associated dictionaries.
The parser get word helper function was modified to only check if an open parentheses is present and add it to the identifier string as before, but not take it from the input stream. The parentheses is still needed for functions when searching the table. The get identifier function already removed the parentheses if a table entry was not found, but was modified to remove the parentheses from the input for internal functions only. A new get parentheses access function was added to check if the next character in the stream is an open parentheses, remove the parentheses, and return whether it was a parentheses.
The translator get operand function was modified accordingly. For the single internal function token type, the process internal function routine is only called if the function has operands. For the single defined function token type, the process parentheses token is only called if the next character in the parser is a parentheses (calls the new get parentheses function to remove the parentheses). For the generic parentheses token type (array or user function), the get parentheses function is called to remove the parentheses.
The two separate table entries for defined functions with and without parentheses remain for now with their associated code enumerators and are used to determine if parentheses are present. Once defined functions are fully implemented, these two codes will be replaced with six codes (as described above) and the defined function dictionary will contain whether there should be parentheses (if there are operands).
The test token stream inserter and print token functions were modified for the change in token type enumerators. The token has parentheses access function was only used by the print token function, which used a static map member. The print token function was modified to not require this access function, so it and the static map member were removed. There was also a static map member for precedence. Since all tokens now have a code assigned, all precedences can be obtained from the table, so this static member and its access function were also removed. The expected parser results for tests #2, #4 and #5 were updated for the change in the token types.
[branch table commit 3dd768f604]
The codes being discussed are the codes with operands: constants, variables, arrays, defined functions and user functions. Each have separate codes for each of the three data types, and each (except constants) have an additional three of each data type for reference codes. Currently only constants and variables are fully implemented. For variables, there will be a single Variable code type for each of its six codes. The translator will only need to check this single code type for a variable code.
There was a distinction between token types with parentheses and without (internal functions, defined functions, and generic tokens). For defined and internal functions, this distinction was removed (the token type enumerators were combined for each). For internal functions, instead of checking the token type to determine if there are parentheses, the number of operands is now checked. Eventually defined functions (and later user functions) will have a similar check as the number of operands will be stored in their associated dictionaries.
The parser get word helper function was modified to only check if an open parentheses is present and add it to the identifier string as before, but not take it from the input stream. The parentheses is still needed for functions when searching the table. The get identifier function already removed the parentheses if a table entry was not found, but was modified to remove the parentheses from the input for internal functions only. A new get parentheses access function was added to check if the next character in the stream is an open parentheses, remove the parentheses, and return whether it was a parentheses.
The translator get operand function was modified accordingly. For the single internal function token type, the process internal function routine is only called if the function has operands. For the single defined function token type, the process parentheses token is only called if the next character in the parser is a parentheses (calls the new get parentheses function to remove the parentheses). For the generic parentheses token type (array or user function), the get parentheses function is called to remove the parentheses.
The two separate table entries for defined functions with and without parentheses remain for now with their associated code enumerators and are used to determine if parentheses are present. Once defined functions are fully implemented, these two codes will be replaced with six codes (as described above) and the defined function dictionary will contain whether there should be parentheses (if there are operands).
The test token stream inserter and print token functions were modified for the change in token type enumerators. The token has parentheses access function was only used by the print token function, which used a static map member. The print token function was modified to not require this access function, so it and the static map member were removed. There was also a static map member for precedence. Since all tokens now have a code assigned, all precedences can be obtained from the table, so this static member and its access function were also removed. The expected parser results for tests #2, #4 and #5 were updated for the change in the token types.
[branch table commit 3dd768f604]
Friday, January 2, 2015
Token – Header File Refactoring
With the new table model, the token class will contain a pointer to the table entry of the code instead of the code enumerator (currently used as an index). The table header file currently includes the token header file for the many table functions with a token pointer argument. This is no longer necessary since only token pointer and token pointer references are used (forward declarations are sufficient).
When the token class contains a pointer to the table entry, optimal access to the table entry members will be achieved if the entry access functions are defined in-line. Similarly, optimal access indirectly through to the token access functions to the table members is desired. This will require the table header file to be included in the token header file.
Therefore, some header file refactoring was performed where the token header include statements were moved from all the other header files to the associated source file with the exception of the table header file, which is uses the token type enumeration. When this enumeration is replaced with the code type enumeration, this statement can be removed. Forward declarations were added as needed.
[branch table commit 0777959464]
When the token class contains a pointer to the table entry, optimal access to the table entry members will be achieved if the entry access functions are defined in-line. Similarly, optimal access indirectly through to the token access functions to the table members is desired. This will require the table header file to be included in the token header file.
Therefore, some header file refactoring was performed where the token header include statements were moved from all the other header files to the associated source file with the exception of the table header file, which is uses the token type enumeration. When this enumeration is replaced with the code type enumeration, this statement can be removed. Forward declarations were added as needed.
[branch table commit 0777959464]
Parser – Token Creation
A side effect of the last change was that tokens for both codes for operators, functions and commands and codes with operands (constants, variables, arrays, defined functions and user functions) were using the same token constructor. This token constructor searched through alternate codes for the code with the appropriate return data type.
This was unnecessary for operator, function and command codes. The table new token function called for these codes passed in the return data type from the table entry of the code. The token constructor then called the new table set token code function. Since the data type matched the return data type (which was just passed in), no alternates were checked and the code, type and data type of the token was set. This was extra unnecessary work.
A new token constructor was added for operator, function and command codes, which only required arguments for the code, column, length and string. The string argument is only used for the REM and REM operator codes. This constructor replaces the table new token function. This constructor calls the table set code function which just sets the code, type and data type of the token from the table entry of the code. For consistency the code argument was put first in the other token constructor for codes with operands.
While looking at the creation of tokens, I decided that using the standard unique pointer within the parser was unnecessary. The parser can just allocate a token and return its pointer. The translator then can put the allocated tokens into a standard shared pointer. The parser was changed to use plain token pointers. The translator routines were changed to use the new token constructor directly via the standard make shared function. The translator get operand was changed to use the reset function to set the token member since shared pointers cannot be assigned directly to a pointer.
[branch table commit 1bfb76ae0a]
This was unnecessary for operator, function and command codes. The table new token function called for these codes passed in the return data type from the table entry of the code. The token constructor then called the new table set token code function. Since the data type matched the return data type (which was just passed in), no alternates were checked and the code, type and data type of the token was set. This was extra unnecessary work.
A new token constructor was added for operator, function and command codes, which only required arguments for the code, column, length and string. The string argument is only used for the REM and REM operator codes. This constructor replaces the table new token function. This constructor calls the table set code function which just sets the code, type and data type of the token from the table entry of the code. For consistency the code argument was put first in the other token constructor for codes with operands.
While looking at the creation of tokens, I decided that using the standard unique pointer within the parser was unnecessary. The parser can just allocate a token and return its pointer. The translator then can put the allocated tokens into a standard shared pointer. The parser was changed to use plain token pointers. The translator routines were changed to use the new token constructor directly via the standard make shared function. The translator get operand was changed to use the reset function to set the token member since shared pointers cannot be assigned directly to a pointer.
[branch table commit 1bfb76ae0a]
Parser – Codes With Operands
The last token type not being set fully in the parser were codes with operands (constants, variables, arrays, defined functions and user functions). Constant tokens were corrected with the last change. Arrays, defined functions and user functions are not fully implemented and so did not need to be changed. Variables however, were only partially set in the parser (only to the base Variable or Variable Reference code) and weren't set for the data type of the variable until the translator.
The parser get identifier function was modified to set the data type to Double if the word obtained from the input does not have a type. This applies to all identifiers not found in the table. The token constructor for codes is used for commands, operators, functions and codes with operands. The type argument was unnecessary since that is set from the table entry. However, an issue was found with how codes were found in the table.
For operators and functions, the [return] data type of the token is set from the table entry. (This issue doesn't affect commands since command don't have a return data type.) For codes with operands, the data type of the identifier is used to find the appropriate table entry (for example, Variable, Variable Integer, or Variable String) by looking at the data types of alternate codes. The current table set token code function did not work correctly because it searches alternate codes by operand data type. For this instance, the alternate codes need to be search by return data type.
A new set token code function was added without an operand index argument to search by return data type. If the data type (of the identifier) does match the code passed, then the alternate codes are searched for a matching return data type. If there are no alternates or none were found, then the code passed is set in the token along with the token type of the code. The data type is set to the data type of the identifier and not from the table entry (which may not match for codes like arrays that are not fully implemented yet).
The type argument was removed from the token constructor for codes. The type from the table entry of the code was passed (and the new set token code now does this). A call to the new set token code was added to the body of the constructor (previously empty).
Since codes for constants, variable, and variable references were found in the table incorrectly by operand data type, these table entries contained operand data types so that it would work. These codes do not have operands (in the sense that operands and functions do within expressions; not to be confused that in the program, these codes do have an operand index). These table entries were corrected with expression info instances containing no operands.
The translator get operand previously set the default data type of the token just obtained (set to Double if None and not a function). This was removed since the parser now does this. The token set default data type function called to do this was removed. The call to set the code for a no parentheses (variable) token was also no longer needed. With the parser now setting the default data type to Double, the expected results to the parser tests (#2, #3 and #5) needed to be updated.
[branch table commit acc37f0650]
The parser get identifier function was modified to set the data type to Double if the word obtained from the input does not have a type. This applies to all identifiers not found in the table. The token constructor for codes is used for commands, operators, functions and codes with operands. The type argument was unnecessary since that is set from the table entry. However, an issue was found with how codes were found in the table.
For operators and functions, the [return] data type of the token is set from the table entry. (This issue doesn't affect commands since command don't have a return data type.) For codes with operands, the data type of the identifier is used to find the appropriate table entry (for example, Variable, Variable Integer, or Variable String) by looking at the data types of alternate codes. The current table set token code function did not work correctly because it searches alternate codes by operand data type. For this instance, the alternate codes need to be search by return data type.
A new set token code function was added without an operand index argument to search by return data type. If the data type (of the identifier) does match the code passed, then the alternate codes are searched for a matching return data type. If there are no alternates or none were found, then the code passed is set in the token along with the token type of the code. The data type is set to the data type of the identifier and not from the table entry (which may not match for codes like arrays that are not fully implemented yet).
The type argument was removed from the token constructor for codes. The type from the table entry of the code was passed (and the new set token code now does this). A call to the new set token code was added to the body of the constructor (previously empty).
Since codes for constants, variable, and variable references were found in the table incorrectly by operand data type, these table entries contained operand data types so that it would work. These codes do not have operands (in the sense that operands and functions do within expressions; not to be confused that in the program, these codes do have an operand index). These table entries were corrected with expression info instances containing no operands.
The translator get operand previously set the default data type of the token just obtained (set to Double if None and not a function). This was removed since the parser now does this. The token set default data type function called to do this was removed. The call to set the code for a no parentheses (variable) token was also no longer needed. With the parser now setting the default data type to Double, the expected results to the parser tests (#2, #3 and #5) needed to be updated.
[branch table commit acc37f0650]
Wednesday, December 31, 2014
Constant Token Codes
The code and token type enumerations will be combined into a single code type enumeration. Before proceeding, the parser needs to return all tokens assigned to a code. This has been mostly accomplished, though one exception is constant tokens, which were still being assigned codes in the translator. This is complicated because the type of numerical constants may not be known when the constant is parsed. Consider these statements:
Instead of passing whether a number is allowed flag to the parser, the requested data type is now passed. If the data type is integer or double, the code of a constant token is fixed (the Integer Constant sub-code is not needed and is cleared if set). For other requested data types, either the Constant or Constant Integer code is set is described above with the Integer Constant sub-code set for small doubles. The parser also now sets the Constant String code for string constants. The parser makes no attempt to report any errors for data type mismatches.
The decimal flag argument of the token constructor for double constants was removed as the data type is set to Double and the Integer Constant sub-code is set if the value is within the integer range. The convert code was cleaned up by making the desired data type the primary switch and there was no need for secondary switches on the token data type since only one of two data types need to be checked for each desired data type. A convert constant helper function was added to handle changing constant token codes.
The table find code function was simplified due to the change on how constants are represented. The first argument of the set token and set token code functions were changed from a standard shared token pointer reference to a straight token pointer so that they can be called from the parser (with a standard unique pointer), translator (with a standard shared pointer) or token member function (with just a pointer). This simply required calling the get access function of the unique or shared pointers.
The translator get operand function no longer sets the code for constants. The get expression and process internal function functions no longer need to look for and set the codes for constants (the later needs to clear the Integer Constant sub-code for functions taking both number argument types, specifically ABS, SGN and STR$). And the get token function now only needs to pass the data type to the parser. The token convert and table find code functions are used by the translator and will finalize constants not set by the parser once the final data type is known.
[branch table commit 3099f8850f]
A = B + 5For numerical constants, both the integer and double representations of the constant is stored in the constant dictionary except for the case where a double constant does not fit into a 32-bit signed integer. Optimally the representation required is used without a hidden conversion code to unnecessarily convert the constant. For the first statement above, the double value of the constant is used. With the other two statements, the integer value of the constant is used. Number constant tokens have three states:
A% = B% + 5
A% = B% + 5.4
- An integer (no decimal point or exponent; fits into 32 bits)
- A small double (has a decimal point or an exponent; fits into 32 bits when converted)
- A large double (does not fit into 32 bits; cannot be converted)
Instead of passing whether a number is allowed flag to the parser, the requested data type is now passed. If the data type is integer or double, the code of a constant token is fixed (the Integer Constant sub-code is not needed and is cleared if set). For other requested data types, either the Constant or Constant Integer code is set is described above with the Integer Constant sub-code set for small doubles. The parser also now sets the Constant String code for string constants. The parser makes no attempt to report any errors for data type mismatches.
The decimal flag argument of the token constructor for double constants was removed as the data type is set to Double and the Integer Constant sub-code is set if the value is within the integer range. The convert code was cleaned up by making the desired data type the primary switch and there was no need for secondary switches on the token data type since only one of two data types need to be checked for each desired data type. A convert constant helper function was added to handle changing constant token codes.
The table find code function was simplified due to the change on how constants are represented. The first argument of the set token and set token code functions were changed from a standard shared token pointer reference to a straight token pointer so that they can be called from the parser (with a standard unique pointer), translator (with a standard shared pointer) or token member function (with just a pointer). This simply required calling the get access function of the unique or shared pointers.
The translator get operand function no longer sets the code for constants. The get expression and process internal function functions no longer need to look for and set the codes for constants (the later needs to clear the Integer Constant sub-code for functions taking both number argument types, specifically ABS, SGN and STR$). And the get token function now only needs to pass the data type to the parser. The token convert and table find code functions are used by the translator and will finalize constants not set by the parser once the final data type is known.
[branch table commit 3099f8850f]
Saturday, December 27, 2014
Table – More Alternate Codes
Since it will be ideal to reduce the number of unique code type enumerators needed, several uses of the current code enumerators was eliminated. Most of these were accomplished by assigning the codes needed as alternate codes to commands that need them in the Alternate Info initializer list and obtaining the codes using the alternate map:
The Input code enumerator was used in the common INPUT translate routine to determine if the routine was called for the INPUT or INPUT PROMPT command. This test was changed to checking if the second name of the command token code is empty, which is empty for INPUT and not empty for INPUT PROMPT.
The Constant String code enumerator was used in the token equality operator function to determine if token string comparison should be case sensitive (for REM, REM operator and string constants) or case insensitive (variables, arrays, etc.). This test was changed to checking if the token type is a constant and the data type is a string.
The Print code enumerator was used in the print recreate function instead of using the name from the table entry. This was done because the print semicolon recreate function called this function after appending a semicolon. If the table entry was used, a semicolon would have been output for the command instead of PRINT. The function was changed to create a temporary RPN item with a temporary token with the Print code, but not the code enumerator using the alternate code. This set of statements is convoluted, but will be far simpler once the new table model is implemented.
Unrelated to removing the use of code enumerators, it was noticed that the print comma recreate and print semicolon recreate functions were using the strings for a comma and semicolon directly but technically should have been using the name from the table entry. Both of these were changed to use the table entry name.
[branch table commit b6850c7c7f]
Assign → Let (alternate 0)Since the command code is available in the translate function needing them (LET, INPUT, or PRINT) , it can be used to find the alternate code needed. The Print code is assigned as an alternate to the Semicolon code for recreation (see below). For a PRINT statement ending with a semicolon, no PRINT command code is stored in the program code so the semicolon recreate needs to also recreate the PRINT keyword.
Input Begin → Input (alternate 0)
Input Begin String → Input Prompt (alternate 0)
Input Assign → Input (alternate 1)
Input Assign → Input Prompt (alternate 1)
Print Double → Print (alternate 0)
Print → Semicolon (alternate 0)
Variable Reference → Variable (alternate 1)
The Input code enumerator was used in the common INPUT translate routine to determine if the routine was called for the INPUT or INPUT PROMPT command. This test was changed to checking if the second name of the command token code is empty, which is empty for INPUT and not empty for INPUT PROMPT.
The Constant String code enumerator was used in the token equality operator function to determine if token string comparison should be case sensitive (for REM, REM operator and string constants) or case insensitive (variables, arrays, etc.). This test was changed to checking if the token type is a constant and the data type is a string.
The Print code enumerator was used in the print recreate function instead of using the name from the table entry. This was done because the print semicolon recreate function called this function after appending a semicolon. If the table entry was used, a semicolon would have been output for the command instead of PRINT. The function was changed to create a temporary RPN item with a temporary token with the Print code, but not the code enumerator using the alternate code. This set of statements is convoluted, but will be far simpler once the new table model is implemented.
Unrelated to removing the use of code enumerators, it was noticed that the print comma recreate and print semicolon recreate functions were using the strings for a comma and semicolon directly but technically should have been using the name from the table entry. Both of these were changed to use the table entry name.
[branch table commit b6850c7c7f]
Table – New Code Type Enumeration
With the current table model, each entry contains a token type that identifies the type of token that is created for the table entry. Each entry has a code enumerator, which is simply used as an index to the entry. The code (index) is put into the program code. The code enumeration was originally automatically generated from comments next to the table entries. This avoided mismatches between the code enumerators and the table entries (though this was a poor design choice). Only some entries were identified using these code enumerators.
For the new table model, each table entry will be a unique class with a unique instance. Each table entry instance will be assigned a unique index by the base table entry constructor. Some table entries will still need to be found be some means by the parser and translator. This could be done directly by referencing the table entry instance, but this would require exposing the derived table entry classes. The plan is only to expose the base table entry class definition.
To find these small number of table entries, an enumeration will still be used. This enumeration will be similar to the token type enumeration currently given to each table entry. Therefore a single Code Type enumeration will be defined and will also replace the token type enumeration. Unlike the current Code enumeration, table entries could have the same Code Type enumerator (for example, all of the six variable codes will be assigned the Variable code type). Only the first table entry assigned to a code type will be returned for a code type enumerator; the others will be assigned as alternate codes of the first.
Code types will generally not be assigned to table entries for commands, operators and functions (which will no longer be referred to as internal functions) except for a few cases (for example, the LET command and the equal operator). New table flags will be assigned instead. There is already a Command table flag, and there will be Operator and Function table flags. Each of the codes with operands (variables, arrays, constants, defined functions, user functions, and subroutines) will have a code type. There will not be separate enumerators for tokens with and without parentheses (more on this later).
For the new table model, each table entry will be a unique class with a unique instance. Each table entry instance will be assigned a unique index by the base table entry constructor. Some table entries will still need to be found be some means by the parser and translator. This could be done directly by referencing the table entry instance, but this would require exposing the derived table entry classes. The plan is only to expose the base table entry class definition.
To find these small number of table entries, an enumeration will still be used. This enumeration will be similar to the token type enumeration currently given to each table entry. Therefore a single Code Type enumeration will be defined and will also replace the token type enumeration. Unlike the current Code enumeration, table entries could have the same Code Type enumerator (for example, all of the six variable codes will be assigned the Variable code type). Only the first table entry assigned to a code type will be returned for a code type enumerator; the others will be assigned as alternate codes of the first.
Code types will generally not be assigned to table entries for commands, operators and functions (which will no longer be referred to as internal functions) except for a few cases (for example, the LET command and the equal operator). New table flags will be assigned instead. There is already a Command table flag, and there will be Operator and Function table flags. Each of the codes with operands (variables, arrays, constants, defined functions, user functions, and subroutines) will have a code type. There will not be separate enumerators for tokens with and without parentheses (more on this later).
Friday, December 26, 2014
Sub-Code Enumeration Refactoring
When the Double sub-code enumerator value was changed to support double identifiers with the double data type character (#), it was noticed that the Parentheses and Colon sub-codes would never be used with the same codes. The Parentheses sub-code will only be set for operators and operands (variable, constants, etc.), and the Colon sub-code will only be set for commands (with some exceptions).
The available sub-code bits are limited - there are only six. Several sub-codes were already combined with the Option sub-code ('LET' for assignment codes, 'Question' for the Input Begin String code, and 'Keep' for the INPUT and INPUT PROMPT command codes). The same thing was done for the Parentheses and Colon sub-codes, but instead of using the same sub-code enumerator for both (creating an appropriate name was impossible), both current enumerators were simply given the same value.
There was one problem with this scheme. Two codes that could have the Colon sub-code are the Comma and Semicolon codes, which are only present in a PRINT statement and will be commands at run-time. These codes are defined as operators. The assignment codes are defined as no types, but are commands and could have the Colon sub-code. To indicate that these are commands, a new Command table flag was added and their table entries were given this flag.
Therefore, a code has the Colon sub-code if it is a command type or has the Command table flag, otherwise the code has the Parentheses sub-code. Checks for the Command table flag were added to the recreator function operator, the stream insert operator for program words, and the test stream insert operator for tokens. A few other minor sub-code related changes were made, click Continue... for details.
The available sub-code bits are limited - there are only six. Several sub-codes were already combined with the Option sub-code ('LET' for assignment codes, 'Question' for the Input Begin String code, and 'Keep' for the INPUT and INPUT PROMPT command codes). The same thing was done for the Parentheses and Colon sub-codes, but instead of using the same sub-code enumerator for both (creating an appropriate name was impossible), both current enumerators were simply given the same value.
There was one problem with this scheme. Two codes that could have the Colon sub-code are the Comma and Semicolon codes, which are only present in a PRINT statement and will be commands at run-time. These codes are defined as operators. The assignment codes are defined as no types, but are commands and could have the Colon sub-code. To indicate that these are commands, a new Command table flag was added and their table entries were given this flag.
Therefore, a code has the Colon sub-code if it is a command type or has the Command table flag, otherwise the code has the Parentheses sub-code. Checks for the Command table flag were added to the recreator function operator, the stream insert operator for program words, and the test stream insert operator for tokens. A few other minor sub-code related changes were made, click Continue... for details.
Thursday, December 25, 2014
Double Identifier Problem
An existing problem was discovered when the parser was modified to not store the data type character of identifiers. The issue was with double identifiers when using the optional # data type character. The identifiers Variable and Variable# were incorrectly added to the dictionary as separate entries when they should have been the same entry.
The parser get identifier function was modified to not store the data type character in the token. This caused a problem when recreating double identifiers where the # character entered would disappear. Recreating all double identifiers with a # character was also not desirable. This was corrected by adding the Double sub-code to the token. A sub-code argument was added to the token constructor for identifiers. This sub-code is encoded into the program code so that the # character is recreated when it is entered.
The Double sub-code was only being used for constants. When the value of a constant is within the integer range, its data type is set to integer, and if a decimal point is present, the Double sub-code is set. The translator uses this sub-code to determine if a constant can be used as a double even though the data type is integer (see post from October 28 for details). This sub-code does not survive past the translator (not put into the program code).
A new string with data type access function was added to the token to add the data type character (#, % for integers, and $ for strings) to the token string returned. A # character is only added if the Double sub-code is set. This function replaced the string access function in the test token stream insert operator, tester print token, and several recreate functions.
A sub-code argument was added to the table entry operand text functions. The variable operand text functions were modified to add the data type character to the variable name. For double variables, the character is only added if the Double sub-code is set. An Ignore sub-code enumerator was added, and when passed to the operand text function, no data type character is added to the variable name. This option was needed for the program model decode function that uses the operand text function to set the string of the token (since tokens no longer store the data type character).
The value of the Double sub-code was changed so that its bit value was within the range of the sub-code bits (not necessary before since this sub-code was not used in the program code). The return type of program code instruction sub-code access function was changed to the Sub-Code enumeration type (from an integer). The expected encoder test results were updated, specifically the dictionaries output since the data type characters are no longer present in the entries.
[branch table commit e97057efca]
The parser get identifier function was modified to not store the data type character in the token. This caused a problem when recreating double identifiers where the # character entered would disappear. Recreating all double identifiers with a # character was also not desirable. This was corrected by adding the Double sub-code to the token. A sub-code argument was added to the token constructor for identifiers. This sub-code is encoded into the program code so that the # character is recreated when it is entered.
The Double sub-code was only being used for constants. When the value of a constant is within the integer range, its data type is set to integer, and if a decimal point is present, the Double sub-code is set. The translator uses this sub-code to determine if a constant can be used as a double even though the data type is integer (see post from October 28 for details). This sub-code does not survive past the translator (not put into the program code).
A new string with data type access function was added to the token to add the data type character (#, % for integers, and $ for strings) to the token string returned. A # character is only added if the Double sub-code is set. This function replaced the string access function in the test token stream insert operator, tester print token, and several recreate functions.
A sub-code argument was added to the table entry operand text functions. The variable operand text functions were modified to add the data type character to the variable name. For double variables, the character is only added if the Double sub-code is set. An Ignore sub-code enumerator was added, and when passed to the operand text function, no data type character is added to the variable name. This option was needed for the program model decode function that uses the operand text function to set the string of the token (since tokens no longer store the data type character).
The value of the Double sub-code was changed so that its bit value was within the range of the sub-code bits (not necessary before since this sub-code was not used in the program code). The return type of program code instruction sub-code access function was changed to the Sub-Code enumeration type (from an integer). The expected encoder test results were updated, specifically the dictionaries output since the data type characters are no longer present in the entries.
[branch table commit e97057efca]
Parser – Identifier Codes
The parser previously set the code for an identifier token only when the word was found in the table (command, operator or function). The codes for other identifiers were set in the translator: defined functions with no parentheses and variables (get operand); arrays, functions, and defined functions with parentheses (process parentheses tokens). This was changed to set all codes in the parser.
To do this in the parser, the parser needed to know if a reference operand was being requested. For now identifiers with no parentheses are set to variables, and with parentheses are set to arrays unless they start with an F (temporary check for testing). Defined functions are identifiers that start with an FN. Eventually the parser will need access to the program dictionaries to fully determine which code to assign to an identifier token.
The get identifier function was modified to set the code as described above for identifiers not found in the table. A reference argument was added, which was also added to the parser function operator. (The Reference enumeration was moved from the translator class to the main header file so that its enumerators are accessible.) The token constructor for codes and identifiers were combined to a single constructor with default arguments for the string and reference members.
For variables, the reference argument is used to determine if the code is a variable or a variable reference. Only the base code is set as the translator changed the code for the data type. In the case of a variable reference, the reference member of the token is not set (the translator did not previously set it either).
Several token type cases in the translator get operand function was modified. For defined functions with no parentheses, the token reference and code members no longer need to be set. For no parentheses tokens (variables), the code is still updated for the data type. The parser will do this once the new table model is implemented. For parentheses tokens (arrays), the token reference member no longer needs to be set.
The translator process parentheses token function no longer does the check for functions (temporarily identifiers starting with F), or set the code of the token. For determining an array (to set the expected expression types to integer for the subscripts), the Array code is checked for. This check will need to be modified when arrays are implemented since there will be different array codes for each data type, which will be set by the parser.
[branch table commit 69dff18e26]
To do this in the parser, the parser needed to know if a reference operand was being requested. For now identifiers with no parentheses are set to variables, and with parentheses are set to arrays unless they start with an F (temporary check for testing). Defined functions are identifiers that start with an FN. Eventually the parser will need access to the program dictionaries to fully determine which code to assign to an identifier token.
The get identifier function was modified to set the code as described above for identifiers not found in the table. A reference argument was added, which was also added to the parser function operator. (The Reference enumeration was moved from the translator class to the main header file so that its enumerators are accessible.) The token constructor for codes and identifiers were combined to a single constructor with default arguments for the string and reference members.
For variables, the reference argument is used to determine if the code is a variable or a variable reference. Only the base code is set as the translator changed the code for the data type. In the case of a variable reference, the reference member of the token is not set (the translator did not previously set it either).
Several token type cases in the translator get operand function was modified. For defined functions with no parentheses, the token reference and code members no longer need to be set. For no parentheses tokens (variables), the code is still updated for the data type. The parser will do this once the new table model is implemented. For parentheses tokens (arrays), the token reference member no longer needs to be set.
The translator process parentheses token function no longer does the check for functions (temporarily identifiers starting with F), or set the code of the token. For determining an array (to set the expected expression types to integer for the subscripts), the Array code is checked for. This check will need to be modified when arrays are implemented since there will be different array codes for each data type, which will be set by the parser.
[branch table commit 69dff18e26]
Sunday, December 21, 2014
Table – Entry Pointers
The new table implementation will have a single Table class that will represent a table entry for a single code and will serve as the base class to all of the derived table entry classes. Global table information (name to entry map, alternate code map, etc.) will be static members of this class. This is equivalent to the current since table instance.
Currently the Table class and Table Entry structure are separate definitions, but eventually the table entry members will be members of the new base table class. The current table class has many access functions where their first argument is a code enumerator (which is currently used as an index). There are also many access functions that have a token pointer as their first argument, but this is mostly used to get the code from the token.
The new table model will access the table entries by a pointer to the entry instead of be a code enumerator used as an index. The code argument access functions will become simply access functions to the table entry. The next step in this transition to the new table model will be to use pointers to table entries instead of code enumerators. The token code member will become a table entry pointer.
To start this transition, the Table Entry structure was moved from the table source file to the table header file. The members were also renamed with the member (m_) prefix except for the function pointer members (which will be replaced with virtual functions in the new table model).
[branch table commit 94decad26c]
Currently the Table class and Table Entry structure are separate definitions, but eventually the table entry members will be members of the new base table class. The current table class has many access functions where their first argument is a code enumerator (which is currently used as an index). There are also many access functions that have a token pointer as their first argument, but this is mostly used to get the code from the token.
The new table model will access the table entries by a pointer to the entry instead of be a code enumerator used as an index. The code argument access functions will become simply access functions to the table entry. The next step in this transition to the new table model will be to use pointers to table entries instead of code enumerators. The token code member will become a table entry pointer.
To start this transition, the Table Entry structure was moved from the table source file to the table header file. The members were also renamed with the member (m_) prefix except for the function pointer members (which will be replaced with virtual functions in the new table model).
[branch table commit 94decad26c]
Table – Internal Code Token Types
There were several internal code table entries (null, assignment, print item, input assign and input parse) that were assigned to either an Operator or Internal Function token type. These internal codes do not require a token type because they are not produced by the parser (the token type is only used for tokens from the parser). These table entries were changed to the default token type (in other words, no type).
The reason for this change will become more evident when the new table class hierarchy is implemented. One of the goals of which is to reduce the amount of unnecessary initialization values. It may even turn out that the token type member of all table entries will be unnecessary, but this is not clear yet.
These changes did cause a minor issue in test output. By default the test output stream insert operator for a token outputs nothing for a token without a type, which cause the above modified codes to not produce their debug name. This function was modified to output the debug name in the default case instead of doing nothing. Since the internal function types also only output the debug name, these cases were removed to let the default case handle these types.
[branch table commit 232a97f6d6]
The reason for this change will become more evident when the new table class hierarchy is implemented. One of the goals of which is to reduce the amount of unnecessary initialization values. It may even turn out that the token type member of all table entries will be unnecessary, but this is not clear yet.
These changes did cause a minor issue in test output. By default the test output stream insert operator for a token outputs nothing for a token without a type, which cause the above modified codes to not produce their debug name. This function was modified to output the debug name in the default case instead of doing nothing. Since the internal function types also only output the debug name, these cases were removed to let the default case handle these types.
[branch table commit 232a97f6d6]
Table – New Token Consolidation
The next major change to the table class will be to start transitioning to using table entry pointers instead of code enumerators and to remove the use of code enumerators as indexes. The code and index values will be separate members of table entries. All table entries will have an index (which is put into the program code), but only a few table entries will have a code (only those the require specific lookup like some of the special symbols including comma, parentheses, colon, etc.). Before proceeding with this, a small simplification was made first.
There were two new token functions in the table class, one taking a single code argument and the other taking column, length and code arguments. The single argument version relied on the default token constructor. This was one of three uses of the default token constructor. The default token constructor contained optional column and length arguments (default to -1 indicating unset), but there were no callers of the default token constructor that used these arguments.
The single code argument new token function was removed along with the default token constructor. The code argument was made the first argument of the other version of the new token function with default -1 values provided for the optional column and length arguments. This second version does not use the default token constructor. All callers of this function were in the parser and were modified for the reordering of the arguments.
The second use of the default token constructor was by the decode function in the program model class to create a default token, which it then used the token set code access function to set the code. This function was changed to use the new token function.
The third use of the default token constructor was in the INPUT translate function where a new token is needed for an input assign code and another token (comma or semicolon) is not available for reuse. This was changed to use the new token function with a Null code.
[branch table commit b5dd96c272]
There were two new token functions in the table class, one taking a single code argument and the other taking column, length and code arguments. The single argument version relied on the default token constructor. This was one of three uses of the default token constructor. The default token constructor contained optional column and length arguments (default to -1 indicating unset), but there were no callers of the default token constructor that used these arguments.
The single code argument new token function was removed along with the default token constructor. The code argument was made the first argument of the other version of the new token function with default -1 values provided for the optional column and length arguments. This second version does not use the default token constructor. All callers of this function were in the parser and were modified for the reordering of the arguments.
The second use of the default token constructor was by the decode function in the program model class to create a default token, which it then used the token set code access function to set the code. This function was changed to use the new token function.
The third use of the default token constructor was in the INPUT translate function where a new token is needed for an input assign code and another token (comma or semicolon) is not available for reuse. This was changed to use the new token function with a Null code.
[branch table commit b5dd96c272]
Subscribe to:
Posts (Atom)