Table – New Code Type Enumeration

With the current table model, each entry contains a token type that identifies the type of token that is created for the table entry.  Each entry has a code enumerator, which is simply used as an index to the entry.  The code (index) is put into the program code.  The code enumeration was originally automatically generated from comments next to the table entries.  This avoided mismatches between the code enumerators and the table entries (though this was a poor design choice).  Only some entries were identified using these code enumerators.

For the new table model, each table entry will be a unique class with a unique instance.  Each table entry instance will be assigned a unique index by the base table entry constructor.  Some table entries will still need to be found be some means by the parser and translator.  This could be done directly by referencing the table entry instance, but this would require exposing the derived table entry classes.  The plan is only to expose the base table entry class definition.

To find these small number of table entries, an enumeration will still be used.  This enumeration will be similar to the token type enumeration currently given to each table entry.  Therefore a single Code Type enumeration will be defined and will also replace the token type enumeration.  Unlike the current Code enumeration, table entries could have the same Code Type enumerator (for example, all of the six variable codes will be assigned the Variable code type).  Only the first table entry assigned to a code type will be returned for a code type enumerator; the others will be assigned as alternate codes of the first.

Code types will generally not be assigned to table entries for commands, operators and functions (which will no longer be referred to as internal functions) except for a few cases (for example, the LET command and the equal operator).  New table flags will be assigned instead.  There is already a Command table flag, and there will be Operator and Function table flags.  Each of the codes with operands (variables, arrays, constants, defined functions, user functions, and subroutines) will have a code type.  There will not be separate enumerators for tokens with and without parentheses (more on this later).

Sub-Code Enumeration Refactoring

When the Double sub-code enumerator value was changed to support double identifiers with the double data type character (#), it was noticed that the Parentheses and Colon sub-codes would never be used with the same codes.  The Parentheses sub-code will only be set for operators and operands (variable, constants, etc.), and the Colon sub-code will only be set for commands (with some exceptions).

The available sub-code bits are limited - there are only six.  Several sub-codes were already combined with the Option sub-code ('LET' for assignment codes, 'Question' for the Input Begin String code, and 'Keep' for the INPUT and INPUT PROMPT command codes).  The same thing was done for the Parentheses and Colon sub-codes, but instead of using the same sub-code enumerator for both (creating an appropriate name was impossible), both current enumerators were simply given the same value.

There was one problem with this scheme.  Two codes that could have the Colon sub-code are the Comma and Semicolon codes, which are only present in a PRINT statement and will be commands at run-time.  These codes are defined as operators.  The assignment codes are defined as no types, but are commands and could have the Colon sub-code.  To indicate that these are commands, a new Command table flag was added and their table entries were given this flag.

Therefore, a code has the Colon sub-code if it is a command type or has the Command table flag, otherwise the code has the Parentheses sub-code.  Checks for the Command table flag were added to the recreator function operator, the stream insert operator for program words, and the test stream insert operator for tokens.  A few other minor sub-code related changes were made, click Continue... for details.

Double Identifier Problem

An existing problem was discovered when the parser was modified to not store the data type character of identifiers.   The issue was with double identifiers when using the optional # data type character.  The identifiers Variable and Variable# were incorrectly added to the dictionary as separate entries when they should have been the same entry.

The parser get identifier function was modified to not store the data type character in the token.  This caused a problem when recreating double identifiers where the # character entered would disappear.  Recreating all double identifiers with a # character was also not desirable.  This was corrected by adding the Double sub-code to the token.  A sub-code argument was added to the token constructor for identifiers.  This sub-code is encoded into the program code so that the # character is recreated when it is entered.

The Double sub-code was only being used for constants.  When the value of a constant is within the integer range, its data type is set to integer, and if a decimal point is present, the Double sub-code is set.  The translator uses this sub-code to determine if a constant can be used as a double even though the data type is integer (see post from October 28 for details).  This sub-code does not survive past the translator (not put into the program code).

A new string with data type access function was added to the token to add the data type character (#, % for integers, and $ for strings) to the token string returned.  A # character is only added if the Double sub-code is set.  This function replaced the string access function in the test token stream insert operator, tester print token, and several recreate functions.

A sub-code argument was added to the table entry operand text functions.  The variable operand text functions were modified to add the data type character to the variable name.  For double variables, the character is only added if the Double sub-code is set.  An Ignore sub-code enumerator was added, and when passed to the operand text function, no data type character is added to the variable name.  This option was needed for the program model decode function that uses the operand text function to set the string of the token (since tokens no longer store the data type character).

The value of the Double sub-code was changed so that its bit value was within the range of the sub-code bits (not necessary before since this sub-code was not used in the program code).  The return type of program code instruction sub-code access function was changed to the Sub-Code enumeration type (from an integer).  The expected encoder test results were updated, specifically the dictionaries output since the data type characters are no longer present in the entries.

[branch table commit e97057efca]

Parser – Identifier Codes

The parser previously set the code for an identifier token only when the word was found in the table (command, operator or function).  The codes for other identifiers were set in the translator: defined functions with no parentheses and variables (get operand); arrays, functions, and defined functions with parentheses (process parentheses tokens).  This was changed to set all codes in the parser.

To do this in the parser, the parser needed to know if a reference operand was being requested.  For now identifiers with no parentheses are set to variables, and with parentheses are set to arrays unless they start with an F (temporary check for testing).  Defined functions are identifiers that start with an FN.  Eventually the parser will need access to the program dictionaries to fully determine which code to assign to an identifier token.

The get identifier function was modified to set the code as described above for identifiers not found in the table.  A reference argument was added, which was also added to the parser function operator.  (The Reference enumeration was moved from the translator class to the main header file so that its enumerators are accessible.)  The token constructor for codes and identifiers were combined to a single constructor with default arguments for the string and reference members.

For variables, the reference argument is used to determine if the code is a variable or a variable reference.  Only the base code is set as the translator changed the code for the data type.  In the case of a variable reference, the reference member of the token is not set (the translator did not previously set it either).

Several token type cases in the translator get operand function was modified.  For defined functions with no parentheses, the token reference and code members no longer need to be set.  For no parentheses tokens (variables), the code is still updated for the data type.  The parser will do this once the new table model is implemented.  For parentheses tokens (arrays), the token reference member no longer needs to be set.

The translator process parentheses token function no longer does the check for functions (temporarily identifiers starting with F), or set the code of the token.  For determining an array (to set the expected expression types to integer for the subscripts), the Array code is checked for.  This check will need to be modified when arrays are implemented since there will be different array codes for each data type, which will be set by the parser.

[branch table commit 69dff18e26]

Table – Entry Pointers

The new table implementation will have a single Table class that will represent a table entry for a single code and will serve as the base class to all of the derived table entry classes.  Global table information (name to entry map, alternate code map, etc.) will be static members of this class.  This is equivalent to the current since table instance.

Currently the Table class and Table Entry structure are separate definitions, but eventually the table entry members will be members of the new base table class.  The current table class has many access functions where their first argument is a code enumerator (which is currently used as an index).  There are also many access functions that have a token pointer as their first argument, but this is mostly used to get the code from the token.

The new table model will access the table entries by a pointer to the entry instead of be a code enumerator used as an index.  The code argument access functions will become simply access functions to the table entry.  The next step in this transition to the new table model will be to use pointers to table entries instead of code enumerators.  The token code member will become a table entry pointer.

To start this transition, the Table Entry structure was moved from the table source file to the table header file.  The members were also renamed with the member (m_) prefix except for the function pointer members (which will be replaced with virtual functions in the new table model).

[branch table commit 94decad26c]

Table – Internal Code Token Types

There were several internal code table entries (null, assignment, print item, input assign and input parse) that were assigned to either an Operator or Internal Function token type.  These internal codes do not require a token type because they are not produced by the parser (the token type is only used for tokens from the parser).  These table entries were changed to the default token type (in other words, no type).

The reason for this change will become more evident when the new table class hierarchy is implemented.  One of the goals of which is to reduce the amount of unnecessary initialization values.  It may even turn out that the token type member of all table entries will be unnecessary, but this is not clear yet.

These changes did cause a minor issue in test output.  By default the test output stream insert operator for a token outputs nothing for a token without a type, which cause the above modified codes to not produce their debug name.  This function was modified to output the debug name in the default case instead of doing nothing.  Since the internal function types also only output the debug name, these cases were removed to let the default case handle these types.

[branch table commit 232a97f6d6]

Table – New Token Consolidation

The next major change to the table class will be to start transitioning to using table entry pointers instead of code enumerators and to remove the use of code enumerators as indexes.  The code and index values will be separate members of table entries.  All table entries will have an index (which is put into the program code), but only a few table entries will have a code (only those the require specific lookup like some of the special symbols including comma, parentheses, colon, etc.).  Before proceeding with this, a small simplification was made first.

There were two new token functions in the table class, one taking a single code argument and the other taking column, length and code arguments.  The single argument version relied on the default token constructor.  This was one of three uses of the default token constructor.  The default token constructor contained optional column and length arguments (default to -1 indicating unset), but there were no callers of the default token constructor that used these arguments.

The single code argument new token function was removed along with the default token constructor.  The code argument was made the first argument of the other version of the new token function with default -1 values provided for the optional column and length arguments.  This second version does not use the default token constructor.  All callers of this function were in the parser and were modified for the reordering of the arguments.

The second use of the default token constructor was by the decode function in the program model class to create a default token, which it then used the token set code access function to set the code.  This function was changed to use the new token function.

The third use of the default token constructor was in the INPUT translate function where a new token is needed for an input assign code and another token (comma or semicolon) is not available for reuse.  This was changed to use the new token function with a Null code.

[branch table commit b5dd96c272]

Table – Operand Arrays

There was a macro used for generating two arguments to the Expression Info constructor, which took an argument identifying the array less its suffix.  This macro was removed (the last such macro) and the predefined operand data type arrays were replaced with standard initializer lists.  With an initializer list, the size of the list is available.

The operand count and operand data type array pointer arguments of the Expression Info constructor were replaced with a standard initializer list of data types (with a default of a blank list).  The operand count member is initialized to the size of the operands list.  A standard initializer list is implemented as an array internally.  The begin access function is used to access the beginning of the initializer list to initialize the operand data type array.  The arguments of the expression info instances were modified to the initializer lists.

Another set of related changes were also made.  A null expression info instance was added (with no return value or operands).  The table entries that previously had their expression info pointer member initialized to a null pointer were changed to point to this null expression info instance.  This allowed the removal of the check for a null expression info pointer member in several of the access functions and from the add function.

[branch table commit fe5801c227]

Table – Associated Code Removal

Now that use of the associated code variables and access functions has been replaced with the alternate code map, the associated code members could be removed from the Expression Info structure and their access functions removed from the table class.  The predefined associated code arrays and the associated code macros were also removed.

Table entries that created their own expression info instance using one of the associated code macros were replaced with the appropriate pointer to a predefined expression info instance.  A few additional predefined expression info instances were needed.

[branch table commit 18cb29c185]

Table – Expected Data Type

The expected data type table entry member was recently moved from the Expression Info structure (because codes using the same return and operand data types could have different expected data types).  There were several issues with the expected data types initialization implementation (which were initialized automatically to prevent programming mistakes):

• Every table entry had an expected data type even it is was never used (for example, commands).  Even when it was in the Expression Info structure, it was not used for many codes (no argument functions, assignment codes, etc.).
• A separate iteration loop was needed to initialize the expected data types since the alternate code information was used, the alternate code map needed to be initialized first.
• The expected data type initialization took into account if a code had the possibility of having all three data types (Double, Integer, and String) where the expected data type would be set to Any even though there were actually such codes.

This implementation was replaced with a new table entry pointer to expected data type static map member.  Only table entries requiring an expected data type are added to this map.  This includes all primary codes and any alternate primary code.  An alternate primary code is the primary code for the second operand, for example, a binary operator with two integer operands (where the primary has two double operands).  Entries are added to the expected data type in the add function when:

• A new entry is added to the name to entry map (a new primary code).
• An entry is replaced in the name to entry map (a new primary when the operand count is less than the current primary; and in this case the old entry is removed for an internal function).
• A new secondary primary is added to the alternate code map (a binary operator to a unary operator).
• A replacement alternate primary is found (one that has the same operands, see last post; the old alternate primary entry is removed).
• A new alternate primary is found (the alternate is added for the first operand of a binary operator, otherwise the current entry of the primary is modified).

An add expected data type private support function adds or modifies an entry to the expected data type taking table entry pointer and data type arguments.  If there is currently no entry, a new entry is added.  If there is an entry and its data type is Double or Integer, then the entry is changed to Number (the new data type will be either Integer or Double).  Otherwise, the entry is left unchanged.

The expected data type access function was modified to use the new map.  The expected data type member was removed from the Table Entry structure, and its initializer values were removed from the table entries.  The separate iteration loop in the table constructor to initialize the expected data types was removed.

A problem was found in the set token code function (used to set the code in a token, possibly an alternate code, depending on its data type) where it could incorrectly add a new blank element to the alternate code map.  This did not appear to cause a problem, but was corrected by checking if the code is present before iterating over alternate codes.  This issue was that for a standard map, the bracket operator adds blank elements if the key does not exist.

[branch table commit 1374657d7e]

Table – Other Alternate Codes

There were many alternate codes that couldn't be initialized automatically like with operators and internal functions.  These codes include the assignment, sub-string assignment, internal command (for INPUT and PRINT) and codes with operators.  For now these other alternate codes need to be initialized manually.

An Alternate Information structure was added containing the primary code, the array index for the alternate codes and a initializer list of alternate codes.  An initializer list of these structures was added containing the information for all of these other alternate codes.  After iterating through the list of entries, the constructor iterates through this initializer list to add these other alternate codes to the alternate code map.

The rest of the uses of the associated code arrays were changed to use the alternate code map including the set token code function (used to set the code in a token, possibly an alternate code, depending on its data type), the LET translate function (for setting a sub-string assignment, string keep assignment and list assignment codes), and the INPUT translate function (for setting an input parse code).

The check in the constructor for validating the second associated code index was removed.  The section for setting the expected data type of an operator or internal function was modified to use the alternate code map.  This required a separate entry iteration loop since the alternate code map needs to be initialized completely before looking at the alternate codes.

There was problem with the automatic alternate code map initialization because of the current order of entries.  The issue was that the binary operators with two integer operands was being made an alternate code of the operator with first integer and second double operand.  This was different than how the associated code arrays were initialized.  Instead of moving all of these entries (and their enumerators), a check was added to the alternate map initialization to check for with situation and to swap the entries.

Two additional checks were added to the alternate map initialization to make sure the primary binary operator code has operands with the same data type, throwing an error if not.  In the check if a multiple internal function code with the greater number of operands is before the code with less operands had to also set the Multiple flag.  The code with less operands was already being made the primary code. 

[branch table commit d38f79b8a1]

Table Alternate Codes – Operators/Functions (Use)

With the new alternate codes map implemented and partially filled with all the alternate codes for operators and internal functions, the translator was modified to start using this map instead of the associated codes array.

Two access functions were added to the table class, which included the alternate code and alternate code count functions.  Both take code enumerator and operand index arguments.  These functions have temporary implementations.  When the new table model is fully implemented, these functions won't need the code argument as the this pointer will be used as the key to the map.  They will also return a entry pointer instead of a code enumerator.

The binary operator check for a unary operator in the translator get expression routine was modified to use the new access functions.  In the process internal function routine, the new access function is used to get the alternate code for a function for an operand of a different data type as the primary function code, and when an extra argument is found for a function with multiple forms.

There was an issue with the subtract code table entries.  The current associated code arrays are still being used to process operands of operators because the find code routine is still being used to get associated codes for codes that don't have entries in the new alternate map yet, so couldn't be modified to use the new access functions.

The problem was caused by the change to make the main binary subtract code (two double operands) the second associated code of the negate code, which was made the primary code for minus operator.  This subtract code was moved to after the subtract code with the first integer operand.  Since this code was first, it was made the main binary code and the primary binary alternate to the negate code.  This code did not have the correct associated codes on the current associated code array, so hidden conversion operators were incorrectly added to the output list.

This order of the alternates in the table does not matter with the alternate generation, but for the moment, the new alternate map and the associated code arrays need to agree.  This problem was corrected by moving the main subtract code to before the subtract with first integer operand.  Since the code enumeration is not automatic, the subtract enumerator also had to be moved to match the table.  This is a temporary situation.

[branch table commit 01db8002ba]

Table Alternate Codes – Operators/Functions

The alternate codes map will be implemented in a number of steps including adding alternate codes automatically for operators and internal functions, using the alternate map for operators and functions, removing the associated codes for operators and functions, manually adding alternate codes for internal codes, using the alternate map for internal codes, and adding additional alternate codes (to further reduce the need for code enumerators).

First, the definition for the alternate code static map member was added to the table class along with its instantiation in the table source file.  The standard array is new to the C++11 STL and is just as efficient as a built-in array with improvements.  Since the definition is quite lengthy, it was broken into two definitions (there was no reason to define a constant for the 3 since this is the only place that it is needed; indicating up to three operands or arguments):

using EntryVectorArray = std::array<std::vector<TableEntry *>, 3>;
static std::unordered_map<TableEntry *, EntryVectorArray> s_alternate;

The add function was modified to add an entry as an alternate code if appropriate.  Alternate codes are based on having the same name as another code.  If an entry name is newly added to the name to entry static map, then the routine returns immediately, in other words, the code is a primary code.  If the name is already in this map, has an expression information structure, has operands, and does not have its Reference flag set, then the entry can be added as an alternate code.

If the operand count of the entry is less than the count of the primary code, then it should be the primary code.  In this case, the pointer to the entry replaces the value in the name to entry map, and the previous primary is made an alternate code of the entry, by adding to the array element for its operand count minus one, and returning.  This was a better solution then reporting an error.

The routine does a series of comparisons between the operand data types of the entry with that of the primary to identify which primary code it should be added as an alternate to.  If all the operand data types of the primary code match that of entry and the entry is an internal function with more operands, then it is made an alternate of the primary in the array element one less than the operand count.  This is a multiple argument entry (ASC, INSTR or MID$) so the Multiple flag is set on the primary code.  Otherwise, the entry has duplicate operands as the primary and an error is thrown.

Since the Multiple flag is set automatically, it no longer needs to be specified in the table entry array initialization.  Also since this is automatic, and the requirement that a multiple entry be in the following entry in the table was eliminated, the validation of multiple non-assignment entries was removed from the table constructor.

Using the debugger within QtCreator, the alternate map was verified to be setup correctly.  At this point however, this new alternate map is not being used.  This will be the subject of the next change, which will be to use the alternate map for the operators and internal functions instead of the associated code arrays.

[branch table commit b7021a78ae]

Table – Alternate Codes Map

The handling of alternate codes (formally known as associated codes) will be handled differently in the new table model.  In the current table model, each code in its expression information structure contained a single array of associated codes along with a count and an index to a second set of associated codes within the array.  In the new table model, these will be removed from the expression information structure.

The new table model will contain the information for a code in a single table entry instance, which will be handled by a pointer.  The table class will contain some static data members (members shared by all instances).  The alternate code information will be stored in one of these new static data members, specifically a map from a primary code table entry pointer (the key) to its alternate codes (the value).

The value of this alternate map will contain an array (a standard array will be used) of three elements.  Each element represents the alternate codes for a particular operand.  Generally, the first element (index of 0) will have alternate codes where the data type of the first operand is different from the primary code.  The second element will have alternate codes where the data type of the second operand is different from the primary code.  This was roughly the purpose of the second associated codes.

The third element of the array is applicable only for three argument internal functions, which is new.  There are currently no planned internal functions that have different data types in the third argument.  This third element will be used to associate three argument functions to there primary code with two arguments.  This applies to the MID$ and INSTR functions which have two and three argument versions.  This similarly applies to the ASC function, but its second form has two arguments, so the second element of the array is used.

Each element of this array will contain a vector of alternate code table entry pointers.  A particular element may have an empty vector indicating no alternate codes with different data types for that operand or argument.  The first step will be to automatically generate this map from operator and internal function table entries from the operand data type information.

Table – Code Enumeration Increment Functions

Before the code enumeration can be changed to a C++11 enumeration class, the increment operators need to be removed (though they could be made to work with an enumeration class by using static casts, but this not desirable).  There were only two uses of this increment operator.

One use of the code increment operator was in the assign string recreate function for sub-string assignments, where the name of the function was used to find the original sub-string function code.  If the sub-string function has multiple entries (MID$), then the sub-string code is incremented.  For functions with variable number of arguments (ASC, INSTR, and MID$), the second code with an additional argument followed the first code, which was required for the increment method to work.  The sub-string code was used to recreate the sub-string assignment.  The sub-string code was needed since it had the correct number of operands (the sub-string assignments did not).

The assignment, list assignment and string keep assignment code entries were given one operand for the data type of the value being assigned.  Only this first operand data type was used and it did not matter if there were more operands (the count of operands was not used), so the sub-string assignments were given the same operand data types as the sub-string functions (the sub-string keep assignments already had these operands).  With the correct operands, the sub-string assignments assignments can be recreated directly without having to look up the sub-string function code, eliminating the need for the code increment operator.

The set token code function is used to set the correct code for the operand data type.  One use is for operators to set the correct code for the data type of its operand.  The second associated codes  are used for the second operand.  A negative second associated code index indicates no second associated codes.  A negative index was only used by the sub-string functions to prevent it from using any of its associated codes).  The set token code function is no longer called for functions, so this check was unnecessary.  The negative index was removed from the sub-string code entries.

The other use of the code increment operator was in the translator process internal function to move to the next code a function with variable number of operands (ASC, INSTR, and MID$mentioned above).  The token next code access function was used to increment the code.  The second code of these functions were associated to the first code, so now when a comma is processed for the next argument instead of a closing parentheses, the second code is obtained by getting the associated code of the first code.  The next code function was removed along with the code increment operator functions.

[branch table commit e5b55fa271]

Table – Code Enumeration

With the new table model, the code enumeration will be a subset of the current code enumeration and will only include codes that are referenced (for example, Comma, Equal, Semicolon, Open Parentheses, Closing Parentheses, etc.).  With the bracketing codes removed, all the remaining code enumerators represent actual codes.  The current auto-generated code enumeration was copied to the main header file.

Temporarily, the code enumeration definition must match the table entry array and there are no checks to insure this (which was the purpose of auto-generating the code enumeration).  Since the code enumerators are still used as indexes, this enumeration was not changed to a C++11 enumeration class yet.  There are also some in-line functions for incrementing a code enumerator, which is possible since plain enumerators can be used as indexes, but more difficult with an enumeration class.  Notes were added to which enumerators will be removed.

The enumerations awk script was removed.  The CMake build file was modified to remove the auto-generation of the code enumeration header file.  Since the awk program is no longer used, looking for this program was also removed.  The next goal will be remove the use of code enumerators as indexes so that this enumeration can be changed an enumeration class.

[branch table commit 9a3c075ba0]