Table – New Token Consolidation

The next major change to the table class will be to start transitioning to using table entry pointers instead of code enumerators and to remove the use of code enumerators as indexes.  The code and index values will be separate members of table entries.  All table entries will have an index (which is put into the program code), but only a few table entries will have a code (only those the require specific lookup like some of the special symbols including comma, parentheses, colon, etc.).  Before proceeding with this, a small simplification was made first.

There were two new token functions in the table class, one taking a single code argument and the other taking column, length and code arguments.  The single argument version relied on the default token constructor.  This was one of three uses of the default token constructor.  The default token constructor contained optional column and length arguments (default to -1 indicating unset), but there were no callers of the default token constructor that used these arguments.

The single code argument new token function was removed along with the default token constructor.  The code argument was made the first argument of the other version of the new token function with default -1 values provided for the optional column and length arguments.  This second version does not use the default token constructor.  All callers of this function were in the parser and were modified for the reordering of the arguments.

The second use of the default token constructor was by the decode function in the program model class to create a default token, which it then used the token set code access function to set the code.  This function was changed to use the new token function.

The third use of the default token constructor was in the INPUT translate function where a new token is needed for an input assign code and another token (comma or semicolon) is not available for reuse.  This was changed to use the new token function with a Null code.

[branch table commit b5dd96c272]