Parser – Token Creation

A side effect of the last change was that tokens for both codes for operators, functions and commands and codes with operands (constants, variables, arrays, defined functions and user functions) were using the same token constructor.  This token constructor searched through alternate codes for the code with the appropriate return data type.

This was unnecessary for operator, function and command codes.  The table new token function called for these codes passed in the return data type from the table entry of the code.  The token constructor then called the new table set token code function.  Since the data type matched the return data type (which was just passed in), no alternates were checked and the code, type and data type of the token was set.  This was extra unnecessary work.

A new token constructor was added for operator, function and command codes, which only required arguments for the code, column, length and string.  The string argument is only used for the REM and REM operator codes.  This constructor replaces the table new token function.  This constructor calls the table set code function which just sets the code, type and data type of the token from the table entry of the code.  For consistency the code argument was put first in the other token constructor for codes with operands.

While looking at the creation of tokens, I decided that using the standard unique pointer within the parser was unnecessary.  The parser can just allocate a token and return its pointer.  The translator then can put the allocated tokens into a standard shared pointer.  The parser was changed to use plain token pointers.  The translator routines were changed to use the new token constructor directly via the standard make shared function.  The translator get operand was changed to use the reset function to set the token member since shared pointers cannot be assigned directly to a pointer.

[branch table commit 1bfb76ae0a]