Saturday, September 14, 2013

Translator Token Code Transition (Begin)

The translator will be transitioned to assign token codes to token types without a code as soon as possible.  This includes the constant, identifiers with and without parentheses, and defined functions with and without parentheses token types.  The handling of the defined functions types will be delayed if possible until define functions are implemented (much later).

Development of this transition will be done on the tokenCode0.5 sub-branch in case the changes do not work out and need to be abandoned, otherwise when complete, the main branch0.5 branch will be fast-forwarded to this sub-branch (which can then be deleted).

The first change was to give all tokens a code when they are created by the parser.  Since the mentioned token types do not have a code, these will be assigned to the Invalid code value.  This was accomplished by initializing the code to Invalid in the token constructor.  As currently implemented, when the parser finds a command, operator, or internal function, the token is then assigned a valid code.

Since all tokens now have a code (some with the Invalid code), the token code can now be used to determine if the token has a table entry (it doesn't if the code is invalid).  The has table entry access function is now longer needed and was removed along with the static table array indexed by token type that was used to determine if the token had a table entry.  A new has valid code access function was added in its place.  Users of the has table entry access function were updated to use the token code (see the commit for the specifics).

[commit 57d65135de]