Translator – Token Sub-Codes (Implementation)

The sub-code flag was implemented, which consisted of adding the sub-code memory to the Token class, adding the sub-code flag value definitions and modifying the print token test routine to output the flags.

The setting of the parentheses sub-code flag was handled in the do pending parentheses function, which checks if unnecessary parentheses were entered and appended a dummy parentheses token. The code was changed to set the parentheses sub-code flag of the last  token appended to the output. Two issues were discovered.

The first issue found was if two sets of unnecessary parentheses are entered, for example, A=((B)), then the parentheses sub-code flag can only be set once. Upon reproducing the original source, the Recreator will not know that there were two sets of unnecessary parentheses. So for this case, a dummy parentheses token will still be appending for each additional set of unnecessary parentheses entered.

Curiously, with this change, if three (or and odd number of) unnecessary sets of parentheses are entered, for the third (fifth, etc.) set, the parentheses sub-code flag gets set in the second (fourth, etc.) set's dummy parentheses token. Something the Recreator will need to handle.

The second issue found was if the last token appended was a hidden conversion code, the conversion code's token parentheses sub-code flag gets set. It is anticipated that this would cause a problem for the Recreator. The Recreator should be able to safely ignore the conversion codes, but if it needs to look for sub-code flags, these can't be ignored. To avoid this, the code was modified that if token's table entry has the new Hidden flag set, then the item previous to the conversion code has its parentheses sub-code set.

The setting of the comma sub-code flag was handled in the new equal token handler function. When an equal token is received, if the mode was a multiple comma assignment, then the comma sub-code flag is set when the token's code is set to the assign list operator. The comma sub-code flag will not be set of the mode was a multiple equal assignment.