New Translator – Remaining Issues

There are at least two major issues remaining in the new translator routines that are impacting the failures in the other LET tests, though test #10 (Expression Errors) passes.  The first issue is how operators are detected, specifically that some tokens that are considered operators, but are not expression operators (for example, open and closing parentheses, commas, semicolons, colons, the remark operator and end-of-line tokens).
The second issue is part of the design of the new translator has not yet been realized, namely that when getting an expression, if given a particular data type, it should check that the expression is of that type, or can be converted to that type via a hidden conversion code, else return an error.  Currently the data type is only being used for reporting errors with operands.  Correcting and implementing these issues has been a major undertaking, so a bunch of preliminary changes were made leading up to these changes.

Previously, all tokens with a command or operator type were considered operators.  This was necessary for the token-centric old translator, because some commands like THEN and ELSE needed to be considered operators since they can come at the end of expressions.  The first change made was to remove the Token::isOperator() access function along with the static Token::s_op[] array used by the access function.  Uses of the access function were replaced with the Token::isType() access function.  Not all uses required checking for both operator and command token types.  [commit ffd1ba6462]

Since the number of old translator expected results files are growing (because of corrections and changes to translations), the regtest script was modified to look for an old expected results file (ending with an 'o') and comparing to that file if found instead.  An old expected results file for translator test #3 was also added (print functions used in expressions do not report through the closing parentheses.  [commit 7aef54c63e]

The fact that the open parentheses operator token was configured as a unary operator in the table was going to cause issues with the new translator routines.  Therefore, it is now not configured as a unary operand and the old translator routine was modified to look for an open parentheses token before checking if the operator was unary (the check was simply moved from after to before).  [commit e75adeddcb]

The segmentation fault on expression test #3 was becoming a nuisance, and was only caused by the additional of more error tests for the new translator.  Therefore, the problems with old translator routines were corrected, including considering the initial state as an operand state, allowing for an empty command stack in the case of expression mode, and assuming the Any type at the beginning of an expression before any operands or unary operators are received.  The memtest script was also updated to allow for comparing old translator expected results files.  [commit f0f6cb57f7]