Saturday, August 10, 2013

Old Translator Issues With Token Caching

While debugging the new PRINT translate routine, I realized that the old translator routines were never tested with new token caching and error reporting code, because when they were, crashes and errors occurred.  Two small problems were found and were corrected.

The first issue occurred when an error is detected and the error token was not the current token obtained from the parser (meaning the error token had already been added to the output and cannot be deleted).  The old translator routine obtained a new token and copied the error token into it before calling the set error token routine, which called the output list set error routine and then deleted the token.  When the token was copied into the new token obtained, the index of the original token was copied over the new token's index confusing the used token handling.  This was corrected by calling the output list set error routine directly (which only grabs the column and length of the error token) so no temporary token is needed.

The second issue occurred when there was a REMARK operator (') at the beginning of a line.  The Null token put on the bottom of the hold stack was not removed.  This was detected as a token leak and the detection code reported and deleted the token.  Unfortunately, the translator still had the token on the hold stack.  This did not cause a problem until the next command with an error was processed.  When an error occurs, the the hold stack is empty and its tokens are deleted.  When it got to the deleted Null token, the index was garbage (because the memory for the token had been reused) and a segmentation fault occurred.

The REMARK operator token handler was modified to pop and delete the Null token from the hold stack if currently in command mode.  If not in command mode, then another part of the translator correctly handled the the Null token.  Curiously for this situation, the EOL token at the end of the line is never obtained and processed.  Therefore, this fix is a hack, but since the old translator routines will be abandoned, there is no reason for a correct fix.

One last minor issue was that when token leaks or extra deletes were reported, if the token was a Null or End-of-line token, a blank token was output with just an index.  The "NULL" and "EOL" strings were added to the debug name entry in the table to identify these blank tokens.

[commit cf93288fcb]