REM Command Correction

The operands of program instructions will hold indexes to dictionary entries that will contain the text of the instruction (for example, variable names, the original strings of constants, the strings of remarks, etc.).  The remark dictionary will be implemented first since it will just hold the strings of the comments.  The other dictionaries will required looking up strings to find if a variable or constant already exists.

But first, a problem was discovered with how REM statements are parsed.  The REM command should be recognized regardless of what characters follow the command.  The parser for the most part did this already except that a space was required after the command.  However, the parser should not require the space, consider these examples:

REMARK this should be a valid commented
REM       any number of spaces should be allowed

The first statement was rejected because it was assumed to be an assignment of the REMARK variable and expected an equals instead of this.  The second statement was valid but all the spaces were removed from the comment string.

The parser get identifier was modified to first look for a statement starting with the three characters R-E-M and store all the characters after this before scanning for a word (valid identifier characters up to a invalid identifier character).  Some minor code simplification was also done in replacing the sequence of setting the token code, type and data type with a call to the existing set token table routine that performed these steps.  Two additional statements were added to translator test #15 (remark tests) similar to the two examples above.

[commit c52d65a479]