Encoder – Assigning Positional Indexes

The next step in preparing the tokens in the RPN list for encoding is assigning a position index to each token.  This is a position, or offset, that the token will have in the encoded program code within the line.  This index will be used later for calculating offsets of a single line statements like from the IF token to the ELSE, END IF or last token on a line.

For now however, this index is not needed, but after assigning the indexes to all the tokens, the final count or size of the encoded program code for the line will be known.  This value will be used for allocating a program word array that will be filled in during the next step of encoding.

It was more efficient to assign the position index after assigning codes to tokens instead of having a separate routine with another loop, the assign codes routine was renamed to the prepare tokens routine.  A count variable was added to the routine, which is initialized to zero.  After the token type switch, the index of the token is set to the count and the count is incremented.  If the token will have an an operand (the token code has the has operand flag), the count is incremented again for the operand word.

The prepare tokens routine was also changed to return the size required for the encoder program line instead of a success/fail boolean status.  For errors, a -1 value is returned.  The calling encode routine was updated accordingly for the new return value.

The RPN list, item and token text routines were modified to optionally output the index of each token.  The token text routine was also modified to output an index for the operand word, to make sure that the sub-codes are output after the code word (not after the operand word), and to treat the comment of remark as the operand word (when index output is selected).  Another test line with multiple statements was added to encoder test #1 and the expected results were updated for the indexes now output on each token.

[commit 9e0e442171]