Translator – End Statement State

Once a semicolon is received at the end of an INPUT statement, no more tokens should be received except for an end-of-statement token. If another token is received, then an error should be reported. To accomplish this, a new end statement state similar to the end expression state is needed. While the end expression state only needs to be checked when operators are expected, the end statement state needs to be checked for all tokens.

The end statement state check was added just before the check for operand or first operand state in the main translator add token routine. When in end statement state, if the token does not have the end statement flag set in its table entry, then an “expected end-of-statement” error is reported against the token.

The access function for getting the table entry flags for a token already checks if the token has a table entry, and if there is not table entry, then zero (no flags) is returned. Currently only the EOL code has the end statement flag set, but eventually the Colon, ELSE and ENDIF tokens will also and possibly other codes.

Translation – Define Function Token Issues

For now in the process operand routine, define function with parentheses tokens will not be allowed in command or assignment mode. This will need to be changed later when the DEF command is implemented. This check was also made in the check assignment list item routine.

However, since a define function without a parentheses token is allowed in assignments, the error was set to point to the open parentheses as an "expected equal or comma for assignment" error. The open parentheses is at the end of the token, so to get the error to point to it, the column of the token was incremented by the length of the token minus one.

Previously in the close parentheses token handler, the reference flag was being set for token with parentheses and define function with parentheses tokens. This check was modified to only set the reference flag for token with parentheses.

The reference flag for define function without parentheses tokens was already being set, so no change was needed for these tokens.

Language – Define Functions

Before defining what needs to be done with define function tokens in the Translator, a quick review of their syntax is required. The will be two forms of define functions that will be supported, a single line and a multiple line. An example of a single line define function would be:

DEF FNHypot(X,Y)=SQR(X*X+Y*Y)

Notice that this form has the same format as an assignment except for the DEF command at the beginning. The define function token in this statement is FNHypot( - in other words, a define function with parentheses. This implies that a define function with parentheses token could appear in assignment mode (assuming the DEF command sets this mode), but only for the DEF command. An example of the multiple line form of the same function would be:

DEF FNHypot(X,Y)
    FNHypot=SQR(X*X+Y*Y)
END DEF

The assignment of the define function name (a define function without a parentheses) returns the value for the function. Here a define function without a parentheses can appear in an assignment statement, but only inside a DEF/END DEF block. Since the Translator is not aware of blocks, it will permit an assignment of define function without a parentheses token. It will be the Encoder's job to verify if the assignment is valid.

Translation – Assignment Token Issues

Finally, some issues were discovered when making the change from command mode to assignment after processing an operand token. There are two main types of operand tokens, ones with parentheses and ones without parentheses.

The operand tokens with parentheses include internal functions, defined user functions (DEF FN) and generic tokens with parentheses (which can be arrays or user functions). Internal functions were already invalid for command or assignment mode except for sub-string functions. A check was added for defined functions with parentheses.

The operand tokens with no parentheses include internal functions with no arguments (currently only RND), constants, define user functions with no arguments, and generic tokens with no parentheses (which can be variables or user functions with no arguments). Internal functions and constants were already invalid because they didn't have the reference flag set when the comma or equal token looked for the reference flag. That left define function tokens...

Translation – Command Token Issues

The last problem statements related to unexpected command tokens were:

A PRINT B
MID$(A$ PRINT,4)=""

The first statement gave an “expected operator or end-of-statement” error at the B token. The second statement was actually accepted, but with a strange translation. The problems were caused because when the PRINT command token was received, it was immediately pushed to the command stack because the mode was still set to command.

Both statements start as assignment statements, but assignment mode was not being set (unless preceded by the LET keyword). When an equal token is received expression mode is set, or when a comma token is received assignment list mode is set. The “unexpected command” error was only occurring when the mode was not set to command, which didn't occur with the statements above. Also, this message again does fit with the “expected ...” type of message.

Once a command token is received in command mode, the mode is set according to the token mode in the command's table entry. A change was made to the process operand routine that once an operand token is processed, if in command mode, the operand is assumed the beginning of an assignment statement and so the mode is changed to assignment.

To remove the “unexpected command” error and report a more appropriate error, the command token has to be passed through the rest of the Translator. This will occur when the Translator is not in command mode. So the main add token routine was changed to not report this error if a command token is received and the mode is not command.

Command tokens received in operand state were already being reported correctly since commands are also considered operators, which are not valid operands (unless the operator is a unary operator, which commands are not). Commands are considered operators because some commands can be found where an operator is expected, for example, THEN and ELSE.

Commands tokens received in operator state are only valid if they have a token handler. In the process operator routine, when an operator token does not have a token handler a default operator token handler is called. Before the default operator token handler is called, a check was added to return an appropriate error if the token is a command.

Translation – Other Errors

After changing the “item cannot be assigned” to “expecting item for assignment” error, there were several other errors that didn't fit the “expecting ...” type of message. It turned out that most of these were not actually being used, so there were removed.

Another remaining message was the “missing open parentheses” error that occurs when there is a  parentheses with no open parentheses, function or array. After some consideration of possibly leaving this message as is, it was decided to change this to an “expected operator or end-of-expression” error since the problem could also be a missing function or array, or even that the open parentheses was just a mistake.

Again assuming that everything is correct up to the problem, this change seemed appropriate, and “...expression” was used instead of “...statement” because the next token could be a comma or semicolon in a PRINT statement or a THEN in an IF statement.

The last message was the “unexpected command” error that occurs when there is a command token when not in command mode. However, there were a number of additional problems with command tokens received when not expected...

Translation – Parentheses Issue

The next problem statement was:

MID$((A$),4)=""

This was reported as an “item cannot be assigned” error and then crashed. Again, this error didn't fit the “expected ...” messages. This error is also returned for statements like 3=A and 1,A=B and was renamed to the “expecting item for assignment” error. For the statement above, the “expecting string variable” error should be returned.

The crash occurred because the open parentheses token was returned for the error with its range extended to the closing parentheses to report the entire (A$). The caller deleted the error token since it was an open parentheses to prevent a memory leak. However, in this case, the A$ token was still on top of the done stack with the open parentheses attached as the first operand. When the Translator clean up routine (called upon an error) was emptying the done stack, it deletes each item's first and last operand – the open parentheses was getting deleted twice causing the crash.

Initially to fix this problem, when an error occurs and the first through last operand is returned, the first operand pointer for the item on the done stack was set to null to prevent it from being deleted a second time. While this fix was sufficient for the statement above, this statement still were not being reported correctly:

MID$(-A$,4)=""

The error was “expected numeric expression” pointing to the A$. Both the open parentheses and the minus are initially processed in the process unary operator routine. So a check was added to this routine to return an error if there is a sub-string function on top of the hold stack with its reference flag set (sub-string assignment) and it is as the first operand. Both of theses statements then correctly reported “expected string variable” at the open parentheses and minus. The initial fix was not necessary since the error was being caught sooner.

Translation – Another Print Function Issue

While the new reference mode was being implemented, some problems were discovered in the code that carefully constructed statements would exploit giving incorrect results. The first problem statement was:

TAB(10)=A

This caused a “command stack empty for expression” bug error. This should have been “expecting command” error. The check was if the command stack was empty or there was a PRINT command on top of the command stack or if the top of the hold stack was not empty (except for the null token).

The error occurred when trying to get the type of expression because the hold stack was empty (first item to figure out the expression type) so it when to the command stack, which was also empty. This check should not have caught this because there was a check after this with for catching all internal functions in command mode. The check was modified to if the command stack is not empty and if there is a PRINT on top of the command stack or the hold stack is not empty.

Translator – Reference Mode

A new token mode is needed for the INPUT commands. The current token modes are command, assignment, assignment list and expression. Consider these invalid INPUT statements:

INPUT A+B*C
INPUT A+B*C$

The INPUT commands must have variables, not expressions. If expression mode was used, at each comma and the end of the statement, the INPUT command handler would need to check the token on top of the done stack to see if its reference flag is set. The first statement above, the INPUT command handler will see the multiplication token on top of the stack. Its first token will be A and last token will be C. An “expecting variable” error would be reported pointing to the whole A+B*C expression. This would be acceptable.

However, in the second statement, an error occurs before the INPUT command handler gets a chance to report an error. When the multiplication token checks its second operand, it will report an “expecting numeric expression” error pointing at the C$ token. This would make no sense. The proper error for both of these statements would be an “expecting semicolon, comma or end-of-statement” error pointing at the add token.

The new reference mode will only accept variables and array elements, specifically tokens with no parentheses and tokens with parentheses. If these tokens turn out to be user functions, which the Translator cannot determine, the error will be reported by the Encoder. Reference mode will be set when the INPUT command is received and after the comma or semicolon of the prompt string expression of the INPUT PROMPT command. It will also be used later for the READ command.

In reference mode, internal functions, define functions, and unary operators will be reported as invalid (“expecting variable” error). After the variable or array element token is pushed to the done stack, the state will be set to end expression so that only end expression tokens are valid. Binary operators will then be correctly be reported as invalid.

While reference mode was being implemented, some more problems were found in the Translator code...

Translation – PRINT Function Problem

Due to the problems found with the print codes, some additional error tests were added to translator test 10 (PRINT statement tests) including these statements:

PRINT (TAB(10))
PRINT INT(TAB(10))
PRINT A(TAB(10))
PRINT A+TAB(10)
PRINT TAB(10)+A

The first three test the situation of a print function inside a parentheses, internal function and an array or user function, which were caught by adding count stack is not empty check when a print function token is received.

In the fourth statement, the error can be caught by checking if the hold stack is not empty (an empty hold stack has only a null token). In this case, an operator will be on top of the hold stack. In fact, this check can replace the count stack is not empty check because the open parentheses, internal function and array/user function will be on top of the hold stack.

The last statement was more difficult to check for - the expression should end after the print function. When the operator was received and checked its operand, a bug occurred because the done stack is empty since the print function didn't get pushed to the done stack. The error should be “expected semicolon, comma or end-of-statement” and point to the operator.

To catch this error, a new end expression state was added. The closing parentheses is received when in binary operator state. After the closing parentheses, the state was left at binary operator since another binary operator is expected (an end of expression token is acceptable as a binary operator). For the end expression state, only operators with the end expression flag are acceptable, which currently include the semicolon, comma and end-of-line tokens – other operators will generate the error.

In resolving these issues, the “invalid used of print function” error did not match the other “expecting...” errors (remember the goal is to help the user by suggesting what is expected at the location of an error). Therefore, this error was changed to be one of the “expecting xxx expression” depending on the current expression type (xxx would be blank, numeric or string).

Translation – PRINT Code Issues

The process final operand currently doesn't push print codes to the done stack. This needs to be expanded to include the input begin prompt codes. It would be inconvenient not to keep expanding this test to include additional codes that don't need to be pushed to the done stack. The check could be changed to see if the code does not have a return data type (it is set to none). This would include the print codes, the input begin codes, the input parse type codes, the input assign type codes, and probably many more.

However, when this change was made, it did not work for the TAB and SPC print functions because the token data type for these had been incorrectly set to double. This occurred in the set default data type function when the token was received. This function was fixed to not set the data type for internal functions (these are set from their table entry data type).

Now when the token's data type is none, it won't be pushed to the done stack. The check if the command on top of the command stack is the PRINT command only applies to print functions, but is not necessary here as this check was already made when the print function was first received, so the check was removed. The check if the token's has the print flag remains (to set the stay and print function command flags, which is used by the print command handler to determine if the final print code should be appended to the output).

For the print type codes, when process final operand routine is called by the add print code routine, the second token passed is a null, not a closing parentheses. The process final operand was deleting the second token for print functions assuming that it was a closing parentheses (which only applies to TAB and SPC). For some reason, doing a delete with a null argument was not causing a problem. None the less, a check was added to only delete the second token if it is not a null.

Another problem was discovered for print functions. The situation if a TAB or SPC was contained within parentheses, internal function, or and array/user function, was not caught since it was only checking if there was a print command. Therefore, an additional check was added to make sure the count stack is also empty.

INPUT Translation – Some Issues

As the INPUT command handler was being implemented, some issues were found. When the new element pointer was added to the command item, it was noticed that there was a code member. This member is no longer needed because the token now contains the code (replacing the index member, through the code is now an index), so the code member was removed from the command item.

There was no table entry for the two word INPUT PROMPT command, so one was added along with the three input begin codes. It was noticed that some table entries still had the string flag set. This is not necessary because the string flag is now set automatically during table initialization it there are any string arguments, so these string flags were removed.

To look up which input begin code to append for the INPUT PROMPT command, the process final operand routine will be called, which in turn will call the find code routine that will pick the correct code based on the type (string or temporary string) that is on the done stack. The input begin codes will not push anything to the done stack (accomplished by setting the done push flag to false). Some additional issues were found in these routines...

INPUT Translation – Variables

Processing variables is a bit more complicated. With the PRINT statement, the appropriate print value type code was simply appended to the output. However, with the INPUT statement, an input parse code needs to be inserted after the begin code or after last parse code (before all of the variables), and an input assign code needs to be appended to the end of the output (after the variable).

There needs to be a way to point to the location where the input parse codes are to be inserted. This will be accomplished with a new output list element pointer member to the command stack item. This pointer will be initialized to null when a new command token is pushed to the command stack.

When an input begin code is appended to the output, this new element pointer will be set to the input begin code element. This pointer can also be used to indicate if any input variables have been received yet (when it does not contain a null).

To insert a input parse code, the input parse token will be appended to the list at (after) this element pointer. The element pointer will then be set to the output list element of the input parse token just inserted so that the next input parse token will be inserted after this input parse token.

INPUT Translation – Beginning

Execution of the INPUT command has been defined and therefore the form of the translation, the actual process of the translation can now be defined. The translation begins with one of the InputBegin codes, which will be triggered by a comma or semicolon token. The INPUT or INPUT PROMPT token will already be on top of the command stack.

For the INPUT command, when the first comma, a semicolon or an end-of-line token is received, an InputBegin token will be appended to the output. The token received can be converted to the InputBegin token (more efficient to change the token than to delete the token not needed and then create a new one).

For the INPUT PROMPT command, when a comma or semicolon token is received, there must be a string on top of the done stack (the prompt string expression). Depending on whether this string is temporary or not will determine whether an InputBeginStr or InputBeginTmp will be appended to the output. For InputBeginStr, the string will be attached since the translator will not know if it is a variable, array or a user function. The token received can be converted to this token. An end-of-line token at this point would produce an “expected comma or semicolon” error.

Automatic Code Enumeration Generation

To have the Code enumeration generated automatically from the table entries, the table source was structured so the awk scripts can read it. Since the Code enumeration value will now be the same as the table entry index, the code member of the table entry is not necessary and was removed. The code name initializers in the table entries was moved to a comment on the line of the entries' open brace.

The awk scripts were rewritten to read the table source file instead of the main include file. The awk script were also changed to read the table source file directly and write the output files directly. This eliminates the requirement to redirect the input and output to the correct files when running the awk scripts. Logic was also added to the codes awk script to check for duplicate code names.

The code to index conversion array that was initialized in the table class constructor, along with the check for duplicate and missing codes, was removed. The code and index access functions in the table class were also removed. The token class index member was replaced with a code member. All the code was updated to use the code enumeration value instead of the index, though the code will be used as an index.

One problem with using an enumeration value instead of an integer index is that normal math functions cannot be used, like the add and increment operators. These operators are needed, so operator functions were created for the code enumeration, which includes the add, prefix increment and postfix increment operators. These functions type cast to integer to add and then type cast back to the code enumeration value.

The null code entry at the end of the table was moved to the beginning so that that null code enumeration value (index) would be zero. The table search function for searching for an immediate command previously assumed that the immediate commands were at the beginning of the table entry array. Moving the null code to be the beginning of the array complicated this. Therefore, immediate command bracketing codes were put around these entries for this search function.

Due to the these table entry changes, the parser test output files were updated since all the code indexes changed. This would be a good time to make another pre-release, but since there have been no download activity for recent pre-releases, there will not be a pre-release at this time. Now the translation of the INPUT command can begin...

Code Enumeration vs. Table Entry Indexes

While designing the error handling mechanism for the INPUT command, a thought occurred related to program codes. For efficient program execution at run-time, the index of the table entry will be stored in the internal program code, not the code enumeration value.

If the code enumeration value was used, the index for the table entry (needed to get the run-time handler function pointer) would first need to be converted to an index by going through the code to index array setup during table initialization. The intention all along has been to use table entry indexes in the internal program code.

For the INPUT command's error recovery, when it is backing up execution and checking for input parse codes, it will need to convert the table entry indexes to a code before it can check if it is an input parse code. This would not efficient during program execution. Even though for the INPUT command, execution time is not critical since it is about to stop and wait for user input. A few extra program cycles won't matter much. But this problem could occur for other more critical commands.

Therefore, it is desirable if the code enumeration values were the same as the table entry indexes. One simple solution is to make sure the code enumeration values matched the table entries. Unfortunately this relies on the programmer to keep the two in sync, is very error prone and is just a general pain to begin with.

There is a better way where the code enumeration is generated automatically from the table entries using an awk script. This method would be similar to how the test_codes.h file (used by the test_ibcp.cpp source file) is generated automatically by scanning for codes in the ibcp.h file.