Translation – Parentheses Issue

The next problem statement was:

MID$((A$),4)=""

This was reported as an “item cannot be assigned” error and then crashed. Again, this error didn't fit the “expected ...” messages. This error is also returned for statements like 3=A and 1,A=B and was renamed to the “expecting item for assignment” error. For the statement above, the “expecting string variable” error should be returned.

The crash occurred because the open parentheses token was returned for the error with its range extended to the closing parentheses to report the entire (A$). The caller deleted the error token since it was an open parentheses to prevent a memory leak. However, in this case, the A$ token was still on top of the done stack with the open parentheses attached as the first operand. When the Translator clean up routine (called upon an error) was emptying the done stack, it deletes each item's first and last operand – the open parentheses was getting deleted twice causing the crash.

Initially to fix this problem, when an error occurs and the first through last operand is returned, the first operand pointer for the item on the done stack was set to null to prevent it from being deleted a second time. While this fix was sufficient for the statement above, this statement still were not being reported correctly:

MID$(-A$,4)=""

The error was “expected numeric expression” pointing to the A$. Both the open parentheses and the minus are initially processed in the process unary operator routine. So a check was added to this routine to return an error if there is a sub-string function on top of the hold stack with its reference flag set (sub-string assignment) and it is as the first operand. Both of theses statements then correctly reported “expected string variable” at the open parentheses and minus. The initial fix was not necessary since the error was being caught sooner.

Translation – Another Print Function Issue

While the new reference mode was being implemented, some problems were discovered in the code that carefully constructed statements would exploit giving incorrect results. The first problem statement was:

TAB(10)=A

This caused a “command stack empty for expression” bug error. This should have been “expecting command” error. The check was if the command stack was empty or there was a PRINT command on top of the command stack or if the top of the hold stack was not empty (except for the null token).

The error occurred when trying to get the type of expression because the hold stack was empty (first item to figure out the expression type) so it when to the command stack, which was also empty. This check should not have caught this because there was a check after this with for catching all internal functions in command mode. The check was modified to if the command stack is not empty and if there is a PRINT on top of the command stack or the hold stack is not empty.

Translator – Reference Mode

A new token mode is needed for the INPUT commands. The current token modes are command, assignment, assignment list and expression. Consider these invalid INPUT statements:

INPUT A+B*C
INPUT A+B*C$

The INPUT commands must have variables, not expressions. If expression mode was used, at each comma and the end of the statement, the INPUT command handler would need to check the token on top of the done stack to see if its reference flag is set. The first statement above, the INPUT command handler will see the multiplication token on top of the stack. Its first token will be A and last token will be C. An “expecting variable” error would be reported pointing to the whole A+B*C expression. This would be acceptable.

However, in the second statement, an error occurs before the INPUT command handler gets a chance to report an error. When the multiplication token checks its second operand, it will report an “expecting numeric expression” error pointing at the C$ token. This would make no sense. The proper error for both of these statements would be an “expecting semicolon, comma or end-of-statement” error pointing at the add token.

The new reference mode will only accept variables and array elements, specifically tokens with no parentheses and tokens with parentheses. If these tokens turn out to be user functions, which the Translator cannot determine, the error will be reported by the Encoder. Reference mode will be set when the INPUT command is received and after the comma or semicolon of the prompt string expression of the INPUT PROMPT command. It will also be used later for the READ command.

In reference mode, internal functions, define functions, and unary operators will be reported as invalid (“expecting variable” error). After the variable or array element token is pushed to the done stack, the state will be set to end expression so that only end expression tokens are valid. Binary operators will then be correctly be reported as invalid.

While reference mode was being implemented, some more problems were found in the Translator code...

Translation – PRINT Function Problem

Due to the problems found with the print codes, some additional error tests were added to translator test 10 (PRINT statement tests) including these statements:

PRINT (TAB(10))
PRINT INT(TAB(10))
PRINT A(TAB(10))
PRINT A+TAB(10)
PRINT TAB(10)+A

The first three test the situation of a print function inside a parentheses, internal function and an array or user function, which were caught by adding count stack is not empty check when a print function token is received.

In the fourth statement, the error can be caught by checking if the hold stack is not empty (an empty hold stack has only a null token). In this case, an operator will be on top of the hold stack. In fact, this check can replace the count stack is not empty check because the open parentheses, internal function and array/user function will be on top of the hold stack.

The last statement was more difficult to check for - the expression should end after the print function. When the operator was received and checked its operand, a bug occurred because the done stack is empty since the print function didn't get pushed to the done stack. The error should be “expected semicolon, comma or end-of-statement” and point to the operator.

To catch this error, a new end expression state was added. The closing parentheses is received when in binary operator state. After the closing parentheses, the state was left at binary operator since another binary operator is expected (an end of expression token is acceptable as a binary operator). For the end expression state, only operators with the end expression flag are acceptable, which currently include the semicolon, comma and end-of-line tokens – other operators will generate the error.

In resolving these issues, the “invalid used of print function” error did not match the other “expecting...” errors (remember the goal is to help the user by suggesting what is expected at the location of an error). Therefore, this error was changed to be one of the “expecting xxx expression” depending on the current expression type (xxx would be blank, numeric or string).

Translation – PRINT Code Issues

The process final operand currently doesn't push print codes to the done stack. This needs to be expanded to include the input begin prompt codes. It would be inconvenient not to keep expanding this test to include additional codes that don't need to be pushed to the done stack. The check could be changed to see if the code does not have a return data type (it is set to none). This would include the print codes, the input begin codes, the input parse type codes, the input assign type codes, and probably many more.

However, when this change was made, it did not work for the TAB and SPC print functions because the token data type for these had been incorrectly set to double. This occurred in the set default data type function when the token was received. This function was fixed to not set the data type for internal functions (these are set from their table entry data type).

Now when the token's data type is none, it won't be pushed to the done stack. The check if the command on top of the command stack is the PRINT command only applies to print functions, but is not necessary here as this check was already made when the print function was first received, so the check was removed. The check if the token's has the print flag remains (to set the stay and print function command flags, which is used by the print command handler to determine if the final print code should be appended to the output).

For the print type codes, when process final operand routine is called by the add print code routine, the second token passed is a null, not a closing parentheses. The process final operand was deleting the second token for print functions assuming that it was a closing parentheses (which only applies to TAB and SPC). For some reason, doing a delete with a null argument was not causing a problem. None the less, a check was added to only delete the second token if it is not a null.

Another problem was discovered for print functions. The situation if a TAB or SPC was contained within parentheses, internal function, or and array/user function, was not caught since it was only checking if there was a print command. Therefore, an additional check was added to make sure the count stack is also empty.

INPUT Translation – Some Issues

As the INPUT command handler was being implemented, some issues were found. When the new element pointer was added to the command item, it was noticed that there was a code member. This member is no longer needed because the token now contains the code (replacing the index member, through the code is now an index), so the code member was removed from the command item.

There was no table entry for the two word INPUT PROMPT command, so one was added along with the three input begin codes. It was noticed that some table entries still had the string flag set. This is not necessary because the string flag is now set automatically during table initialization it there are any string arguments, so these string flags were removed.

To look up which input begin code to append for the INPUT PROMPT command, the process final operand routine will be called, which in turn will call the find code routine that will pick the correct code based on the type (string or temporary string) that is on the done stack. The input begin codes will not push anything to the done stack (accomplished by setting the done push flag to false). Some additional issues were found in these routines...

INPUT Translation – Variables

Processing variables is a bit more complicated. With the PRINT statement, the appropriate print value type code was simply appended to the output. However, with the INPUT statement, an input parse code needs to be inserted after the begin code or after last parse code (before all of the variables), and an input assign code needs to be appended to the end of the output (after the variable).

There needs to be a way to point to the location where the input parse codes are to be inserted. This will be accomplished with a new output list element pointer member to the command stack item. This pointer will be initialized to null when a new command token is pushed to the command stack.

When an input begin code is appended to the output, this new element pointer will be set to the input begin code element. This pointer can also be used to indicate if any input variables have been received yet (when it does not contain a null).

To insert a input parse code, the input parse token will be appended to the list at (after) this element pointer. The element pointer will then be set to the output list element of the input parse token just inserted so that the next input parse token will be inserted after this input parse token.

INPUT Translation – Beginning

Execution of the INPUT command has been defined and therefore the form of the translation, the actual process of the translation can now be defined. The translation begins with one of the InputBegin codes, which will be triggered by a comma or semicolon token. The INPUT or INPUT PROMPT token will already be on top of the command stack.

For the INPUT command, when the first comma, a semicolon or an end-of-line token is received, an InputBegin token will be appended to the output. The token received can be converted to the InputBegin token (more efficient to change the token than to delete the token not needed and then create a new one).

For the INPUT PROMPT command, when a comma or semicolon token is received, there must be a string on top of the done stack (the prompt string expression). Depending on whether this string is temporary or not will determine whether an InputBeginStr or InputBeginTmp will be appended to the output. For InputBeginStr, the string will be attached since the translator will not know if it is a variable, array or a user function. The token received can be converted to this token. An end-of-line token at this point would produce an “expected comma or semicolon” error.

Automatic Code Enumeration Generation

To have the Code enumeration generated automatically from the table entries, the table source was structured so the awk scripts can read it. Since the Code enumeration value will now be the same as the table entry index, the code member of the table entry is not necessary and was removed. The code name initializers in the table entries was moved to a comment on the line of the entries' open brace.

The awk scripts were rewritten to read the table source file instead of the main include file. The awk script were also changed to read the table source file directly and write the output files directly. This eliminates the requirement to redirect the input and output to the correct files when running the awk scripts. Logic was also added to the codes awk script to check for duplicate code names.

The code to index conversion array that was initialized in the table class constructor, along with the check for duplicate and missing codes, was removed. The code and index access functions in the table class were also removed. The token class index member was replaced with a code member. All the code was updated to use the code enumeration value instead of the index, though the code will be used as an index.

One problem with using an enumeration value instead of an integer index is that normal math functions cannot be used, like the add and increment operators. These operators are needed, so operator functions were created for the code enumeration, which includes the add, prefix increment and postfix increment operators. These functions type cast to integer to add and then type cast back to the code enumeration value.

The null code entry at the end of the table was moved to the beginning so that that null code enumeration value (index) would be zero. The table search function for searching for an immediate command previously assumed that the immediate commands were at the beginning of the table entry array. Moving the null code to be the beginning of the array complicated this. Therefore, immediate command bracketing codes were put around these entries for this search function.

Due to the these table entry changes, the parser test output files were updated since all the code indexes changed. This would be a good time to make another pre-release, but since there have been no download activity for recent pre-releases, there will not be a pre-release at this time. Now the translation of the INPUT command can begin...

Code Enumeration vs. Table Entry Indexes

While designing the error handling mechanism for the INPUT command, a thought occurred related to program codes. For efficient program execution at run-time, the index of the table entry will be stored in the internal program code, not the code enumeration value.

If the code enumeration value was used, the index for the table entry (needed to get the run-time handler function pointer) would first need to be converted to an index by going through the code to index array setup during table initialization. The intention all along has been to use table entry indexes in the internal program code.

For the INPUT command's error recovery, when it is backing up execution and checking for input parse codes, it will need to convert the table entry indexes to a code before it can check if it is an input parse code. This would not efficient during program execution. Even though for the INPUT command, execution time is not critical since it is about to stop and wait for user input. A few extra program cycles won't matter much. But this problem could occur for other more critical commands.

Therefore, it is desirable if the code enumeration values were the same as the table entry indexes. One simple solution is to make sure the code enumeration values matched the table entries. Unfortunately this relies on the programmer to keep the two in sync, is very error prone and is just a general pain to begin with.

There is a better way where the code enumeration is generated automatically from the table entries using an awk script. This method would be similar to how the test_codes.h file (used by the test_ibcp.cpp source file) is generated automatically by scanning for codes in the ibcp.h file.

INPUT Execution – Error Handling

Errors can occur while parsing the input – in one of the input parse codes. When an error occurs, after it is reported, the already parsed values in the temporary input values stack need to be thrown away and execution needs to resume at the beginning of the INPUT statement, except instead of issuing the prompt again, the cursor will be positioned at the beginning of the input, which will contain the previously erroneous input to allow the user to correct the input instead of reentering it (the traditional “redo from start”).

The temporary input values stack can't simply be reset (setting the internal index to -1, the empty stack indicator) because elements may contain allocated string values, which need to be deleted to prevent memory leaks. Like the evaluation stack, the elements in the temporary input values stack won't have any indicator what data type they are. Consider the basic format of the INPUT statement (only up to the parsing codes is shown):

InputBegin InputParseType1 InputParseType2 InputParseType3'End' ...

Say an error occurs on the second parse code (the program execution pointer will be pointing at next code, the InputParseType3, in other words, the pointer is incremented after reading each program code word, then the code read is executed by calling its run-time handler). Execution needs to be backed up until the InputBegin is reached (reading the program codes in reverse).

As each code is passed, one element will be popped from the temporary input values stack. If the code passed is an InputParseStr code, then the element popped is a string that needs to be deleted. The beginning is reached when a non input parse code is reached (it could be InputBegin, InputBeginStr or InputBeginTmp).  The stack will now be empty.

The INPUT begin code will be executed again and the begin code will call the get input routine. The get input routine will normally allocates the temporary input values stack. For error recovery, it will see that the stack is already allocated, so instead of outputting the prompt and saving the cursor position to the beginning of the input, it will restore the saved cursor position and get the input starting with the previously entered erroneous input. Execution will then resume with the corrected input.

INPUT Execution – Temporary Values

The values parsed from the entered input to be assigned to the input variables need to be stored somewhere other than the evaluation stack. The logical place is another temporary input values stack. This stack will be allocated and initialized in the get input routine and will be removed by the final INPUT command code.

After a value is parsed by one of the input parse codes, it will be saved (pushed) to a temporary input values stack. At the last input parse code (with the 'End' sub-code), an index to be used to access this stack will be set to zero. As each value is assigned by an input assign code, this index will be incremented. In other words, this stack is being used as First-In-First-Out out list instead of a Last-In-First-Out standard stack.

The SimpleStack (to be renamed to just Stack since there is no other class named Stack), does not currently have this mechanism. This will be added when the run-time code implemented, but it is not needed currently for translating the INPUT command, so this addition will wait.

INPUT Execution Codes – Ending

There will be the final INPUT command code at the end of the input statement. Besides cleaning up, the only action that needs to be performed is to advance to the next line if the 'Keep' sub-code is not set (this sub-code is set when there is a semicolon at the end of the INPUT statement). In summary, the “INPUT I%,A(I%)” statement will be translated as:

InputBegin InputParseInt InputParseDbl'End' I%<ref> InputAssignInt I% A(<ref> InputAssignDbl Input

As usual for RPN format, the command is at the end of the translation. What remains to be designed is where the temporary input values will be stored and how errors will be handled. An error can occur in the parsing codes. When an error occurs, execution must go back to the InputBegin, however, the prompt does not need to be output again, the cursor only needs to be positioned back where is was after the prompt was output (after the error is reported).

INPUT Execution Codes - Assigning

The assigning of the input values must be done separately from the parsing of the values entered due to the two input rules. For the example, this is steps 7 through 11. Some of these steps are standard expression codes: push a reference to an integer variable (step 7), push a value to an integer variable (step 9), and calculating a reference to an array element by popping a integer subscript value and pushing the reference to the element (step 11).

There will be a code to assign an input value to an input variable for each data type: InputAssignInt, InputAssignDbl and InputAssignStr. For InputAssignStr, the InputParseStr will created a string from the input value. This string will be assigned to the string variable replacing the previous string value, which will be deleted. Therefore, there will be no need to deal with temporary strings.

The values being assigned will need to stored temporarily somewhere other than the evaluation stack.

INPUT Execution Codes - Parsing

The parsing of the entered input values must be done separately from the assignment of the values entered to the input variables. This a departure from the design previously described and is due to the two input rules. For the example, this is steps 3 through 6. Notice that after a value is parsed (steps 3 and 5), a check is made for the next character. This character must be a comma after each value except for the last value where an end-of-line (no character) is expected.

There will be a code to parse an input value for each data type: InputParseInt, InputParseDbl and InputParseStr. An 'End' sub-code will be set on the last parse code. If this sub-code is not set, then the next character must be a comma, otherwise an end-of-line (no character) is expected.

The values that are parsed need to be put somewhere. If input values were pushed on to the evaluation stack, then when the references to the input variables are pushed, the input values would be down the stack and would not be easily accessible. Therefore, the evaluation stack can't be used.

INPUT Execution Codes - Prompting

Execution and translation of the INPUT command was previously described mostly in posts on June 24, 2010, June 25, 2010 and June 27, 2010 . This now needs to be revised since the execution broke the two rules listed at the end on February 16, 2011. Steps 1 and 2 handle issuing the prompt and getting input from the user. This is almost the same as previously defined, which are these codes:

InputBegin    – output default prompt and get input
InputBeginStr – output string prompt and get input
InputBeginTmp – output string prompt, get input and delete temporary string

During execution, the run-time handlers for each of these codes will call a common routine for getting input. This common get input routine can also handle outputting the prompt and can simply use the string on top of the evaluation stack. There will be an argument for whether to output the prompt string on top of the stack and InputBegin will set this argument to false.

There will be another argument for whether to output the default prompt where InputBegin will set this to true and the other two will set to true if the 'Question' sub-code is set (if the prompt string expression was followed by a comma instead of a semicolon).

For InputBeginTmp, the temporary string can be deleted upon returning from the get input routine since it will no longer be needed with the improvement in execution described in last Saturday's posts.