Translator – Assignments (Development)

The last several days was spent implementing the new design for handling assignments. The assignment operators are no longer handled by the operator routines, and now by the comma and equal token handlers and the assign command handler (which previously didn't do anything). Two support functions were also implemented, one to put the appropriate assignment command on the command stack based on the first (perhaps only) item being assigned, and the other to check each assignment item for the correct data type (allowing for mixed string and sub-strings).

At the end of the statement, the value (expression) being assigned is checked for the correct type in the assign command handler, adding a hidden conversion as needed for the numeric data types. Because the assignment operators are no longer handled as binary operators, the table entries for the assignment operators were modified where each only has one operand (for the value being assigned).

Also, tokens with parentheses being assigned can be assumed to be arrays since a function with arguments cannot be assigned (only the function name alone, without parentheses, can be assigned). Therefore, unlike tokens with parentheses in expressions that can be either an array or a function call, the values in parentheses of an array being assigned can be assumed to be subscripts, which must be integers (or doubles with conversion). If it turns out that the name is not an array when encoded, the Encoder will report the error.

Several other data type reporting error issue were also corrected but without adding any special expression type handling as was planned in the failed design concept. A new concept was developed, which probably won't require the Translator to keep track of the expression type while translating. More details to follow...

Translator – Expression Type (Failed Design)

The first idea on how to implement expression type checking into the Translator failed. It's not necessary to go into the details of the design, but basically the idea was to set the expression type at the start of the expression. For assignments, the expression type would be the same as the variable(s) being assigned. For PRINT statements, any expression type would be allowed, but would be set based on the first operand. For INPUT PROMPT, the expression type would be string. However, certain expressions would trip this up. First consider this statement:

Z = A$ + B$ + C$

Currently this would report an “expected numeric expression” at the second plus. It should report the error at the A$. The idea was at the equal, the expression type would be set to numeric, then upon seeing A$, which is a string, an error would be reported. However, consider this valid statement:

Z = A$ + B$ < C$

The Translator can't set the expression type to numeric and then report the error at the A$ because the expression eventually becomes numeric at the less than.

The bottom line is that the Translator can't determine if the expression is an the correct type until the entire expression is translated, but, still needs to report at the first instance where the data type error occurs, not at the last operator processed. The goal now is to get the new assignment implementation working (needed first anyway), and then get the expression type handling implemented...

Translator – Assignments (New Implementation)

The new implementation of assignment statements consist of removing the existing code list assignment processing from the add operator routine and adding processing to the equal token handler, comma token handler and the assign command handler (which currently does nothing).

Comma: When the first comma token is received, the data type of the assignment is set to the first assignment item and an assign list code appropriate for the item's data type is pushed to the command stack instead of pushing the main assign list code to the hold stack. If there is a LET command already on the command stack, it will be replaced with the assign list code.

For each additional comma token, the data type of the assignment item will be checked to make sure it matches the current data type. If the data type is a string or sub-string and the new item's data type is also a string or sub-string, but not the same, the assign list code will be changed to the AssignListMixStr code. When an equal token is received in the comma assignment, the mode is set to expression after the last list assignment item is checked and the expression type is changed to Numeric or String.

Equal: When an equal token is received first, the data type of the assignment is set to the assignment item and an assign code appropriate for the item's data type is pushed to the command stack instead of the hold stack. If there is a LET command already on the command stack, it will be replaced with the assign code. The expression type is changed to Numeric or String in case the expression starts after the equal.

If a second equal token is received, the assign token on top of the command stack is changed to the appropriate assign list code. For each additional equal token, the data type of the assignment item will be checked to make sure it matches the current data type. The same check for strings made for each additional comma token is also made for each additional equal.

I realized that it is not necessary to save string variables in assignment and list assignment statements – it is already known these operands will not be temporary strings. Only a string value needs to be saved for string assignments as it may be a temporary string. Also, the Translator can identify some strings as temporaries (the result of the concatenate operator and all but the sub-string internal functions that return a string). Modifying the Translator for these temporary strings will be done after the INPUT command is implemented.

Translator – Assignments (Change)

The way assignment and list assignment statements are processed needs to be changed to work with the expression type handling. Currently the assignment tokens are processed as operators, pushed to the hold stack when received and processed when emptied be a lower or same precedence token is received (an EOL token). The processing is performed by the find code routine activate by seeing the reference table flag set, with the additional list assignment processing in the add operator routine after find code returns.

A better method is to process assignments and list assignments as the statement is processed. This means as each comma and equal tokens are received, the data types of items being assigned are checked immediately. The first item received (a token with no parentheses, a token with parentheses or a sub-string function) determines the type of the assignment. For each additional item received in the assignment list, the data type of the item must match exactly, except for the string data type where strings and sub-strings may be mixed in the same list assignment statement.

Once the expression starts, the expression type is set to either Numeric (for the double and integer data type since both can accept either for an assignment value) or String. At the end of the statement, the data type of the expression will be checked, a hidden conversion operator will be added if necessary, followed by the assign or assign list token.

This new method is simpler than the current method, which contained involved code for detecting errors in the list assignment – the first error in the statement needed to be reported. Complicating matters was the fact that the assignment list items are processed backwards because the items are popped from the done stack in the reverse order from how they were received.

Translation – Expression Type

The Translator needs to keep track of the expression type as an expression is translated. The expression type is similar to the data type, but not the same. There are two types of expressions, Numeric (the double and integer data types are interchangeable in expressions) and String (including the sub-string data type). The issue was discovered when working on the INPUT PROMPT string expression, but the problem applies to all expressions. Consider these examples that will currently report an error at an inappropriate token:

Z = A$ + B$ + C$
            ^-- expected double

Z$ = A + B * C
           ^-- expected string

These errors will be confusing. The errors in both cases should be pointing to the A$ or A variables and should report “expected numeric expression” and “expected string expression” errors. Using just the word double in the message might imply that an integer expression is unacceptable, which is not the case.

The expression type may need to be set to a generic Any expression type, like would be needed for the PRINT command's expressions or the arguments of non-internal functions. For internal functions, the expression type can be set to the correct value for each argument (using the Numeric expression type if the argument is a double or an integer).

For arrays, the subscripts need to be integers (Numeric), but remember that the Translator does not know if a token with a parentheses is an array or a user function. So the expression type will have to be set to the Any expression type and the Encoder will have to do the checking once it is known whether the token is an array or user function.

Translation – INPUT and Error Reporting

Upon designing the actions required by the add input codes routine, I realized that it may need to point to a different token when an error is detected. Consider this example:

INPUT PROMPT A;B

The error will be “expected string expression” and should be pointing to the A, but the current token being processed is the semicolon token. Therefore, instead of passing the token code of the token being processed, a reference to the token pointer will be passed so that it can be changed to point to the actual token with the error.

For The INPUT command handler (called at end of statement tokens), a Colon code was going to be passed regardless of the actual end of statement token, but now a reference to a token pointer will be passed. This will be whatever end of statement token, which was passed to the INPUT command handler and will be passed on to the add input codes routine.

This led to another problem, consider this example:

INPUT PROMPT A*B+C;D

The + operator will be on top of the done stack after the expression is processed, so the error will be pointing to + since it will have the double data type. The error should be pointing to the A indicating that a string expression was expected. This problem applies to other statements as well, so some sort of expression type detection is needed...

Translator – INPUT Command (Design)

As entries were being written for INPUT with Semicolons, INPUT with Commas, and INPUT Command Handler, there was a lot of copying and pasting of the same text for what the will need to do for translation. In considering how to reduce the amount of words, a realization was made that this is an indication that since the code is the same or very similar that perhaps the common code should be put into a single routine.

The plan was already to have an add input code routine like the add print code routine.  But since there is more common code for the INPUT and INPUT PROMPT commands, the functionality of this add input codes (plural since multiple codes are involved) routine will be expanded.  This routine will need to know the command being translated (INPUT or INPUT PROMPT) along with the current command flags and will need to know which token is being processed (Semicolon, Comma, or an end of statement token).

The first argument then will be the command item from the command stack, which contains the current command code, command flags and the command's token. From the Semicolon and Comma token handlers, this will be the top command item on the command stack.  From the INPUT command  handler, the command item has already been popped from the command stack and is passed as an argument. This is why the add input codes routine can't just use the top of the command stack.

The second argument will just need to be the code for the token being processed.  The Semicolon and Comma token handlers just need to pass their own code.  The INPUT command handler will pass the Colon code, which will be interpreted as the end of statement.  (Noting that the INPUT command handler could be called from any end of statement token like EOL, Colon, ELSE, and ENDIF; so using the Colon code is appropriate.)

Translator – New Token Modes

Translating INPUT statements will require additional token modes. The current token modes are:

Command – Translator is expecting a command token (or start of an assignment)
Assignment – Translator is expecting an item for an assignment statement
EqualAssignment – An equal token was received when the mode was Command or Assignment; another equal token would indicate a multiple assignment statement (commas are not permitted)
CommaAssignment – A comma token was received when the mode was Command or Assignment; another comma token would indicate continuation of a multiple assignment statement (an equal token would indicate the end of the list and the begin of the expression)
Expression – Translator is expecting operands of operators depending on the current state

When a semicolon appears at the end of an INPUT statement, no further tokens should be received except for an end of statement token (EOL, colon, ELSE, and ENDIF). A new mode is required so the Translator can make sure no additional non end of statement tokens are received:

EOS – Translator is expecting an end of statement token only

An INPUT statement contains variable(s) that are to be input. Expressions are not allowed (except for the string expression after the PROMPT keyword, or within subscripts of array variables). The INPUT translation could be implemented to check if the token on top of the done stack has the reference flag set, and if not, report an “expected variable” error. But that could leave to strange errors being reported, consider this example (with the translation of the expression):

INPUT A*B+C  A B * C +

The + will be on top of the done stack after this expression is translated (being the result of the translated expression). The + token will not have the reference flag set since it is an operator. The INPUT is expecting  a reference, so it would report “expected variable” pointing to the + token. This would be very confusing – why would a variable be expected at the +? The correct error should be “expecting comma or end of statement” pointing to the * token. A new mode is required so the Translator can make sure no operators (except for end of expression operators comma, semicolon and EOL) are received:

Reference – Translator is only expecting reference tokens and end of expression operator

Reference tokens include tokens without parentheses and tokens with parentheses. However, these type of tokens could be variables, arrays or functions, but the Translator is not able to determine which. Therefore, the Encoder could still find errors if a user function was placed in an INPUT statement. Lastly, sub-string functions (while valid in assignment statements) are not valid in INPUT statements, therefore the Reference mode needs to check for these and report an error.

Translator – INPUT command

Similar to the translation of the PRINT statement, a lot of the translation work of the INPUT statement will take place in the semicolon and comma token handlers, with the INPUT and INPUT PROMPT command handlers being called at the end of the statement. The codes have been renamed for consistency and clarity, InputGet code will now be InputBegin, and InputPromptStr and InputPrompTmp will now be InputBeginStr and InputBeginTmp (the Tmp versions are not handled by the Translator).

There will be two command flags to keep track of the INPUT statement translation. The first is the InputBegin command flag, which indicates whether an InputBegin has been appended to the output yet. The semicolon and comma token handlers will use this flag to determine if the InputBegin code has been appended (INPUT) and whether a prompt string expression result is expected (INPUT PROMPT).

The semicolon will also use the InputBegin command flag to determine if it is at the end of the statement, which determines whether to set the second command flag, InputKeep. The InputKeep flag will be used by the command handlers at the end of the statement to determine whether the InputKeep sub-code flag should be set in the Input and InputPrompt codes at the end of the translated statement.

Translator – Error Handling (Preliminary Release)

I decided to try using CVS branching for development of new releases. This way, working code (but not ready to release) could be committed to the CVS repository and development can continue. Differences made can then be easily checked. What has been done was good files were copied to a different file name, which is fine for one file, but when a whole set of files are involved, it becomes pain. This accounts for the strange CVS revision numbers in the source file, but should clear up once an official release is made.

Since the known error reporting issues have been resolved, this is a good time to make a preliminary release before the changes to implement the INPUT command begin. There are now many test inputs for testing errors (there should probably be more). The file ibcp_0.1.14-dev-1-src.zip has been uploaded at Sourceforge IBCP Project along with the binary for the program. Now implementation of the INPUT command can begin...

Translator – Error Handling (Corrected)

Between working on the development of the INPUT command this week, some time was spent on working out the remaining error reporting issues. Three major areas needed additional checks to report the desired errors:

1. Internal functions not inside expressions, except for functions that return a sub-string  (which can be used in assignments) – this was an issue because the Translator would allow internal functions in assignments, but only the sub-string functions are allowed.
2. At the end of an expression if the mode is assignment, checks were added to see if the Translator was inside an array or sub-string function – this was an issue because the Translator was in expression mode for the subscripts/arguments, but the errors were related to the assignment, therefore the count stack was checked to determine the current situation.
3. There were several places where checks were needed to determine if the done stack was empty to report the desired error – this included the end of expression in comma assignment mode, comma in assignment mode, semicolon in assignment mode, and LET command with no variable after keyword.

One more check was required at the end of a statement (currently just end-of-line, but will be required for colon when that is implemented). If an EOL was received with the mode set to comma assignment (meaning more items in the list are expected – the mode is set to expression once an equal token is received), then an “expected equal or comma for in assignment” error is reported. A new end statement table entry flag was added for this.

The giant switch statement for the error messages in the test source file was not desirable. These eventually need to be placed elsewhere. Putting them in an array is the logical choice, but how to index the array? One way is to make sure the entries are in the same order as the token status enumeration, but this requires care to make sure it is correct and when it is not, the problem is difficult to debug. So a structure was added that contains the token status value and the string. During initialization, an index conversion array is set up, and during set up is validated to make sure there are none duplicated or missing – just like with the codes enumeration values in the table entries.

Since the error mechanism is almost identical to that used with table errors, instead of cloning this code (a common, but bad, practice of many programmers), an error template structure was implemented to be used for the token status message errors and table errors (the token status message errors only require the duplicate and missing error types). The routine to print the errors in the error list is also defined in the template in the include file. Putting calls to the fprintf() standard library function in this function is a bad idea, so a pointer to generic print function is passed to the error reporting template function. For now the print function passed just outputs to the standard error stream.

Execution – INPUT command (Revised)

Both the INPUT and INPUT PROMPT commands will share routines at run-time. Normally it will not be desirable to call extra functions or check sub-code flags at run-time as this uses execution time (there is overhead calling functions in C/C++). Better if these decisions can be made before hand. But since this is the INPUT command, execution time will not be critical. Therefore, there will be a function for outputting the “? ” (optionally based on a flag argument) and reading the input from the keyboard.

Instead of just pushing the location of the input get code to the evaluation stack (so that it can be re-executed upon an error), a flag for whether there is a prompt expression and a flag whether the prompt is a temporary string will be pushed along side the location. In other words, there will be an input stack item structure containing the location, prompt flag and temporary flag. So when the Input or InputPrompt code is executed, they will have access to whether there was a prompt string (that needs to be popped) and whether it is a temporary string (that needs to be deleted).

But is it necessary to know there is a prompt since Input will know that there is no prompt and InputPrompt will know there is a prompt (though it still needs to know if the prompt is a temporary string). Both Input and InputPrompt will need to check for an EOL in the input buffer, process the assignments on the stack, pop the location entry from the stack, and check if the cursor should be kept on the same line.

Since the temporary flag needs to be carried from the InputPromptStr and InputPromptTmp codes (why not also carry the prompt flag), and both Input and InputPrompt share a lot of the same work, both will share the same run-time routine. This is verses having three routines (one for Input, one for InputPrompt, and one for the common work). The single run-time routine will check for a prompt (which will be popped from the stack) and if it is temporary (which will be deleted).

The InputGet code will push the location with no prompt flag, and then call the get input routine with the question flag argument set. The InputPromptStr will push the location with the prompt flag set and no temporary flag, and then call the get input routine with the question flag argument set if the Question sub-code flag is set. The InputPromptTmp will push the location with the prompt flag and temporary flag set, and then call the get input routine with the question flag argument set if the Question sub-code flag is set. Now back to the implementation of the INPUT statement translation...

Translation – INPUT command (Revised)

Now with the updated syntax to the INPUT statement with the addition of the optional PROMPT keyword, the translation needs to be revised. Since the PROMPT keyword will always follow the INPUT keyword, it will be easiest to handle it as the double keyword command INPUT PROMPT, in other words, it will be a separate command. The translations of these two statements will now be:

InputGet {<variable> InputType}... Input
<expr> InputPromptStr {<variable> InputType}... InputPrompt

This mirrors other commands where the command token is placed at the end of the translation. The InputGet code does not need any sub-codes as it will always output “? ” before getting input. The InputPromptStr will have a Question sub-code to determine if an “? ” should be output after the prompt string.

There will also be an InputPromptTmp code for when the prompt string expression is a temporary string, but this determination won't occur until the Encoder can determine if the string is temporary or not. This also mirrors how temporary strings are handled for other codes. If the Tmp sub-code flag was used, this would be a different scheme and would complicate the Encoder implementation of temporary strings. There is a way to avoid the Input or InputPrompt code needing a Tmp sub-code flag (these are where during run-time that a temporary prompt string is will be deleted).

Both the Input and InputPrompt codes will still have the Keep sub-code for when the cursor should not be advanced to the next line. Here are the revised examples originally shown here with their new translations:

INPUT A InputGet A<ref> InputDbl Input
INPUT PROMPT "Qty: ",A% "Qty: " InputPromptStr A% InputInt InputPrompt
INPUT PROMPT P$+C$,A$; P$ C$ +$ InputPromptTmp A$ InputStr InputPrompt'Keep'
INPUT PROMPT "",A,B% "" InputPromptStr A<ref> InputDbl B%<ref> InputInt InputPrompt

Note there is no equivalent to the last INPUT example command that disabled the prompt – the INPUT PROMPT will have to be used with an empty string as shown above. This should not be an issue since prompt-less INPUT statements will not be common. Also notice that there are three codes that will be outputting a prompt and getting the input. Next, how execution is affected by these changes...

Language – INPUT command (Change)

As the translation of the INPUT command was being designed, a serious flaw was uncovered with the current syntax. This can be shown with an example. Say there is a user function named Prompt$ that takes a single argument, in other words FUNCTION Prompt$(Index). The intention is to use this function as the prompt string expression in an INPUT statement, which looks like this (using the comma separator to suppress the addition of the default question mark prompt):

INPUT Prompt$(I),Array(I)

To the Translator, the Prompt$ is a token that has a parentheses, which could be either an array or a user function. Requiring parentheses to be inserted around Prompt$(I) to force it to be detected as a prompt expression instead of a variable is not acceptable. The translation of the above is different depending on whether it is an array or a user function:

Array:  INPUT'Question' I Prompt$(<ref> InputStr I Array(<ref> InputDbl Input
Function:  I Prompt$( INPUT'Prompt+Question' I Array(<ref> Input Dbl Input'Tmp'

These two translations are radically different, so one can't assumed and then changed later by the Encoder when it has determined whether it is an array or a function. The Translator needs to know whether it is a prompt or not. If it knows that it is a prompt, then it does not need to know whether it is an array or a function (the translations are the same). For a semicolon, this is not an issue because the item before semicolon is a prompt (unless it is at the end of the line).

To resolve this issue, there are two alternatives. Either the comma separator be eliminated (where a semicolon only indicates a prompt and suppresses the “? ” leaving no way to add the “? ” except to actually add it to the prompt – not desirable), or use the ANSI BASIC syntax of the PROMPT keyword to indicate a prompt string expression follows. The latter will be used and the functionality of the comma (no “? ”) and semicolon (add “? ”) will remain.

Using the PROMPT keyword is the more logically because it clearly shows that there is a prompt, and this keeps with the spirit of a beginner's language (which is probably why it is part of ANSI BASIC). Therefore the syntax of the INPUT statement will now be:

INPUT [PROMPT <string-expression> {,|;}] <variable>[,<variable> ...][;]

Next, how this will affect the resulting translation and execution of the INPUT statement...

Execution – INPUT command (More Details)

The characters in the input buffer are parsed into tokens separated by commas. If there is a double quote at the beginning of a token, then a string constant is scanned for a matching closing parentheses. The surrounding double quotes will not be included in the string value. Two double quotes within the string constant are treated as a single double quote. If the token begins with a numeric constant character, then an attempt is made to see if the entire token is a valid double or integer constant (otherwise it is considered a string constant). After each value token is parsed, the status flag is set to Comma or EOL depending on what terminated the value.

The InputDbl code will check for a double or integer value, otherwise an error occurs. An integer constant is converted to a double. The InputInt code will check for a double or integer value, otherwise an error occurs. A double constant is converted to an integer if possible, otherwise an error occurs. An InputStr code will accept any value type. For double and integer values, the characters entered are used. Double quotes are needed around a string value that contains a comma.

When an error occurs, everything pushed to the evaluation stack needs to be popped up to the prompt (if there is one) and execution needs to repeat the INPUT statement. The reason the location of the InputGet code is pushed to the stack is so that execution can resume there upon an error. Any string prompt on the stack is not popped so that it can be reused.

The code the error occurs at (an InputType or Input) and the input counter value will determine how many entries need to be popped from the stack. For an InputType code, an error would occur after the variable reference is pushed to the stack but before the value and assignment function pointer. For InputType, the number of entries to pop is count*3+1. For Input, the number of entries is count*3. Once these entries are popped, the location of the InputGet is popped. The string “Redo from start” is output and execution continues at the location of the InputGet.

Notice that at no time during the execution of the INPUT statement, including error processing, is it required to know what type of value is at any stack entry – it already knows. Next, onto the translation of the INPUT statement...

Execution – INPUT command

To prove that the selected translation is appropriate, consider what occurs during execution, here are some example INPUT statements along with their translations:

INPUT A  InputGet'Question' A<ref> InputDbl Input
INPUT "How many: ",A%  "How many: " InputGet'Prompt' A%<ref> InputInt Input
INPUT P$+C$,A$;  P$ C$ +$ Input'Prompt' A$<ref> InputStr Input'Tmp+Keep'
INPUT,A,B%  InputGet A<ref> InputDbl B%<ref> InputInt Input

If there is a prompt string expression, its result will be on top of the evaluation stack. At the InputGet code, if the Prompt sub-code flag is set, then the string on top of the stack is output. If the Question sub-code flag is set, then a “? ” is output. The location of the InputGet code is pushed to the stack (to be used if an error occurs so that the InputGet can be returned to). The characters are input into a buffer until an Enter key terminates the input. There will be a status flag variable to keep parsing status of the input (initialized to None) and a value counter variable to count the number of input values (initialized to zero).

A variable reference is pushed to the evaluation stack. At an InputType code, the status flag is checked for a None or a Comma status, otherwise an error has occurred. The input buffer is scanned for a value token. If the value is the correct type, or can be converted to the correct type, then it is pushed to the stack, otherwise an error occurs. Finally, a pointer to an assign code run-time function for the data type is pushed to the stack.

At the Input code, the status flag is checked to make sure that it is set to EOL (a Comma status indicates there is more input, which is an error). An assign function pointer is popped from the evaluation stack and then called, which assigns the value to the variable that is the stack (both are popped by the assignment). This repeats for each value until the input value counter is decremented to zero. The InputGet code location is popped from the stack and finally the prompt string is popped from the stack. If the TmpStr sub-code flag is set, then the temporary prompt string is deleted. If the Keep sub-code flag is not set, the Print function is called to advance to the next line.  Next, some more execution details...