As the INPUT command handler was being implemented, some issues were found. When the new element pointer was added to the command item, it was noticed that there was a code member. This member is no longer needed because the token now contains the code (replacing the index member, through the code is now an index), so the code member was removed from the command item.
There was no table entry for the two word INPUT PROMPT command, so one was added along with the three input begin codes. It was noticed that some table entries still had the string flag set. This is not necessary because the string flag is now set automatically during table initialization it there are any string arguments, so these string flags were removed.
To look up which input begin code to append for the INPUT PROMPT command, the process final operand routine will be called, which in turn will call the find code routine that will pick the correct code based on the type (string or temporary string) that is on the done stack. The input begin codes will not push anything to the done stack (accomplished by setting the done push flag to false). Some additional issues were found in these routines...
Thursday, March 3, 2011
Wednesday, March 2, 2011
INPUT Translation – Variables
Processing variables is a bit more complicated. With the PRINT statement, the appropriate print value type code was simply appended to the output. However, with the INPUT statement, an input parse code needs to be inserted after the begin code or after last parse code (before all of the variables), and an input assign code needs to be appended to the end of the output (after the variable).
There needs to be a way to point to the location where the input parse codes are to be inserted. This will be accomplished with a new output list element pointer member to the command stack item. This pointer will be initialized to null when a new command token is pushed to the command stack.
When an input begin code is appended to the output, this new element pointer will be set to the input begin code element. This pointer can also be used to indicate if any input variables have been received yet (when it does not contain a null).
To insert a input parse code, the input parse token will be appended to the list at (after) this element pointer. The element pointer will then be set to the output list element of the input parse token just inserted so that the next input parse token will be inserted after this input parse token.
There needs to be a way to point to the location where the input parse codes are to be inserted. This will be accomplished with a new output list element pointer member to the command stack item. This pointer will be initialized to null when a new command token is pushed to the command stack.
When an input begin code is appended to the output, this new element pointer will be set to the input begin code element. This pointer can also be used to indicate if any input variables have been received yet (when it does not contain a null).
To insert a input parse code, the input parse token will be appended to the list at (after) this element pointer. The element pointer will then be set to the output list element of the input parse token just inserted so that the next input parse token will be inserted after this input parse token.
Monday, February 28, 2011
INPUT Translation – Beginning
Execution of the INPUT command has been defined and therefore the form of the translation, the actual process of the translation can now be defined. The translation begins with one of the InputBegin codes, which will be triggered by a comma or semicolon token. The INPUT or INPUT PROMPT token will already be on top of the command stack.
For the INPUT command, when the first comma, a semicolon or an end-of-line token is received, an InputBegin token will be appended to the output. The token received can be converted to the InputBegin token (more efficient to change the token than to delete the token not needed and then create a new one).
For the INPUT PROMPT command, when a comma or semicolon token is received, there must be a string on top of the done stack (the prompt string expression). Depending on whether this string is temporary or not will determine whether an InputBeginStr or InputBeginTmp will be appended to the output. For InputBeginStr, the string will be attached since the translator will not know if it is a variable, array or a user function. The token received can be converted to this token. An end-of-line token at this point would produce an “expected comma or semicolon” error.
For the INPUT command, when the first comma, a semicolon or an end-of-line token is received, an InputBegin token will be appended to the output. The token received can be converted to the InputBegin token (more efficient to change the token than to delete the token not needed and then create a new one).
For the INPUT PROMPT command, when a comma or semicolon token is received, there must be a string on top of the done stack (the prompt string expression). Depending on whether this string is temporary or not will determine whether an InputBeginStr or InputBeginTmp will be appended to the output. For InputBeginStr, the string will be attached since the translator will not know if it is a variable, array or a user function. The token received can be converted to this token. An end-of-line token at this point would produce an “expected comma or semicolon” error.
Sunday, February 27, 2011
Automatic Code Enumeration Generation
To have the Code enumeration generated automatically from the table entries, the table source was structured so the awk scripts can read it. Since the Code enumeration value will now be the same as the table entry index, the code member of the table entry is not necessary and was removed. The code name initializers in the table entries was moved to a comment on the line of the entries' open brace.
The awk scripts were rewritten to read the table source file instead of the main include file. The awk script were also changed to read the table source file directly and write the output files directly. This eliminates the requirement to redirect the input and output to the correct files when running the awk scripts. Logic was also added to the codes awk script to check for duplicate code names.
The code to index conversion array that was initialized in the table class constructor, along with the check for duplicate and missing codes, was removed. The code and index access functions in the table class were also removed. The token class index member was replaced with a code member. All the code was updated to use the code enumeration value instead of the index, though the code will be used as an index.
One problem with using an enumeration value instead of an integer index is that normal math functions cannot be used, like the add and increment operators. These operators are needed, so operator functions were created for the code enumeration, which includes the add, prefix increment and postfix increment operators. These functions type cast to integer to add and then type cast back to the code enumeration value.
The null code entry at the end of the table was moved to the beginning so that that null code enumeration value (index) would be zero. The table search function for searching for an immediate command previously assumed that the immediate commands were at the beginning of the table entry array. Moving the null code to be the beginning of the array complicated this. Therefore, immediate command bracketing codes were put around these entries for this search function.
Due to the these table entry changes, the parser test output files were updated since all the code indexes changed. This would be a good time to make another pre-release, but since there have been no download activity for recent pre-releases, there will not be a pre-release at this time. Now the translation of the INPUT command can begin...
The awk scripts were rewritten to read the table source file instead of the main include file. The awk script were also changed to read the table source file directly and write the output files directly. This eliminates the requirement to redirect the input and output to the correct files when running the awk scripts. Logic was also added to the codes awk script to check for duplicate code names.
The code to index conversion array that was initialized in the table class constructor, along with the check for duplicate and missing codes, was removed. The code and index access functions in the table class were also removed. The token class index member was replaced with a code member. All the code was updated to use the code enumeration value instead of the index, though the code will be used as an index.
One problem with using an enumeration value instead of an integer index is that normal math functions cannot be used, like the add and increment operators. These operators are needed, so operator functions were created for the code enumeration, which includes the add, prefix increment and postfix increment operators. These functions type cast to integer to add and then type cast back to the code enumeration value.
The null code entry at the end of the table was moved to the beginning so that that null code enumeration value (index) would be zero. The table search function for searching for an immediate command previously assumed that the immediate commands were at the beginning of the table entry array. Moving the null code to be the beginning of the array complicated this. Therefore, immediate command bracketing codes were put around these entries for this search function.
Due to the these table entry changes, the parser test output files were updated since all the code indexes changed. This would be a good time to make another pre-release, but since there have been no download activity for recent pre-releases, there will not be a pre-release at this time. Now the translation of the INPUT command can begin...
Saturday, February 26, 2011
Code Enumeration vs. Table Entry Indexes
While designing the error handling mechanism for the INPUT command, a thought occurred related to program codes. For efficient program execution at run-time, the index of the table entry will be stored in the internal program code, not the code enumeration value.
If the code enumeration value was used, the index for the table entry (needed to get the run-time handler function pointer) would first need to be converted to an index by going through the code to index array setup during table initialization. The intention all along has been to use table entry indexes in the internal program code.
For the INPUT command's error recovery, when it is backing up execution and checking for input parse codes, it will need to convert the table entry indexes to a code before it can check if it is an input parse code. This would not efficient during program execution. Even though for the INPUT command, execution time is not critical since it is about to stop and wait for user input. A few extra program cycles won't matter much. But this problem could occur for other more critical commands.
Therefore, it is desirable if the code enumeration values were the same as the table entry indexes. One simple solution is to make sure the code enumeration values matched the table entries. Unfortunately this relies on the programmer to keep the two in sync, is very error prone and is just a general pain to begin with.
There is a better way where the code enumeration is generated automatically from the table entries using an awk script. This method would be similar to how the test_codes.h file (used by the test_ibcp.cpp source file) is generated automatically by scanning for codes in the ibcp.h file.
If the code enumeration value was used, the index for the table entry (needed to get the run-time handler function pointer) would first need to be converted to an index by going through the code to index array setup during table initialization. The intention all along has been to use table entry indexes in the internal program code.
For the INPUT command's error recovery, when it is backing up execution and checking for input parse codes, it will need to convert the table entry indexes to a code before it can check if it is an input parse code. This would not efficient during program execution. Even though for the INPUT command, execution time is not critical since it is about to stop and wait for user input. A few extra program cycles won't matter much. But this problem could occur for other more critical commands.
Therefore, it is desirable if the code enumeration values were the same as the table entry indexes. One simple solution is to make sure the code enumeration values matched the table entries. Unfortunately this relies on the programmer to keep the two in sync, is very error prone and is just a general pain to begin with.
There is a better way where the code enumeration is generated automatically from the table entries using an awk script. This method would be similar to how the test_codes.h file (used by the test_ibcp.cpp source file) is generated automatically by scanning for codes in the ibcp.h file.
INPUT Execution – Error Handling
Errors can occur while parsing the input – in one of the input parse codes. When an error occurs, after it is reported, the already parsed values in the temporary input values stack need to be thrown away and execution needs to resume at the beginning of the INPUT statement, except instead of issuing the prompt again, the cursor will be positioned at the beginning of the input, which will contain the previously erroneous input to allow the user to correct the input instead of reentering it (the traditional “redo from start”).
The temporary input values stack can't simply be reset (setting the internal index to -1, the empty stack indicator) because elements may contain allocated string values, which need to be deleted to prevent memory leaks. Like the evaluation stack, the elements in the temporary input values stack won't have any indicator what data type they are. Consider the basic format of the INPUT statement (only up to the parsing codes is shown):
As each code is passed, one element will be popped from the temporary input values stack. If the code passed is an InputParseStr code, then the element popped is a string that needs to be deleted. The beginning is reached when a non input parse code is reached (it could be InputBegin, InputBeginStr or InputBeginTmp). The stack will now be empty.
The INPUT begin code will be executed again and the begin code will call the get input routine. The get input routine will normally allocates the temporary input values stack. For error recovery, it will see that the stack is already allocated, so instead of outputting the prompt and saving the cursor position to the beginning of the input, it will restore the saved cursor position and get the input starting with the previously entered erroneous input. Execution will then resume with the corrected input.
The temporary input values stack can't simply be reset (setting the internal index to -1, the empty stack indicator) because elements may contain allocated string values, which need to be deleted to prevent memory leaks. Like the evaluation stack, the elements in the temporary input values stack won't have any indicator what data type they are. Consider the basic format of the INPUT statement (only up to the parsing codes is shown):
InputBegin InputParseType1 InputParseType2 InputParseType3'End' ...Say an error occurs on the second parse code (the program execution pointer will be pointing at next code, the InputParseType3, in other words, the pointer is incremented after reading each program code word, then the code read is executed by calling its run-time handler). Execution needs to be backed up until the InputBegin is reached (reading the program codes in reverse).
As each code is passed, one element will be popped from the temporary input values stack. If the code passed is an InputParseStr code, then the element popped is a string that needs to be deleted. The beginning is reached when a non input parse code is reached (it could be InputBegin, InputBeginStr or InputBeginTmp). The stack will now be empty.
The INPUT begin code will be executed again and the begin code will call the get input routine. The get input routine will normally allocates the temporary input values stack. For error recovery, it will see that the stack is already allocated, so instead of outputting the prompt and saving the cursor position to the beginning of the input, it will restore the saved cursor position and get the input starting with the previously entered erroneous input. Execution will then resume with the corrected input.
Friday, February 25, 2011
INPUT Execution – Temporary Values
The values parsed from the entered input to be assigned to the input variables need to be stored somewhere other than the evaluation stack. The logical place is another temporary input values stack. This stack will be allocated and initialized in the get input routine and will be removed by the final INPUT command code.
After a value is parsed by one of the input parse codes, it will be saved (pushed) to a temporary input values stack. At the last input parse code (with the 'End' sub-code), an index to be used to access this stack will be set to zero. As each value is assigned by an input assign code, this index will be incremented. In other words, this stack is being used as First-In-First-Out out list instead of a Last-In-First-Out standard stack.
The SimpleStack (to be renamed to just Stack since there is no other class named Stack), does not currently have this mechanism. This will be added when the run-time code implemented, but it is not needed currently for translating the INPUT command, so this addition will wait.
After a value is parsed by one of the input parse codes, it will be saved (pushed) to a temporary input values stack. At the last input parse code (with the 'End' sub-code), an index to be used to access this stack will be set to zero. As each value is assigned by an input assign code, this index will be incremented. In other words, this stack is being used as First-In-First-Out out list instead of a Last-In-First-Out standard stack.
The SimpleStack (to be renamed to just Stack since there is no other class named Stack), does not currently have this mechanism. This will be added when the run-time code implemented, but it is not needed currently for translating the INPUT command, so this addition will wait.
Thursday, February 24, 2011
INPUT Execution Codes – Ending
There will be the final INPUT command code at the end of the input statement. Besides cleaning up, the only action that needs to be performed is to advance to the next line if the 'Keep' sub-code is not set (this sub-code is set when there is a semicolon at the end of the INPUT statement). In summary, the “INPUT I%,A(I%)” statement will be translated as:
InputBegin InputParseInt InputParseDbl'End' I%<ref> InputAssignInt I% A(<ref> InputAssignDbl InputAs usual for RPN format, the command is at the end of the translation. What remains to be designed is where the temporary input values will be stored and how errors will be handled. An error can occur in the parsing codes. When an error occurs, execution must go back to the InputBegin, however, the prompt does not need to be output again, the cursor only needs to be positioned back where is was after the prompt was output (after the error is reported).
Wednesday, February 23, 2011
INPUT Execution Codes - Assigning
The assigning of the input values must be done separately from the parsing of the values entered due to the two input rules. For the example, this is steps 7 through 11. Some of these steps are standard expression codes: push a reference to an integer variable (step 7), push a value to an integer variable (step 9), and calculating a reference to an array element by popping a integer subscript value and pushing the reference to the element (step 11).
There will be a code to assign an input value to an input variable for each data type: InputAssignInt, InputAssignDbl and InputAssignStr. For InputAssignStr, the InputParseStr will created a string from the input value. This string will be assigned to the string variable replacing the previous string value, which will be deleted. Therefore, there will be no need to deal with temporary strings.
The values being assigned will need to stored temporarily somewhere other than the evaluation stack.
There will be a code to assign an input value to an input variable for each data type: InputAssignInt, InputAssignDbl and InputAssignStr. For InputAssignStr, the InputParseStr will created a string from the input value. This string will be assigned to the string variable replacing the previous string value, which will be deleted. Therefore, there will be no need to deal with temporary strings.
The values being assigned will need to stored temporarily somewhere other than the evaluation stack.
Tuesday, February 22, 2011
INPUT Execution Codes - Parsing
The parsing of the entered input values must be done separately from the assignment of the values entered to the input variables. This a departure from the design previously described and is due to the two input rules. For the example, this is steps 3 through 6. Notice that after a value is parsed (steps 3 and 5), a check is made for the next character. This character must be a comma after each value except for the last value where an end-of-line (no character) is expected.
There will be a code to parse an input value for each data type: InputParseInt, InputParseDbl and InputParseStr. An 'End' sub-code will be set on the last parse code. If this sub-code is not set, then the next character must be a comma, otherwise an end-of-line (no character) is expected.
The values that are parsed need to be put somewhere. If input values were pushed on to the evaluation stack, then when the references to the input variables are pushed, the input values would be down the stack and would not be easily accessible. Therefore, the evaluation stack can't be used.
There will be a code to parse an input value for each data type: InputParseInt, InputParseDbl and InputParseStr. An 'End' sub-code will be set on the last parse code. If this sub-code is not set, then the next character must be a comma, otherwise an end-of-line (no character) is expected.
The values that are parsed need to be put somewhere. If input values were pushed on to the evaluation stack, then when the references to the input variables are pushed, the input values would be down the stack and would not be easily accessible. Therefore, the evaluation stack can't be used.
Monday, February 21, 2011
INPUT Execution Codes - Prompting
Execution and translation of the INPUT command was previously described mostly in posts on June 24, 2010, June 25, 2010 and June 27, 2010 . This now needs to be revised since the execution broke the two rules listed at the end on February 16, 2011. Steps 1 and 2 handle issuing the prompt and getting input from the user. This is almost the same as previously defined, which are these codes:
There will be another argument for whether to output the default prompt where InputBegin will set this to true and the other two will set to true if the 'Question' sub-code is set (if the prompt string expression was followed by a comma instead of a semicolon).
For InputBeginTmp, the temporary string can be deleted upon returning from the get input routine since it will no longer be needed with the improvement in execution described in last Saturday's posts.
InputBegin – output default prompt and get inputDuring execution, the run-time handlers for each of these codes will call a common routine for getting input. This common get input routine can also handle outputting the prompt and can simply use the string on top of the evaluation stack. There will be an argument for whether to output the prompt string on top of the stack and InputBegin will set this argument to false.
InputBeginStr – output string prompt and get input
InputBeginTmp – output string prompt, get input and delete temporary string
There will be another argument for whether to output the default prompt where InputBegin will set this to true and the other two will set to true if the 'Question' sub-code is set (if the prompt string expression was followed by a comma instead of a semicolon).
For InputBeginTmp, the temporary string can be deleted upon returning from the get input routine since it will no longer be needed with the improvement in execution described in last Saturday's posts.
Saturday, February 19, 2011
INPUT Command – Execution Procedure
To determine how the INPUT command will be encoded into internal memory, the internal codes need to be arranged according to how the INPUT command will be executed including handling errors. Using the same “INPUT I%,A(I%)” example statement from before, here is the procedure (for the moment without taking into account error handling):
- Issue prompt (for this statement, the default “? ” prompt)
- Get the input from the user (allowing for editing like backspace, cursor left/right, insert/overwrite, delete, and terminated by enter)
- Parse an integer value from the input for I% and save it (I% can't actually be assigned yet)
- Check if the next character in the input is a comma
- Parse a double value from the input for A(I%) and save it (A(I%) can't be assigned yet since I% hasn't been assigned)
- Check if there are no more characters
- Push reference of I% to evaluation stack
- Assign saved integer value to reference on top of stack – I%
- Push value of I% to evaluation stack (I% has now been assigned)
- Calculate reference for array A by popping index from evaluation stack, push calculated reference, A(I%), to stack
- Assign saved double value to reference on top of stack – A(%)
- End of INPUT command, advance to next line of output
INPUT Command – Execution Improvement
An improvement can be made to the execution of the INPUT command that will simplify the execution and will work nicer from the user standpoint. The traditional implementation of the INPUT command was designed to work with Teletypes. This has no place on modern computers. This has to do with what happens when the input entered by the user is invalid.
After surveying several BASIC implementations, all do a form of “Redo from start” on a new line and then reissuing the prompt on another new line for the input again and forcing the user to start their input from the beginning. Many of the implementations don't even check the presence for all the values requested or even accept string values and then simply set the non-entered or invalid values to zero. This is very sloppy programming and forces the programmer to do more work validating input.
A better alternative is to properly parse and validate the input entered. If something isn't valid, then output a temporary error message, point to where the error is (so the user doesn't have to guess what was wrong) and then allow the user to edit their input. Using extra output lines is unnecessary. The error message will be removed from the screen when INPUT is done. This also simplifies run-time in that the prompt string does not need to be saved until the end of the INPUT statement (deleting it if is a temporary string) since it now only needs to be output once.
There are other enhancements that can be made to the INPUT command, like having a fixed length input field with an optional template and accepting special exit keys (function keys, escape, page up/down, arrow up/down, etc) that the programmer can check for, but this will be later once the project is up and running. For now, the translation of current INPUT command needs to be implemented. As always, before the translation can be implemented, some idea how the INPUT command will work at run-time is needed to determine what tokens need to be put into the RPN output list.
After surveying several BASIC implementations, all do a form of “Redo from start” on a new line and then reissuing the prompt on another new line for the input again and forcing the user to start their input from the beginning. Many of the implementations don't even check the presence for all the values requested or even accept string values and then simply set the non-entered or invalid values to zero. This is very sloppy programming and forces the programmer to do more work validating input.
A better alternative is to properly parse and validate the input entered. If something isn't valid, then output a temporary error message, point to where the error is (so the user doesn't have to guess what was wrong) and then allow the user to edit their input. Using extra output lines is unnecessary. The error message will be removed from the screen when INPUT is done. This also simplifies run-time in that the prompt string does not need to be saved until the end of the INPUT statement (deleting it if is a temporary string) since it now only needs to be output once.
There are other enhancements that can be made to the INPUT command, like having a fixed length input field with an optional template and accepting special exit keys (function keys, escape, page up/down, arrow up/down, etc) that the programmer can check for, but this will be later once the project is up and running. For now, the translation of current INPUT command needs to be implemented. As always, before the translation can be implemented, some idea how the INPUT command will work at run-time is needed to determine what tokens need to be put into the RPN output list.
Wednesday, February 16, 2011
Translator – INPUT Command – New Problem
A few new ideas for the INPUT command have been developed that may simplify the execution of the INPUT command and work better than traditional implementation (more on this in a bit). Upon reviewing the previous posts on the INPUT command and reading the ANSI Standard document for the INPUT command again, a problem was discovered with the previous design. The problem can be shown with this INPUT statement:
INPUT I%,A(I%)This is probably bad programming - if the value entered is outside the range of the array, an exception occurs. In any case, it is allowed. Both GW-Basic and QBASIC act as expected where the first value entered becomes the index of the element that is assigned the second value. However, the design laid before won't act as expected. Consider the planned translation for this statement:
InputGet I%<ref> InputInt I% A(<ref> InputDbl InputThis will not work, consider the items pushed to the evaluation stack upon reaching the Input code at the end:
- Input information (prompt is specified, question flag, and location)
- I%<ref>
- Parsed integer value from input
- Pointer to the integer assign routine
- A(I%)<ref> (after I% pushed, popped and reference to A(I%) calculated)
- Parsed double value from input
- Pointer to the double assign routine
- Can't put references to the evaluation stack until previous values have been assigned.
- Can't assign any variables to the input values until the entire input has been parsed and validated.
Monday, February 14, 2011
Translator – Restructure Pre-Release
Since the restructuring of the code was significant, it is probably a good time to make a pre-release of the code before commencing with the INPUT command. The file ibcp_0.1.15-pre-1-src.zip has been uploaded at Sourceforge IBCP Project along with the binary for the program. When uploading these files, it was discovered that the ibcp_0.1.14-src.zip was missing the complete set of test files, and so was uploaded again. The shell script used to automatically generate a release was using the wrong list to generate the source zip file. Now on to the INPUT command (finally)...
Sunday, February 13, 2011
Translator – More Code Restructuring
It turns out there were really no more issues, just some minor bugs that needed to be corrected. However, in looking at the error messages, the “expected statement” just didn't seem to be as clear as it could be and so was changed to “expected command” along with the name of the corresponding token status.
The size of the add token routine has been getting out of hand for some time. It's not really a good idea to let a function get so big because it becomes much harder to understand and maintain. So it is time to break it up into smaller functions. Generally, no variables need to be passed between these functions except for a reference to the token pointer and token status (usually the return value). The add token function was broken up into these functions:
The size of the add token routine has been getting out of hand for some time. It's not really a good idea to let a function get so big because it becomes much harder to understand and maintain. So it is time to break it up into smaller functions. Generally, no variables need to be passed between these functions except for a reference to the token pointer and token status (usually the return value). The add token function was broken up into these functions:
process operand – handles operands when in operand status and token is not an operator
end expression error – gets the error when an expression is ended prematurely (an end expression token is received in operand state) and is only called when the state is not first operand
process unary operator – handles the checking if an operator token is a unary operator received in operand state including processing open parentheses tokens
process binary operator – handles tokens when in binary operator state
process_operator – empties higher precedence operators from hold stack adding them to the output list (functionality merged with the add operator function, which was removed since it only contains a few lines), then calls the token handler for the operator tokenNow the code will be a bit more manageable as the INPUT and other commands are implemented.
Subscribe to:
Posts (Atom)