The implementation of the new translator routines is now complete for all the items previously completed in the old translator routines (LET, PRINT, INPUT and REM, with the additional item of multiple statement support). Before removing the old translator routines, this is a good place to set a tag even though it has been only three commits from the last tag. This \will be the last commit that will contain the old translator routines.
Version v0.4.5 has been tagged. All tests, including the new multiple statement test (#16) with the new translator routines. Test #16 does not pass with the old translator routines since multiple statement support was never added (would have required a new colon token handler). Some comments were added indicating items that need to be removed with the old translators, which will now commence.
[commit 51ef63448b]
Friday, August 23, 2013
New Translator – Colons
Multiple BASIC statements per line will be supported where statements are separated by a colon. As mentioned in the last post, the newly implemented get commands routine, as indicated by its name, was intended to be able to process multiple statements per line, though the code to do that was not implemented.
For multiple statements, colons will not be stored in the program. Instead, there will be a colon sub-code set on the last token of the statement, which is usually the command token, but not always (for example, an assign of a LET or a semicolon of a PRINT). Multiple consecutive colons or a lone trailing colon will not be permitted as they add nothing.
To add support for multiple statements, the get commands routine was modified by adding a check for a colon token after checking for a RemOp token. For a colon token, the token is not needed and so it is deleted. The colon sub-code is set on the last token added to the RPN output list. The loop then continues with looking for the another command. Other changes requires was to add the end-statement flag to the colon table entry, to add the colon sub-code definition, and to modify the token text routine to detect and output the colon sub-code.
A new translator test (#16) was added for testing multiple statements with various commands before colons, and with several statements with errors. The old translator routines crashes on this test as expected since multiple statements are not supported. This will not be fixed since now that the new translator supports everything the old translator supported, the old translator routines can now be removed.
[commit 734521d17f]
For multiple statements, colons will not be stored in the program. Instead, there will be a colon sub-code set on the last token of the statement, which is usually the command token, but not always (for example, an assign of a LET or a semicolon of a PRINT). Multiple consecutive colons or a lone trailing colon will not be permitted as they add nothing.
To add support for multiple statements, the get commands routine was modified by adding a check for a colon token after checking for a RemOp token. For a colon token, the token is not needed and so it is deleted. The colon sub-code is set on the last token added to the RPN output list. The loop then continues with looking for the another command. Other changes requires was to add the end-statement flag to the colon table entry, to add the colon sub-code definition, and to modify the token text routine to detect and output the colon sub-code.
A new translator test (#16) was added for testing multiple statements with various commands before colons, and with several statements with errors. The old translator routines crashes on this test as expected since multiple statements are not supported. This will not be fixed since now that the new translator supports everything the old translator supported, the old translator routines can now be removed.
[commit 734521d17f]
Thursday, August 22, 2013
New Translator – Remarks
There are two forms of remarks (comments), the REM command and the remark operator (a single quote). The remark operator can be placed anywhere an end-of-statement token can be put including at the beginning of the line where a command is. There are no commands after a remark since all characters are part of the comment up to the end of the line. Remarks were implemented a couple of different ways and while these all worked, these solutions were not very clean (there were multiple checks for the Rem and RemOp tokens in several routines).
The final solution was to implement a new get commands routine, which as the name implies will be able to handle multiple commands on the line separated by colons (which wasn't implemented yet). The new routine contains a loop, which begins by getting a token. If the token is a Rem or RemOp, the loop is exited. Otherwise the token is processed by calling the process command routine (see below). If the terminating token of the command is a RemOp, the loop is exited. Otherwise, for now, the routine returns the terminating token from the command translator routine and the done status.
When the loop is exited in the new get commands routine, which will be due to a Rem or RemOp token, the token is appended to the RPN output list, and the next token is obtained, which should be an end-of-line token unless there is a bug in the parser. The end-of-line token is returned as the terminating token with the done status.
The original get command routine was renamed to the process command routine, which made more sense and this routine won't be public as only the get commands routine will be the only caller. It was modified to receive the first token from the token argument instead of getting the token itself.
The expected results for translator test #15 (Remark tests) needed to be updated due to the change in translation of the PRINT and INPUT commands (the old results were saved). Now all the translator tests pass with the new translator routines.
[commit a3a71526f9]
The final solution was to implement a new get commands routine, which as the name implies will be able to handle multiple commands on the line separated by colons (which wasn't implemented yet). The new routine contains a loop, which begins by getting a token. If the token is a Rem or RemOp, the loop is exited. Otherwise the token is processed by calling the process command routine (see below). If the terminating token of the command is a RemOp, the loop is exited. Otherwise, for now, the routine returns the terminating token from the command translator routine and the done status.
When the loop is exited in the new get commands routine, which will be due to a Rem or RemOp token, the token is appended to the RPN output list, and the next token is obtained, which should be an end-of-line token unless there is a bug in the parser. The end-of-line token is returned as the terminating token with the done status.
The original get command routine was renamed to the process command routine, which made more sense and this routine won't be public as only the get commands routine will be the only caller. It was modified to receive the first token from the token argument instead of getting the token itself.
The expected results for translator test #15 (Remark tests) needed to be updated due to the change in translation of the PRINT and INPUT commands (the old results were saved). Now all the translator tests pass with the new translator routines.
[commit a3a71526f9]
Sunday, August 18, 2013
New Translator – INPUT Statements (Tagged)
The implementation of INPUT statement translation went fairly quickly using the new translation routines. Only a couple of days were required using the new command centric translator routines, compared to well over a month using the old token centric translator routines. This should be an indicator how the implementation should go for the rest of the BASIC commands.
The INPUT statements implementation in the new translator is now complete and version v0.4.4 has been tagged. All tests still pass with the old translator routines. All translator tests pass with the new translator routines with the exception of the REM statements (test #15) as this has not been implemented yet. Some slight cleanup was done with the latest commit along with updating the files for v0.4.4. Implementation of REM statements and operators can now commence in the new translator.
[commit 00815b4bac]
The INPUT statements implementation in the new translator is now complete and version v0.4.4 has been tagged. All tests still pass with the old translator routines. All translator tests pass with the new translator routines with the exception of the REM statements (test #15) as this has not been implemented yet. Some slight cleanup was done with the latest commit along with updating the files for v0.4.4. Implementation of REM statements and operators can now commence in the new translator.
[commit 00815b4bac]
New INPUT Translation
The new INPUT translate routine required several local variables besides a status variable, including an index into the RPN output list of the InputBegin or InputBeginStr code where to insert the input parse codes (see below), a done flag, and an input token. This routine will be used for both the INPUT and INPUT PROMPT commands.
For the INPUT command a new token is created for the InputBegin code. For the INPUT PROMPT command, the get expression routine is called to get a string expression. The done stack top item is dropped and the terminating token is checked. If the terminating token is a comma, the 'Question' sub-code is set in the token, else if the token is not a semicolon, the "expecting operator, semicolon or comma" error is returned. The token is set to the InputBeginStr code.
The index to where the input begin token will be inserted into the RPN output list is obtained by the current count of the number of items in the list. The token is appended to the output and a loop is entered to get each of the input variables.
The get operand routine is called to get any type of variable reference. The next token is obtained. If the token is a comma, the done flag is set to false and the input token is set to the token (to be reused for the input assign token). If the token is a semicolon, the 'Keep' sub-code is set in the command token, the done flag is set to true, the input token is set to the token (for reuse), and the next token is obtained. For all other tokens, the done flag is set to true, and the input token is set to a new token.
The input token is set to the InputAssign code and the process final operand routine is called to process the reference on top of the done stack and set the input token code to the appropriate input assign code for the data type of the reference. A new token is created with the second associated code of the input assign code, which will be the coordinating input parse code. The input parse token is then inserted into the output list at the begin index. The input parse token for each subsequent reference will be inserted in front of the last input parse token, and therefore all the input parse tokens will be in the desired reverse order (making execution easier). The loop continues while the done flag is not set.
Upon exit from the loop, if the status is set to an error, the error is returned. Otherwise, the command token is appended to the output and if terminating token is not an end-of-statement, the "expecting comma, semicolon or end-of-statement" error is returned. (Note the reversal of the "comma" and "semicolon" words in the error message used for the PRINT statement, which makes more sense for the INPUT statement.) Finally the done status is returned. See the commit log for the other minor changes required to implement the INPUT translate routine.
[commit 8b079faf61]
For the INPUT command a new token is created for the InputBegin code. For the INPUT PROMPT command, the get expression routine is called to get a string expression. The done stack top item is dropped and the terminating token is checked. If the terminating token is a comma, the 'Question' sub-code is set in the token, else if the token is not a semicolon, the "expecting operator, semicolon or comma" error is returned. The token is set to the InputBeginStr code.
The index to where the input begin token will be inserted into the RPN output list is obtained by the current count of the number of items in the list. The token is appended to the output and a loop is entered to get each of the input variables.
The get operand routine is called to get any type of variable reference. The next token is obtained. If the token is a comma, the done flag is set to false and the input token is set to the token (to be reused for the input assign token). If the token is a semicolon, the 'Keep' sub-code is set in the command token, the done flag is set to true, the input token is set to the token (for reuse), and the next token is obtained. For all other tokens, the done flag is set to true, and the input token is set to a new token.
The input token is set to the InputAssign code and the process final operand routine is called to process the reference on top of the done stack and set the input token code to the appropriate input assign code for the data type of the reference. A new token is created with the second associated code of the input assign code, which will be the coordinating input parse code. The input parse token is then inserted into the output list at the begin index. The input parse token for each subsequent reference will be inserted in front of the last input parse token, and therefore all the input parse tokens will be in the desired reverse order (making execution easier). The loop continues while the done flag is not set.
Upon exit from the loop, if the status is set to an error, the error is returned. Otherwise, the command token is appended to the output and if terminating token is not an end-of-statement, the "expecting comma, semicolon or end-of-statement" error is returned. (Note the reversal of the "comma" and "semicolon" words in the error message used for the PRINT statement, which makes more sense for the INPUT statement.) Finally the done status is returned. See the commit log for the other minor changes required to implement the INPUT translate routine.
[commit 8b079faf61]
Saturday, August 17, 2013
New INPUT Translation – Preparation
In preparation for the new INPUT translate routine, some minor changes were needed. The first change was add default values to the process final operand routine for the second token and the operand index arguments. The second token argument is used when processing operators where it is given the first token of the operator being processed and this second token is used for binary operators to set its first token when it is pushed on the done stack.
The old translator routines also set the second token to the closing parentheses token for tokens with parentheses, but the new translator routines do not call the process final operand routine for these tokens. Also for the new translator, the operand index will only be used for operators. In addition to operators, the new translator will only use the process final operand routine for special codes like PrintDbl and InputAssignDbl, and these will not require the second operand or operand index arguments.
The other change involved adding a new reference type in additional to None, Variable and All. The new VarDefFn type is to allow variables and defined function references only and not sub-string references (as in sub-string assignments). The Variable reference type was modified to only allow variables and not defined function assignments. The get operand routine was modified to support these changes. The INPUT command will use the Variable reference type since defined functions cannot be input. The get internal function routine was modified to use the new reference type for sub-string assignments (because defined functions of type string will support sub-string assignments, at least this is the current plan).
[commit b670ab38f6] [commit a79f117c52]
The old translator routines also set the second token to the closing parentheses token for tokens with parentheses, but the new translator routines do not call the process final operand routine for these tokens. Also for the new translator, the operand index will only be used for operators. In addition to operators, the new translator will only use the process final operand routine for special codes like PrintDbl and InputAssignDbl, and these will not require the second operand or operand index arguments.
The other change involved adding a new reference type in additional to None, Variable and All. The new VarDefFn type is to allow variables and defined function references only and not sub-string references (as in sub-string assignments). The Variable reference type was modified to only allow variables and not defined function assignments. The get operand routine was modified to support these changes. The INPUT command will use the Variable reference type since defined functions cannot be input. The get internal function routine was modified to use the new reference type for sub-string assignments (because defined functions of type string will support sub-string assignments, at least this is the current plan).
[commit b670ab38f6] [commit a79f117c52]
Associated Code Processing Reorganization
During the design of the INPUT statement translation, some code reorganization was needed, specifically in the Translator find code routine, which processes the done stack top item (that was last added to the RPN output list) to check its data type to see if it matches the expected operand data type of the operator or internal function being processed, and if not looks to see if there is an associated code with that expected data type. If no associated code is found, then checks to see if the data type of the operand can be converted (a hidden convert code is added to the output if it can), or else determines the appropriate error. Upon success, the done stack top item is dropped.
The find code routine had evolved into a somewhat complicated function. It is also used to take a base code (for example PrintDbl, and will also be used for InputAssignType) and change it to an associated code for the data type of the operand on top of the done stack. The functionality of looking up an associated code from a base code for a given data type will also be needed in the Encoder (to be implemented soon now), for example, taking the VarDbl code (that pushes a double variable value to the evaluation stack during run-time), and changing it as needed to VarInt or VarStr for the data type of a token with no parentheses.
Since the future Encoder class should not need to call a Translator class function for this, and since this is really the functionality of the Table class (all the associated code and data type information is in the Table), the part of the find code routine that does this was moved into a new find code routine in the Table class. Some unused code was discovered (part of it even containing a bug, but was causing a problem since the code was not used), which was removed when the code was moved.
The Translator find code routine was renamed to the process done stack top routine to more appropriately describe its functionality. The convert code have need table was also moved from the translator to the table source file. As a result, a new Table convert code access function was needed, that takes a token and data type and returns the convert code needed (including a null code for no convert code or invalid if conversion is not possible). Some comments were also added documenting code that needs to be removed when the old translator routines are finally removed.
[commit b5fa514ca9]
The find code routine had evolved into a somewhat complicated function. It is also used to take a base code (for example PrintDbl, and will also be used for InputAssignType) and change it to an associated code for the data type of the operand on top of the done stack. The functionality of looking up an associated code from a base code for a given data type will also be needed in the Encoder (to be implemented soon now), for example, taking the VarDbl code (that pushes a double variable value to the evaluation stack during run-time), and changing it as needed to VarInt or VarStr for the data type of a token with no parentheses.
Since the future Encoder class should not need to call a Translator class function for this, and since this is really the functionality of the Table class (all the associated code and data type information is in the Table), the part of the find code routine that does this was moved into a new find code routine in the Table class. Some unused code was discovered (part of it even containing a bug, but was causing a problem since the code was not used), which was removed when the code was moved.
The Translator find code routine was renamed to the process done stack top routine to more appropriately describe its functionality. The convert code have need table was also moved from the translator to the table source file. As a result, a new Table convert code access function was needed, that takes a token and data type and returns the convert code needed (including a null code for no convert code or invalid if conversion is not possible). Some comments were also added documenting code that needs to be removed when the old translator routines are finally removed.
[commit b5fa514ca9]
Friday, August 16, 2013
New INPUT Command Design
The design of the INPUT command will be modified slightly. The current design is produces a rather complex translation, which also makes execution complicated. Refer to previous posts for Input Commands particularly February 24, 2011 showing an example translation and February 26, 2011 describing some of the complexity during execution. Unfortunately, simplifying the design much is not possible because of the requirement to handle a statement like INPUT I%,A(I%) where the value of I% needs to be assigned to an inputted value before the reference to A(I%) can be generated since the value of I% needs to be set beforehand. The design can be improved upon for run-time (the goal is for fast execution) especially when dealing with errors entered during input. Consider the previous and new proposed translations for this example statement:
Upon successful input (all values parsed and saved on the input stack), execution proceeds to the next code. Each reference gets pushed to the evaluation stack and each InputAssignType pops the reference, pops a value from the input stack, and assigns the value to the reference.
There is not much left for Input to perform, but it does check for the 'Keep' sub-code (semicolon at the end of the statement to keep the cursor on the same line), and if not set, moves the cursor to the beginning of the next line.
For the INPUT PROMPT command, the beginning of the translation is varied slightly to allow for the prompt string expression. The beginning of the line contains the tokens for the string expression (before the first InputParseType). There is an InputBeginStr instead of InputBegin, which could have an optional 'Question' sub-code depending if the prompt expression was followed by a semicolon (no question) or comma (question). At run-time, this code will pop the result of the string expression from the evaluation stack, and proceeds the same as InputBegin except with the specified prompt.
InputBegin InputParseInt InputParseDbl'End' I%<ref> InputAssignInt I% A(<ref> InputAssignDbl InputThe proposed translation is nearly identical except InputBegin now appears after all the InputParseType codes, which are now reversed, and an 'End' sub-code is no longer needed. At run-time, each InputParseType pushes a data type to an input stack. InputBegin starts by issuing the prompt, gets the input, and processes the input stack without popping the items. Each item on the input stack contains a data type and an input value, which is filled in as each value is successfully parsed. If an error is detected, the error is displayed and the character at which the error occurred is highlighted. At which point, the user is given the opportunity to edit the input and try again.
InputParseDbl InputParseInt InputBegin I%<ref> InputAssignInt I% A(<ref> InputAssignDbl Input
Upon successful input (all values parsed and saved on the input stack), execution proceeds to the next code. Each reference gets pushed to the evaluation stack and each InputAssignType pops the reference, pops a value from the input stack, and assigns the value to the reference.
There is not much left for Input to perform, but it does check for the 'Keep' sub-code (semicolon at the end of the statement to keep the cursor on the same line), and if not set, moves the cursor to the beginning of the next line.
For the INPUT PROMPT command, the beginning of the translation is varied slightly to allow for the prompt string expression. The beginning of the line contains the tokens for the string expression (before the first InputParseType). There is an InputBeginStr instead of InputBegin, which could have an optional 'Question' sub-code depending if the prompt expression was followed by a semicolon (no question) or comma (question). At run-time, this code will pop the result of the string expression from the evaluation stack, and proceeds the same as InputBegin except with the specified prompt.
Thursday, August 15, 2013
New Translator – PRINT Statements (Tagged)
A minor parser issue was discovered with how the length of double word command tokens were being set, which was noticed with the "not yet implemented" errors not pointing at the entire INPUT PROMPT command. The token length was only being set to the length of the second word plus one for the space. This was corrected by setting the token length to the current position (first character after the second word) minus the starting column of the first word. This also takes into account if more than one space is entered between the two words.
The PRINT statements implementation in the new translator is now complete and version v0.4.3 has been tagged. All tests pass with the old translator routines (after some corrections were made). All translator tests pass with the new translator routines with the exception of the INPUT and REM statements as these have not been implemented yet. Some minor cleanup was done with the latest commit along with updating the files for v0.4.3 (see the commit log for details). Implementation of INPUT statements can now commence in the new translator.
[commit bd608f5cf3] [commit 0e316f71ce]
The PRINT statements implementation in the new translator is now complete and version v0.4.3 has been tagged. All tests pass with the old translator routines (after some corrections were made). All translator tests pass with the new translator routines with the exception of the INPUT and REM statements as these have not been implemented yet. Some minor cleanup was done with the latest commit along with updating the files for v0.4.3 (see the commit log for details). Implementation of INPUT statements can now commence in the new translator.
[commit bd608f5cf3] [commit 0e316f71ce]
Wednesday, August 14, 2013
New Translator – PRINT Statements
The implementation of the PRINT translate routine using the new translator routines was mostly straight forward, but some time was spent getting the right errors reported at the correct tokens (the previous post dealt with some of these issues in the translator routines). Three local variables were needed to hold the pointer to the last semicolon token (last one is used instead of the command token if at the end of the statement), a separator flag (indicates when a semicolon is allowed) and a print function flag (only used for error reporting). The routine then enters a loop until the end of the statement.
The loop begins by calling the get expression routine for a None data type to allow for print functions or an empty expression. If a non numeric parser error is returned, then the appropriate error is returned. This and other errors cause the loop exit. If the done stack is not empty and the data type of the top item is None, then the print function flag is set to true. Otherwise a new print code token is created and the process final operand routine is called to get the correct associated code for the top item and the print function flag is cleared. The top item (either a print function or the new print code token) is dropped from the done stack (both have already been added to the RPN output list). The separator flag is set, the last semicolon token is deleted and the pointer is set to null.
If the next token (the terminating token from the get expression routine) is a comma, then if the last semicolon token pointer is set, then a comma is not allowed and the appropriate error is set and the loop exits. Otherwise, the comma token is appended to the RPN output list. The last semicolon token is deleted and the pointer is set to null. The separator flag is cleared (no semicolon is allowed). The loop continues with the next expression.
If the next token is a semicolon, then if the separator flag is not set, then a second semicolon is not allowed and the appropriate error is set depending if the last semicolon token pointer is set and the loop exits. The last semicolon token is deleted and the pointer is set to the current semicolon token. The loop continues with the next expression.
For any other terminating token, the loop exits. Otherwise, the separator flag is cleared for the next expression and the loop continues. If the loop exited with an error, the last semicolon token and command token pointers are deleted and the error is returned. If the last semicolon token pointer is set, the command token is deleted and is set to the last semicolon token. The command token is appended to the RPN output list.
If the terminating token is not an end-of-statement token, then the appropriate error is returned depending on the print function flag. Otherwise a done status is returned. The PRINT translate function was implemented in the new basic/print.cpp source file, which was added the CMake build file, and was added to the table for the PRINT command.
[commit 3fc94d4b18]
The loop begins by calling the get expression routine for a None data type to allow for print functions or an empty expression. If a non numeric parser error is returned, then the appropriate error is returned. This and other errors cause the loop exit. If the done stack is not empty and the data type of the top item is None, then the print function flag is set to true. Otherwise a new print code token is created and the process final operand routine is called to get the correct associated code for the top item and the print function flag is cleared. The top item (either a print function or the new print code token) is dropped from the done stack (both have already been added to the RPN output list). The separator flag is set, the last semicolon token is deleted and the pointer is set to null.
If the next token (the terminating token from the get expression routine) is a comma, then if the last semicolon token pointer is set, then a comma is not allowed and the appropriate error is set and the loop exits. Otherwise, the comma token is appended to the RPN output list. The last semicolon token is deleted and the pointer is set to null. The separator flag is cleared (no semicolon is allowed). The loop continues with the next expression.
If the next token is a semicolon, then if the separator flag is not set, then a second semicolon is not allowed and the appropriate error is set depending if the last semicolon token pointer is set and the loop exits. The last semicolon token is deleted and the pointer is set to the current semicolon token. The loop continues with the next expression.
For any other terminating token, the loop exits. Otherwise, the separator flag is cleared for the next expression and the loop continues. If the loop exited with an error, the last semicolon token and command token pointers are deleted and the error is returned. If the last semicolon token pointer is set, the command token is deleted and is set to the last semicolon token. The command token is appended to the RPN output list.
If the terminating token is not an end-of-statement token, then the appropriate error is returned depending on the print function flag. Otherwise a done status is returned. The PRINT translate function was implemented in the new basic/print.cpp source file, which was added the CMake build file, and was added to the table for the PRINT command.
[commit 3fc94d4b18]
Tuesday, August 13, 2013
Parser Error And None Data Type Handling
The PRINT statement translation implementation is almost complete, but a few more issues were discovered with how parser errors and the None data type are handled. Support for the None data type was added for the print functions (TAB and SPC), but also to support being able to call the get expression routine and have it allow for no expression, just a terminating token. This is needed for PRINT statements that have commas with no expressions in between them (for instance, to skip over two columns).
The get token routine was modified to accept a new No data type value, which will now be the default value. This data type will only be used to call the get token routine when a non-operand token is requested. This was necessary since the None type is now being used for operands.
The get operand routine was modified for command and operator tokens, which are normally invalid for an operand and an "expected XX expression or variable" error is returned, now returns a Done status if the data type passed is None. This allows for no expressions to be returned and the command or operator token becomes the terminating token from the get expression routine (it is up to the PRINT translate routine to determine the validity of the token).
The get expression routine was modified to carry the None data type into the loop in the expected data type variable. This allows the None data type to be passed to the get operand routine upon the first call (to allow for print functions or no expression) so the secondary operand data type variable is no longer needed. After the first operator, the expected data type variable is set accordingly for the operator.
However, for parentheses expressions, the data type needs to be changed to Any before recursively calling the get expression routine since neither a print function nor no expression is valid inside the parentheses. Also, if a parser error is returned, the status is only set to the "expected operator or closing parentheses" error if the token does not have a data type, because a numeric parser error needs to be reported as such.
Finally, the scheme to leave an internal function of parentheses token on the hold stack when an error is detected so that the caller can check if the top of the hold stack is not the null token was flawed. Without going into details, it turned out this scheme was not necessary anyway and was removed. Quite a few more parser error tests were added to translator test #14 for testing additional error cases.
[commit 86292a56c0]
The get token routine was modified to accept a new No data type value, which will now be the default value. This data type will only be used to call the get token routine when a non-operand token is requested. This was necessary since the None type is now being used for operands.
The get operand routine was modified for command and operator tokens, which are normally invalid for an operand and an "expected XX expression or variable" error is returned, now returns a Done status if the data type passed is None. This allows for no expressions to be returned and the command or operator token becomes the terminating token from the get expression routine (it is up to the PRINT translate routine to determine the validity of the token).
The get expression routine was modified to carry the None data type into the loop in the expected data type variable. This allows the None data type to be passed to the get operand routine upon the first call (to allow for print functions or no expression) so the secondary operand data type variable is no longer needed. After the first operator, the expected data type variable is set accordingly for the operator.
However, for parentheses expressions, the data type needs to be changed to Any before recursively calling the get expression routine since neither a print function nor no expression is valid inside the parentheses. Also, if a parser error is returned, the status is only set to the "expected operator or closing parentheses" error if the token does not have a data type, because a numeric parser error needs to be reported as such.
Finally, the scheme to leave an internal function of parentheses token on the hold stack when an error is detected so that the caller can check if the top of the hold stack is not the null token was flawed. Without going into details, it turned out this scheme was not necessary anyway and was removed. Quite a few more parser error tests were added to translator test #14 for testing additional error cases.
[commit 86292a56c0]
Sunday, August 11, 2013
Print Function Support
The PRINT statement consists of expressions separated by semicolons or commas. Commas can also be used with no expressions between them. The expressions may also contain the print functions TAB and SPC. These functions are distinguished from other functions in that they have no return data type. The PRINT translate function could specifically handle these print functions since they are only use in a PRINT statement.
However, the get expression routine also needs to be able handle these print functions in at least to report errors when they are used incorrectly in expressions. It currently does this simply by the fact that the None data type is not acceptable by any operator or data type request (an expression with a data type of None is not currently requested).
Instead of having the PRINT translate routine handle the print functions, the get expression routine was modified to accept the None data type, which affected two things. This allows print functions to be returned. It also allows no expression to be returned (just the next token as the terminating token), which is needed in the PRINT statement for commas. This will be detected by the done stack being empty. The PRINT translate routine will be the only caller with the None data type.
The new translator was changed back to only reporting the TAB( or SPC( token when used incorrectly in expressions. This was necessary due to parser and other errors inside the parentheses of the print function as the error was reported at the wrong place. Consider the expression:
[commit 9502d5b183]
However, the get expression routine also needs to be able handle these print functions in at least to report errors when they are used incorrectly in expressions. It currently does this simply by the fact that the None data type is not acceptable by any operator or data type request (an expression with a data type of None is not currently requested).
Instead of having the PRINT translate routine handle the print functions, the get expression routine was modified to accept the None data type, which affected two things. This allows print functions to be returned. It also allows no expression to be returned (just the next token as the terminating token), which is needed in the PRINT statement for commas. This will be detected by the done stack being empty. The PRINT translate routine will be the only caller with the None data type.
The new translator was changed back to only reporting the TAB( or SPC( token when used incorrectly in expressions. This was necessary due to parser and other errors inside the parentheses of the print function as the error was reported at the wrong place. Consider the expression:
A = TAB(B$)An "expected numeric expression" error was reported at the B$ token, but should point to the TAB( token as it doesn't belong in the expression in the first place. Several more statements were added to translator test #14 that included various parser errors in print functions used incorrectly. See the commit log for more details of the changes made.
[commit 9502d5b183]
Saturday, August 10, 2013
Old Translator Issues With Token Caching
While debugging the new PRINT translate routine, I realized that the old translator routines were never tested with new token caching and error reporting code, because when they were, crashes and errors occurred. Two small problems were found and were corrected.
The first issue occurred when an error is detected and the error token was not the current token obtained from the parser (meaning the error token had already been added to the output and cannot be deleted). The old translator routine obtained a new token and copied the error token into it before calling the set error token routine, which called the output list set error routine and then deleted the token. When the token was copied into the new token obtained, the index of the original token was copied over the new token's index confusing the used token handling. This was corrected by calling the output list set error routine directly (which only grabs the column and length of the error token) so no temporary token is needed.
The second issue occurred when there was a REMARK operator (') at the beginning of a line. The Null token put on the bottom of the hold stack was not removed. This was detected as a token leak and the detection code reported and deleted the token. Unfortunately, the translator still had the token on the hold stack. This did not cause a problem until the next command with an error was processed. When an error occurs, the the hold stack is empty and its tokens are deleted. When it got to the deleted Null token, the index was garbage (because the memory for the token had been reused) and a segmentation fault occurred.
The REMARK operator token handler was modified to pop and delete the Null token from the hold stack if currently in command mode. If not in command mode, then another part of the translator correctly handled the the Null token. Curiously for this situation, the EOL token at the end of the line is never obtained and processed. Therefore, this fix is a hack, but since the old translator routines will be abandoned, there is no reason for a correct fix.
One last minor issue was that when token leaks or extra deletes were reported, if the token was a Null or End-of-line token, a blank token was output with just an index. The "NULL" and "EOL" strings were added to the debug name entry in the table to identify these blank tokens.
[commit cf93288fcb]
The first issue occurred when an error is detected and the error token was not the current token obtained from the parser (meaning the error token had already been added to the output and cannot be deleted). The old translator routine obtained a new token and copied the error token into it before calling the set error token routine, which called the output list set error routine and then deleted the token. When the token was copied into the new token obtained, the index of the original token was copied over the new token's index confusing the used token handling. This was corrected by calling the output list set error routine directly (which only grabs the column and length of the error token) so no temporary token is needed.
The second issue occurred when there was a REMARK operator (') at the beginning of a line. The Null token put on the bottom of the hold stack was not removed. This was detected as a token leak and the detection code reported and deleted the token. Unfortunately, the translator still had the token on the hold stack. This did not cause a problem until the next command with an error was processed. When an error occurs, the the hold stack is empty and its tokens are deleted. When it got to the deleted Null token, the index was garbage (because the memory for the token had been reused) and a segmentation fault occurred.
The REMARK operator token handler was modified to pop and delete the Null token from the hold stack if currently in command mode. If not in command mode, then another part of the translator correctly handled the the Null token. Curiously for this situation, the EOL token at the end of the line is never obtained and processed. Therefore, this fix is a hack, but since the old translator routines will be abandoned, there is no reason for a correct fix.
One last minor issue was that when token leaks or extra deletes were reported, if the token was a Null or End-of-line token, a blank token was output with just an index. The "NULL" and "EOL" strings were added to the debug name entry in the table to identify these blank tokens.
[commit cf93288fcb]
Thursday, August 8, 2013
New PRINT Implementation (Revised)
Once implementation of the new PRINT translation was started, I realized that the design at run-time was more complicated than it needed to be. Most of the original design appeared to be best after all, with a few minor changes. When it comes to run-time, the original design would be more efficient and would not need the extra complexity of a command stack. With print codes immediately after each print item, the print code would simply pop the value from the evaluation stack and print it.
The changes to the original design are how semicolons and the final PRINT command code are handled. Semicolon sub-codes were added to print functions codes except when they appeared at end of a statement. Multiple semicolons were allowed, marked with semicolon sub-codes or extra semicolon tokens with more than one extra semicolons. A PRINT command code would only be added to the end of the statement if there was no print token (semicolon, comma, TAB or SPC) at the end.
Semicolon sub-codes will no longer be needed. Semicolons will be assumed between each print item unless there is a comma between them. Multiple unnecessary semicolons will no long be permitted as these add nothing. If there is a semicolon at the end of the line, then a semicolon code will be added to the end of the line instead of a PRINT code. At run-time, the semicolon will no nothing, but when the line is recreated, the semicolon will produce the PRINT keyword in additional to the semicolon at the end of the line.
The changes to the original design are how semicolons and the final PRINT command code are handled. Semicolon sub-codes were added to print functions codes except when they appeared at end of a statement. Multiple semicolons were allowed, marked with semicolon sub-codes or extra semicolon tokens with more than one extra semicolons. A PRINT command code would only be added to the end of the statement if there was no print token (semicolon, comma, TAB or SPC) at the end.
Semicolon sub-codes will no longer be needed. Semicolons will be assumed between each print item unless there is a comma between them. Multiple unnecessary semicolons will no long be permitted as these add nothing. If there is a semicolon at the end of the line, then a semicolon code will be added to the end of the line instead of a PRINT code. At run-time, the semicolon will no nothing, but when the line is recreated, the semicolon will produce the PRINT keyword in additional to the semicolon at the end of the line.
Wednesday, August 7, 2013
New PRINT At Run-Time (Retracted)
After considering this design, it was decided to be more
complicated than it needed to be, so this post has been retracted, see
revised post.
Consider the previous example statement and its new translation:
When the PRINT (or semicolon) code is executed, it will process the command stack, except it will start at the bottom of the stack instead of popping items from the top. This is possible because the QStack class is derived from the QVector class, which allows its elements to be access by index. The values of the evaluation stack will also be accessed the same way.
When the PRINT (or semicolon) begins, it will start with indexes (one for each stack) set to zero and will process each item on the command stack. For the specific print type code, it will access the value at the evaluation stack index, output it, and increment the index. For a comma code, it will output the appropriate number of spaces (leaving the evaluation stack index as is). For a print function (TAB or SPC), it will access the value at the evaluation stack index, output the appropriate number of spaces, and increment the index.
At the end (top) of the command stack, it will clear both stacks, return the indexes to zero (for the next command), and for the PRINT code only, will advance to the next line. This scheme works because both the evaluation and command stacks will be empty at the beginning of each command.
Consider the previous example statement and its new translation:
PRINT A,B$;TAB(20);C%At run-time, in additional to the evaluation stack used to hold values of operands and results, there will also be a command stack (not to be confused with the current command stack used by the old translator routines, but not needed by the new translator routines). The special print codes (specific print type codes, comma, and print functions), will only push themselves to the command stack.
A B$ 20 C% PrintDbl , PrintStr TAB( PrintInt PRINT
When the PRINT (or semicolon) code is executed, it will process the command stack, except it will start at the bottom of the stack instead of popping items from the top. This is possible because the QStack class is derived from the QVector class, which allows its elements to be access by index. The values of the evaluation stack will also be accessed the same way.
When the PRINT (or semicolon) begins, it will start with indexes (one for each stack) set to zero and will process each item on the command stack. For the specific print type code, it will access the value at the evaluation stack index, output it, and increment the index. For a comma code, it will output the appropriate number of spaces (leaving the evaluation stack index as is). For a print function (TAB or SPC), it will access the value at the evaluation stack index, output the appropriate number of spaces, and increment the index.
At the end (top) of the command stack, it will clear both stacks, return the indexes to zero (for the next command), and for the PRINT code only, will advance to the next line. This scheme works because both the evaluation and command stacks will be empty at the beginning of each command.
New PRINT Implementation (Retracted)
After considering this design, it was decided to be more complicated than it needed to be, so this post has been retracted, see revised post.
The run-time design of the PRINT statement was reconsidered. The old scheme inserted specific type print codes into the translation with an optional PRINT code at the end depending if the line ended with a semicolon, comma or print function (TAB or SPC). Consider this example statement and its old translation:
A semicolon at the end of the line needs special consideration. If there is a semicolon at the end of the line, it will take the place of the PRINT code. At run-time, both perform the same actions except only PRINT advances to the next line. A semicolon can be used this way since no other statement needs a semicolon token. A 'Keep' sub-code or a new PrintKeep code could have been used, but why have a sub-code check at run-time or create a new code when a semicolon token can be used.
The next post will contain the details of how this new PRINT translation will work at run-time...
The run-time design of the PRINT statement was reconsidered. The old scheme inserted specific type print codes into the translation with an optional PRINT code at the end depending if the line ended with a semicolon, comma or print function (TAB or SPC). Consider this example statement and its old translation:
PRINT A,B$;TAB(20);C%The specific type print codes pop a value from the evaluation stack and outputs it. The comma token outputs spaces to advance the cursor to the next column (traditionally groups of 8 characters). The TAB print function outputs spaces to advance to specified column. And the final PRINT advances to the next line. If the line ended with a semicolon, comma or print function, the final PRINT code is not present. The new translation will look like this (details to follow):
A PrintDbl , B$ PrintStr 20 TAB(';' C% PrintInt PRINT
A B$ 20 C% PrintDbl , PrintStr TAB( PrintInt PRINTThe final PRINT code is always present, and at run time will know when to advance to the next line (depends on the code immediately preceding it). Semicolon tokens (or a sub-code) is not necessary and is implied between the items to print. Multiple semicolons, though allowed in the old translator (the semicolon sub-code and any semicolon tokens would do nothing at run-time), will not be permitted as there is no reason to allow them (unlike multiple commas).
A semicolon at the end of the line needs special consideration. If there is a semicolon at the end of the line, it will take the place of the PRINT code. At run-time, both perform the same actions except only PRINT advances to the next line. A semicolon can be used this way since no other statement needs a semicolon token. A 'Keep' sub-code or a new PrintKeep code could have been used, but why have a sub-code check at run-time or create a new code when a semicolon token can be used.
The next post will contain the details of how this new PRINT translation will work at run-time...
Subscribe to:
Posts (Atom)