User functions can take two forms, a full user function (using the FUNCTION syntax) or a simple define function (using the DEF FN syntax). A full user function will pass arguments by reference. This only applies to variables and array elements. Results of expressions will be passed by value. To keep things consistent internally, a temporary value will be allocated and a pointer to it will be passed, so all arguments will be a reference. To force a single variable or array element to be passed by value, it can be surrounded by a set of parentheses.
Define functions will have two forms, a single line form that looks like an assignment and a multiple line form (that will end with an END DEF statement have one or more assignments for the return value). Unlike full user functions, arguments to define functions will only be passed by value. This make sense for the single line form since the arguments can't be assigned inside the function, and to keep the internal code consistent between the two forms, the multiple line form will also have arguments passed by value. Arguments will be local variables, so any assignments will not affect the original variables.
This is the same method of argument passing used by QBASIC, and seems to be a reasonable design choice, so will also be used for this project. Subroutines (the SUB syntax) will use the same pass by reference scheme as full user functions. The passing of entire arrays will be dealt with later. QBASIC allows this by listing the array name followed by an opening and closing parentheses with no subscripts. This same syntax may be used for this project.
To handle function calls in the translator, the reference flag of an operand is set if it is an identifier with or without parentheses. These tokens could end up being function calls, but this will be handled by the encoder. Identifiers with parentheses could also end up being an array, where its arguments are integer subscripts. Again this will be handled by the encoder, which will add any needed internal convert to integer codes (for double subscripts) or report errors for string subscripts. The reference flag in any subscripts will be ignored. Regardless of whether the identifier is an array or function, the translator will attach a pointer to each argument for the encoder.
Friday, July 5, 2013
Thursday, July 4, 2013
New Translator – Internal Functions
To support internal functions, both with arguments (parentheses) and without (no parentheses), the get operand function was updated. Internal functions without arguments are treated the same as constants and identifiers with no parentheses where the tokens are simply added to the RPN output list and pushed to the done stack.
For internal functions with parentheses, a new get internal function routine was implemented and called. This function starts by pushing the internal function token to the hold stack, which being of low precedence, will create a border as the expressions of the arguments are processed. After getting the number arguments, it loops for each argument by first calling the get expression routine and then checking the terminating token for a comma or closing parentheses.
For a comma, if at the last argument, then an error is returned if the internal function code does not have multiple entries (for examine, the MID$ function with two or three arguments). Otherwise the code is changed to the code with an additional argument. The comma token is deleted and the existing find code routine is called to process and check the argument.
For a closing parentheses, if not at the last argument, then an error is returned. Otherwise the existing process final operand function is called to process the final argument, which appends the internal function token to the RPN output list upon success. The internal function token is then dropped from the hold stack.
If the terminating token is neither a comma nor a closing parentheses, then the appropriate error is returned depending which argument it is at taking into account whether the internal function has multiple entries.
Expression test #3 now passes with the new translator routines except for the three lines that contain identifiers with parentheses (an array or a user function), which has not yet been implemented in the new translator. Expression test #4 also now passes successfully. Since none of the expression tests contained a function with no arguments, a new expression was added to test #4 containing the RND function.
[commit 81d3137531]
For internal functions with parentheses, a new get internal function routine was implemented and called. This function starts by pushing the internal function token to the hold stack, which being of low precedence, will create a border as the expressions of the arguments are processed. After getting the number arguments, it loops for each argument by first calling the get expression routine and then checking the terminating token for a comma or closing parentheses.
For a comma, if at the last argument, then an error is returned if the internal function code does not have multiple entries (for examine, the MID$ function with two or three arguments). Otherwise the code is changed to the code with an additional argument. The comma token is deleted and the existing find code routine is called to process and check the argument.
For a closing parentheses, if not at the last argument, then an error is returned. Otherwise the existing process final operand function is called to process the final argument, which appends the internal function token to the RPN output list upon success. The internal function token is then dropped from the hold stack.
If the terminating token is neither a comma nor a closing parentheses, then the appropriate error is returned depending which argument it is at taking into account whether the internal function has multiple entries.
Expression test #3 now passes with the new translator routines except for the three lines that contain identifiers with parentheses (an array or a user function), which has not yet been implemented in the new translator. Expression test #4 also now passes successfully. Since none of the expression tests contained a function with no arguments, a new expression was added to test #4 containing the RND function.
[commit 81d3137531]
New Translator – Checking Token Codes
A minor coding issue was discovered in code that needs to check the code in a token. This is more an issue with the new translator routines because they will be checking tokens that could be of any token type. The old translator routines already knew the token type before checking the code.
Since it is desirable to make the code as easy to implement as possible, the Token class isCode() function was modified to make sure the token is the correct type before checking the code. The correct type is any type that has a table entry. If the calling code already knows the token is a type that has a valid code, then it can use the Token class code() access function and compare to the code directly.
[commit 9bbd1e319f]
Since it is desirable to make the code as easy to implement as possible, the Token class isCode() function was modified to make sure the token is the correct type before checking the code. The correct type is any type that has a table entry. If the calling code already knows the token is a type that has a valid code, then it can use the Token class code() access function and compare to the code directly.
[commit 9bbd1e319f]
Wednesday, July 3, 2013
New Translator – Parentheses
To support simple parenthetical expressions, the get expression routine was modified to look for an open parentheses token just before checking for a unary operator. The open parentheses token is pushed to the hold stack so that when the expression is processed by recursively calling get expression, only operators from inside the expression will be popped from the hold stack (open parentheses has a very low precedence). After processing the closing parentheses, get expression continues on by getting and processing a binary operator of end-of-expression token.
The recursive get expression will terminate upon a closing parentheses token. In the table, the end expression flag had to be added to the closing parentheses code to cause this termination. Upon return, the terminating token is checked for a closing parentheses, otherwise an "expected operator or closing parentheses" error is returned. The open parentheses token is popped from the hold stack and the open and closing parentheses tokens are assigned to the first and last tokens of the item on top of the done stack (used for error reporting; details of this design start with posts on January 16, 2011).
The closing parentheses is saved as a pending parentheses token for later checking. This token is marked as being used as a last operand and as the pending parentheses (to prevent it from being prematurely deleted). The precedence of the last token added to the output list (which is on top of the done stack) is saved for when the pending parentheses token is later checked. If this token is an operator, the precedence is obtained from the table, otherwise it is set to the highest precedence (parentheses around not operators are never necessary).
A new check pending parentheses routine was added starting with the existing do pending parentheses routine (renamed more appropriately instead of adding a '2', but will still eventually replace the old routine). These routines check if parentheses entered are not necessary, in which case, a flag is set in the translated so that these will be reproduced. Necessary parentheses are implied by the translated RPN format. Parentheses necessity is determined by the precedence of the operators (see posts starting on March 21, 2010 for details).
The new check pending parentheses routine is called from two locations in the new process operator routine, one when an operator is popped from the hold stack before its final operand is processed and one for a new operator before it is checked for the end-of-expression or its first operand is processed. The check pending parentheses routine determines if parentheses are not necessary by checking if the precedence of last token added is greater than the current operator or of equal precedence if th operator was just popped from the hold stack. There is a new popped argument to determine from which (the old routine used the current translator state).
Expression test #2 (parentheses tests) now passes with the new translator routines, but one change was needed for the unexpected closing parentheses test that reported the "expected operator or end-of-expression" error (internally named NoOpenParen). When the wording of this error was changed, it should been been removed and the "expected operator or end-of-statement" error used instead. This error needs to be replaced by the caller as appropriate for the command in which it is detected (see post from Tuesday).
[commit d3d57f2235]
The recursive get expression will terminate upon a closing parentheses token. In the table, the end expression flag had to be added to the closing parentheses code to cause this termination. Upon return, the terminating token is checked for a closing parentheses, otherwise an "expected operator or closing parentheses" error is returned. The open parentheses token is popped from the hold stack and the open and closing parentheses tokens are assigned to the first and last tokens of the item on top of the done stack (used for error reporting; details of this design start with posts on January 16, 2011).
The closing parentheses is saved as a pending parentheses token for later checking. This token is marked as being used as a last operand and as the pending parentheses (to prevent it from being prematurely deleted). The precedence of the last token added to the output list (which is on top of the done stack) is saved for when the pending parentheses token is later checked. If this token is an operator, the precedence is obtained from the table, otherwise it is set to the highest precedence (parentheses around not operators are never necessary).
A new check pending parentheses routine was added starting with the existing do pending parentheses routine (renamed more appropriately instead of adding a '2', but will still eventually replace the old routine). These routines check if parentheses entered are not necessary, in which case, a flag is set in the translated so that these will be reproduced. Necessary parentheses are implied by the translated RPN format. Parentheses necessity is determined by the precedence of the operators (see posts starting on March 21, 2010 for details).
The new check pending parentheses routine is called from two locations in the new process operator routine, one when an operator is popped from the hold stack before its final operand is processed and one for a new operator before it is checked for the end-of-expression or its first operand is processed. The check pending parentheses routine determines if parentheses are not necessary by checking if the precedence of last token added is greater than the current operator or of equal precedence if th operator was just popped from the hold stack. There is a new popped argument to determine from which (the old routine used the current translator state).
Expression test #2 (parentheses tests) now passes with the new translator routines, but one change was needed for the unexpected closing parentheses test that reported the "expected operator or end-of-expression" error (internally named NoOpenParen). When the wording of this error was changed, it should been been removed and the "expected operator or end-of-statement" error used instead. This error needs to be replaced by the caller as appropriate for the command in which it is detected (see post from Tuesday).
[commit d3d57f2235]
Tuesday, July 2, 2013
New Translator – Unary Operator Issue
An issue was discovered where if the token after an operand was a unary operator (this token should be a binary operator or a token terminating the expression), the process operator routine incorrectly processed the unary operator assuming it was a binary operator causing the code to malfunction.
A check was added after getting this token where if it is a unary operator, the "expected binary operator or end-of-statement" error is returned. This error, like the "expected operator or end-of-statement" error, will also need to be changed to an appropriate error by the caller. A test for this was added to expression test #1.
[commit 261c9647df]
A check was added after getting this token where if it is a unary operator, the "expected binary operator or end-of-statement" error is returned. This error, like the "expected operator or end-of-statement" error, will also need to be changed to an appropriate error by the caller. A test for this was added to expression test #1.
[commit 261c9647df]
New Translator – Parser Errors
There are currently two types of parser errors, an unrecognizable character and an incorrect number constant (of which there are five different ones). The number constant errors may have an alternate column (for example, when there is an error in exponent of a floating point number, the alternate column points to the error and the column points to the beginning of the number) or a length more than one (for example, two consecutive decimal points at the beginning).
When the translator is expecting an operand, the number constant errors should be reported as is (for example, pointing to the bad exponent or both decimal points). However, when the translator is expecting some other token (like an operator), a different error appropriate for the situation (for example, an "expecting an operator or end-of-statement" error) should be reported and it should be pointing to the only the first character of the bad token.
The get token routine was modified to take a desired data type argument instead of an operand flag. A none data type indicates that the token desired is not an operand. If the token obtained has an error, then it is marked as unused (it needs to be deleted). If not getting an operand token and the data type of the token is double (indicating a number constant error), then the token length is set to one (the error will point to only the first character of the token). For the error to return, if getting an operand token and the data type of the token is not double (indicating a parser error), the error is set to the appropriate "expecting XX expression" error for the desired data type argument, otherwise a generic parser error is returned (so the caller will used the number constant error message in the token).
The loop in the get expression routine was modified to eliminate duplicate code before the loop and at the end of the loop. At the end of the loop, the token pointer is set to null to force getting a new token at the beginning of the loop (I wish there were a way to prevent having this extra check, but there isn't without resorting to a goto statement and a label). Each of the return statements were changed to break statements with the final return at the end outside of the loop.
Also in this routine, the statement for getting the operand and next token was broken into two so that the error can be changed to an "expecting operator or end-of-statement" error when get token returns an error (any parser error). This error will need to be changed to the appropriate error by the caller. For example, in an IF statement when it calls to get its expression, if this error occurs, the error needs to be changed to an "expected operator or THEN" error.
In the new translate routine, when setting the error message in the RPN output list, the error string is obtained from the token for a parser error instead of from the error status. This also needed to be done before an unused token is deleted (tokens with parser errors are marked as unused).
Finally, the set operand state function of the parser was removed. This was always and only called before calling the token routine, so an operand state argument was added to this function. Several parser error tests were added to expression test #1. It was also discovered that the autoenums.h include file was not always being regenerated when the Token class source file was modified, which turned out to be a wrong dependency listed in the CMake build file.
[commit a194cdbfb6] [commit 348ec2230b]
When the translator is expecting an operand, the number constant errors should be reported as is (for example, pointing to the bad exponent or both decimal points). However, when the translator is expecting some other token (like an operator), a different error appropriate for the situation (for example, an "expecting an operator or end-of-statement" error) should be reported and it should be pointing to the only the first character of the bad token.
The get token routine was modified to take a desired data type argument instead of an operand flag. A none data type indicates that the token desired is not an operand. If the token obtained has an error, then it is marked as unused (it needs to be deleted). If not getting an operand token and the data type of the token is double (indicating a number constant error), then the token length is set to one (the error will point to only the first character of the token). For the error to return, if getting an operand token and the data type of the token is not double (indicating a parser error), the error is set to the appropriate "expecting XX expression" error for the desired data type argument, otherwise a generic parser error is returned (so the caller will used the number constant error message in the token).
The loop in the get expression routine was modified to eliminate duplicate code before the loop and at the end of the loop. At the end of the loop, the token pointer is set to null to force getting a new token at the beginning of the loop (I wish there were a way to prevent having this extra check, but there isn't without resorting to a goto statement and a label). Each of the return statements were changed to break statements with the final return at the end outside of the loop.
Also in this routine, the statement for getting the operand and next token was broken into two so that the error can be changed to an "expecting operator or end-of-statement" error when get token returns an error (any parser error). This error will need to be changed to the appropriate error by the caller. For example, in an IF statement when it calls to get its expression, if this error occurs, the error needs to be changed to an "expected operator or THEN" error.
In the new translate routine, when setting the error message in the RPN output list, the error string is obtained from the token for a parser error instead of from the error status. This also needed to be done before an unused token is deleted (tokens with parser errors are marked as unused).
Finally, the set operand state function of the parser was removed. This was always and only called before calling the token routine, so an operand state argument was added to this function. Several parser error tests were added to expression test #1. It was also discovered that the autoenums.h include file was not always being regenerated when the Token class source file was modified, which turned out to be a wrong dependency listed in the CMake build file.
[commit a194cdbfb6] [commit 348ec2230b]
Sunday, June 30, 2013
New Translator – Simple Expressions (Tagged)
The implementation for simple expressions is now complete in the new translator and version v0.4.0 has been tagged. Note that expression test #1 currently fails with the regression and memory test scripts because of problems with the expression mode in the old translator routines. The new translator routines run this test successfully, but should not yet be used on any of the other expression tests. Also, parser errors are not yet handled correctly by the new translator routines, but this is going to require a major change to how token errors are handled.
[commit 552aa6176a]
[commit 552aa6176a]
New Translator – Memory Leaks
Several of the error conditions caused a memory leak because the token at which the error occurred was not deleted. The error token can't just be deleted because it may be in the RPN output list. Tokens in the output list will be deleted when the list is clears when an error is detected. A method was needed to determine when an error token should be deleted.
This was accomplished by added a new UnUsed sub-code. At the locations where an error detected, the routine setting the error needs to set this sub-code if the token has not been added to the output list. This sub-code was set in two places, one in the get operand routine when there is a command or operator token, and the other in the translate routine that called the get expression routine when the terminating token is not the end-of-line token.
One other problem that caused a uninitialized variable used error from valgrind was also in the translate routine when the terminating token is checked for the end-of-line token. The check also needed to test if the token has a table entry before checking the token for the end-of-line code (non-table entry tokens don't have a code).
[commit a9b45327d6]
This was accomplished by added a new UnUsed sub-code. At the locations where an error detected, the routine setting the error needs to set this sub-code if the token has not been added to the output list. This sub-code was set in two places, one in the get operand routine when there is a command or operator token, and the other in the translate routine that called the get expression routine when the terminating token is not the end-of-line token.
One other problem that caused a uninitialized variable used error from valgrind was also in the translate routine when the terminating token is checked for the end-of-line token. The check also needed to test if the token has a table entry before checking the token for the end-of-line code (non-table entry tokens don't have a code).
[commit a9b45327d6]
New Translator – Expression Error Checks
There were several error conditions that were not being caught correctly by the new translator routines. The get expression routine was also rewritten with a loop instead of recursively calling itself, which should be slightly more efficient and use slightly less stack space for complex expressions.
The first error condition occurred when a command or operator token is found when getting an operand. When this occurs, the proper "expected XX expression" error needed to be return, which is why the get operand routine contains the data type argument.
The second error condition occurred with the token after an operand in the end of expression token check. This check tested whether the token was an operator, however, both operators and command type tokens are reported as operators. The test was changed to specifically check for the operator token type.
The third error condition occurred when the hold stack is cleared of higher precedence operators. The check included obtaining the precedence of the current token and whether the token was a unary operator. However, these assumed the token had a valid code (was in the table), but the token could be any type. A different precedence function was used (taking a token instead of a code argument and handles any token type), and a new is unary operator function was implemented that takes a token argument that handles any token types. Precedences for all the token types also needed to be assigned (only some of them were assigned precedences previously).
Several more tests were added to expression test #1 for testing many of the error conditions. Note that the old translator expression mode has trouble with some of these error tests because the routine is not as robust as the new routines. The problems only occur in the expression mode. These problems will not be corrected, so for the time being expressions test #1 will fail when using the old translator (including the regression and memory test scripts).
[commit fab9d675a6] [commit fc7398879f]
The first error condition occurred when a command or operator token is found when getting an operand. When this occurs, the proper "expected XX expression" error needed to be return, which is why the get operand routine contains the data type argument.
The second error condition occurred with the token after an operand in the end of expression token check. This check tested whether the token was an operator, however, both operators and command type tokens are reported as operators. The test was changed to specifically check for the operator token type.
The third error condition occurred when the hold stack is cleared of higher precedence operators. The check included obtaining the precedence of the current token and whether the token was a unary operator. However, these assumed the token had a valid code (was in the table), but the token could be any type. A different precedence function was used (taking a token instead of a code argument and handles any token type), and a new is unary operator function was implemented that takes a token argument that handles any token types. Precedences for all the token types also needed to be assigned (only some of them were assigned precedences previously).
Several more tests were added to expression test #1 for testing many of the error conditions. Note that the old translator expression mode has trouble with some of these error tests because the routine is not as robust as the new routines. The problems only occur in the expression mode. These problems will not be corrected, so for the time being expressions test #1 will fail when using the old translator (including the regression and memory test scripts).
[commit fab9d675a6] [commit fc7398879f]
New Translator – Simple Expressions
To be able to use the new translator routines, two new temporary command lines options were added, namely ‑n and ‑ne, which do the same thing as ‑t and ‑te except that the new translator routine is called. These will be removed once the new translator has been fully implemented.
Several new expression tests were added to expression test #1 (simple expressions) including a test of the "‑E" expression (which previously reported an error), several more unary operator tests (including multiple unary operators), many with mixed data types (double and integer) and some with invalid mixed data type types (number and string).
When running the old translator, it was discovered that a negative constant at the beginning of the expression was interpreted as a negate and a positive constant instead of a negative constant. This occurred because the parser was not set to the operand state being that the translator was in its initial state, so a check was added to set the parser operand state if in the initial state and expression mode is selected.
Currently for simple expressions in the new translator routines, parser errors are not being handled correctly and therefore not reported correctly. Expression test #1 does not contain any expressions demonstrating this (some will be added once this issue is resolved). The old translator routines crash on these when using expression mode (command mode does not have a problem). The new translator is also not handling invalid operands and operators correctly.
[commit 58a03cb51b]
Several new expression tests were added to expression test #1 (simple expressions) including a test of the "‑E" expression (which previously reported an error), several more unary operator tests (including multiple unary operators), many with mixed data types (double and integer) and some with invalid mixed data type types (number and string).
When running the old translator, it was discovered that a negative constant at the beginning of the expression was interpreted as a negate and a positive constant instead of a negative constant. This occurred because the parser was not set to the operand state being that the translator was in its initial state, so a check was added to set the parser operand state if in the initial state and expression mode is selected.
Currently for simple expressions in the new translator routines, parser errors are not being handled correctly and therefore not reported correctly. Expression test #1 does not contain any expressions demonstrating this (some will be added once this issue is resolved). The old translator routines crash on these when using expression mode (command mode does not have a problem). The new translator is also not handling invalid operands and operators correctly.
[commit 58a03cb51b]
New Translator – Get Expression Design
Five new functions were implemented to support simple expressions (simple operands, unary and binary operators) in the new translator and two existing translator functions are utilized. The names of new translator routines that conflict with existing functions are being temporarily suffixed with a '2' character. This functions will replace the existing functions once the new translator is fully implemented and working.
The new routines consist of the top level translate routine (currently only supports expression mode); the get expression routine for getting an expression (to be utilized by any command needing an expression), which will return the token terminating the expression; the process operator routine for handling the precedence of operators; the get operand routine for getting an operand (simple operands to start); and the get token routine for getting a token from the parser. Click Continue... for some more details of each of these new routines.
The new routines consist of the top level translate routine (currently only supports expression mode); the get expression routine for getting an expression (to be utilized by any command needing an expression), which will return the token terminating the expression; the process operator routine for handling the precedence of operators; the get operand routine for getting an operand (simple operands to start); and the get token routine for getting a token from the parser. Click Continue... for some more details of each of these new routines.
New Translator – Expected Data Type
The old translator routines had a somewhat complex method to detect and report data type errors. The new translator will have a simpler method. The new translator will have a routine for getting an expression from the input line and will have an argument for the desired data type. At the end of the expression, if the data type does not match this, then the appropriate hidden conversion code (CvtDbl or CvtInt) will be added or an "expected XX expression" error will be reported.
For example, an IF command will call get expression for an integer expression, the INPUT PROMPT command will call for a string expression and the PRINT command will call for any type of expression. The same method will apply to the arguments of internal functions and operands of operators. Two new data types were implemented to support this, the Number and Any data types.
Since the initial new translator implementation will only handle unary and binary operators, the get expression routine needs to know the expected data type that will follow a unary operator. For the minus (negate) operator, this is a numeric expression (either double or integer). For the NOT operator, this is an integer expression.
For binary operators, the first operand has already been processed and the appropriate associated code of the operator will already have been found. The get expression routine will need to know the expected data type of the second operand for the selected associated code.
Therefore, a new expected data type member was added to the expression information structure stored in the table. Code was added to the table setup and check routine to automatically generate the values for this new member by looking at the table entry for of the code and its associated codes, specifically at the operand data types of the last operand. If multiple data types are found, then the expected data type is set to the Number or Any data type appropriately.
[commit aa7d7bec92]
For example, an IF command will call get expression for an integer expression, the INPUT PROMPT command will call for a string expression and the PRINT command will call for any type of expression. The same method will apply to the arguments of internal functions and operands of operators. Two new data types were implemented to support this, the Number and Any data types.
Since the initial new translator implementation will only handle unary and binary operators, the get expression routine needs to know the expected data type that will follow a unary operator. For the minus (negate) operator, this is a numeric expression (either double or integer). For the NOT operator, this is an integer expression.
For binary operators, the first operand has already been processed and the appropriate associated code of the operator will already have been found. The get expression routine will need to know the expected data type of the second operand for the selected associated code.
Therefore, a new expected data type member was added to the expression information structure stored in the table. Code was added to the table setup and check routine to automatically generate the values for this new member by looking at the table entry for of the code and its associated codes, specifically at the operand data types of the last operand. If multiple data types are found, then the expected data type is set to the Number or Any data type appropriately.
[commit aa7d7bec92]
Saturday, June 29, 2013
Minor Parser Correction
Implementation of the new translator will start with simple expressions consisting of simple operands (constants and identifiers without parentheses), unary operators, binary operators and possibly parenthetical expressions. While implementing the new routines (more details in following posts), a minor parser problem was discovered.
When in operand state (which allows a minus sign in front of a numerical constant for a negative constant), if the expression started with a "-E" string, it was interpreted as an incorrectly formed floating point constant that was missing digits before the start of the exponent. Any other character besides an "E" was correctly interpreted as an operator followed by an identifier without parentheses.
The problem occurred in the parser get number routine when there was an 'E' character that was not preceded by digits, which returned the "expected digits in mantissa of floating point constant" error. For this condition, a check was added if a sign was seen and no decimal point was seen, then the routine returns that no number was found. The parser then proceeds to the get operator routine.
[commit bae8b420b8]
When in operand state (which allows a minus sign in front of a numerical constant for a negative constant), if the expression started with a "-E" string, it was interpreted as an incorrectly formed floating point constant that was missing digits before the start of the exponent. Any other character besides an "E" was correctly interpreted as an operator followed by an identifier without parentheses.
The problem occurred in the parser get number routine when there was an 'E' character that was not preceded by digits, which returned the "expected digits in mantissa of floating point constant" error. For this condition, a check was added if a sign was seen and no decimal point was seen, then the routine returns that no number was found. The parser then proceeds to the get operator routine.
[commit bae8b420b8]
Friday, June 28, 2013
Minor Translator Improvements
Before embarking in the new translator routines, some minor improvements were made to the hold and done stacks. The same improvements could also have been made to the count and command stacks, but (spoiler alert) these stacks won't be needed for the new translator.
For the hold stack, there were several places where the top item on the stack is popped. The pop() function of QStack returns the item popped. But for these uses, the top item is not needed so a somewhat convoluted method of resizing the stack less one item was used for efficiency. It would be much better for code readability to use a simple function. A similar method was used for pushing items, where the stack was resized (up by one) first and then the members of the item filled in.
To make the code easy to work with and increase readability, a new HoldStack class was implemented, based on the QStack class, which contains a new drop() function, for popping the top item without returning it, and a push() function with specific arguments for the token and first token pointers (with the first token pointer defaulting to null since it is not always set for a push operation). Both functions still use the resize method, but these are now hidden in the class definition.
Similarly, a new DoneStack class was implemented except only a push() function was implemented since there were no uses of an empty pop. This push() function takes arguments for the RPN item pointer along with the first and last token pointers (both defaulting to null since these are not always set for push operations).
A minor change was also made to the table initialization routine that checks the maximum number of operands and associated codes. This routine now checks to make sure the fixed constants for these values are set exactly to the actual values found in the table instead of just checking if the constant at least as big as found in the table. The original thought was that this value would never decrease, which was not the case for the last change that decreased the maximum number of associated codes. (See post on May 20, 2010 for details of these constants.)
[commit cdfc70b43f] [commit 9951f9b9da] [commit 0f0fb97142]
For the hold stack, there were several places where the top item on the stack is popped. The pop() function of QStack returns the item popped. But for these uses, the top item is not needed so a somewhat convoluted method of resizing the stack less one item was used for efficiency. It would be much better for code readability to use a simple function. A similar method was used for pushing items, where the stack was resized (up by one) first and then the members of the item filled in.
To make the code easy to work with and increase readability, a new HoldStack class was implemented, based on the QStack class, which contains a new drop() function, for popping the top item without returning it, and a push() function with specific arguments for the token and first token pointers (with the first token pointer defaulting to null since it is not always set for a push operation). Both functions still use the resize method, but these are now hidden in the class definition.
Similarly, a new DoneStack class was implemented except only a push() function was implemented since there were no uses of an empty pop. This push() function takes arguments for the RPN item pointer along with the first and last token pointers (both defaulting to null since these are not always set for push operations).
A minor change was also made to the table initialization routine that checks the maximum number of operands and associated codes. This routine now checks to make sure the fixed constants for these values are set exactly to the actual values found in the table instead of just checking if the constant at least as big as found in the table. The original thought was that this value would never decrease, which was not the case for the last change that decreased the maximum number of associated codes. (See post on May 20, 2010 for details of these constants.)
[commit cdfc70b43f] [commit 9951f9b9da] [commit 0f0fb97142]
Thursday, June 27, 2013
Removed Temporary Strings
The current translator routines will be left in place and kept operational as the new translator routines are implemented. Before proceeding with the new routines, the temporary string data type was removed since it is no longer needed. For now, the sub-string data type remains, but this too will be removed.
String operands were being attached to operators and internal functions that contained string operands since the translator can't determine if operands were regular strings (from variables and arrays) or temporary strings (from functions). The encoder was going to make this decision. Since all strings will now be temporary, strings operands no longer need to be attached.
All the operator and internal functions with various codes for all the combinations of regular strings and temporary strings were removed from the table along with the various operand arrays and expression information structures with temporary strings and associated code arrays. The table also included the number of strings operands along with a flag whether the code contained string operands. These were only used for attaching string operands, and so were removed.
The expected test results files were updated since there are no longer any string operands attached to tokens. Another small change was made to the translator where a single parser instance is now created in the constructor and deleted in destructor instead of creating a local parser instance for every line translated.
[commit 35f201d96c] [commit 86ca56c673]
String operands were being attached to operators and internal functions that contained string operands since the translator can't determine if operands were regular strings (from variables and arrays) or temporary strings (from functions). The encoder was going to make this decision. Since all strings will now be temporary, strings operands no longer need to be attached.
All the operator and internal functions with various codes for all the combinations of regular strings and temporary strings were removed from the table along with the various operand arrays and expression information structures with temporary strings and associated code arrays. The table also included the number of strings operands along with a flag whether the code contained string operands. These were only used for attaching string operands, and so were removed.
The expected test results files were updated since there are no longer any string operands attached to tokens. Another small change was made to the translator where a single parser instance is now created in the constructor and deleted in destructor instead of creating a local parser instance for every line translated.
[commit 35f201d96c] [commit 86ca56c673]
Tuesday, June 25, 2013
New Translator – Strings
How strings will be handled at run-time was the first factor considered in the Translator redesign. Previously there was the concept of temporary strings and sub-strings. Temporary strings were put on the evaluation stack as the result of the concatenate string operator or a string function. Sub-strings were put on the evaluation stack as the result of the sub-string functions (LEFT$, MID$ and RIGHT$). Regular strings were put on the evaluation stack for string variables.
The idea behind temporary strings was the prevent the copying of a string for a variable to the evaluation stack - no point in creating a temporary string for a string variable if it is never going to be modified. Take the example expression A$+B$, which would be translated to A$ B$ +$. A temporary string only needs to be created for the result. Using temporary strings for when A$ and B$ are pushed to the evaluation stack would entail an unnecessary copy.
The idea behind sub-strings was again to prevent unnecessary copying of strings (since a sub-string is part of an existing string). Sub-strings could refer to either regular or temporary strings. Sub-string assignments would also be supported (for example, the statement MID$(A$,4,2)="AB").
The original String class developed supported these concepts. However, the QString class is now being used for strings and this Qt class supports implicit sharing, meaning that when a string is copied, it is not actually copied (only a reference to the original string is copied) until the string is actually modified. So for the sample expression, since the A$ and B$ string values are not modified, no copy would take place.
Therefore, only temporary strings will be put on the evaluation stack. There is no need for all the different forms of the string functions and operators. Sub-strings will need to be handled differently then originally envisioned with the String class. Sub-strings in expressions will now be handled the same as other string functions using the QString member functions left(), mid() and right(). Sub-string assignments will be handled differently using the QString member function replace(). Further details of sub-string assignments will be given when the new LET statement translation is implemented.
The idea behind temporary strings was the prevent the copying of a string for a variable to the evaluation stack - no point in creating a temporary string for a string variable if it is never going to be modified. Take the example expression A$+B$, which would be translated to A$ B$ +$. A temporary string only needs to be created for the result. Using temporary strings for when A$ and B$ are pushed to the evaluation stack would entail an unnecessary copy.
The idea behind sub-strings was again to prevent unnecessary copying of strings (since a sub-string is part of an existing string). Sub-strings could refer to either regular or temporary strings. Sub-string assignments would also be supported (for example, the statement MID$(A$,4,2)="AB").
The original String class developed supported these concepts. However, the QString class is now being used for strings and this Qt class supports implicit sharing, meaning that when a string is copied, it is not actually copied (only a reference to the original string is copied) until the string is actually modified. So for the sample expression, since the A$ and B$ string values are not modified, no copy would take place.
Therefore, only temporary strings will be put on the evaluation stack. There is no need for all the different forms of the string functions and operators. Sub-strings will need to be handled differently then originally envisioned with the String class. Sub-strings in expressions will now be handled the same as other string functions using the QString member functions left(), mid() and right(). Sub-string assignments will be handled differently using the QString member function replace(). Further details of sub-string assignments will be given when the new LET statement translation is implemented.
Subscribe to:
Posts (Atom)