Print Functions Correction

A segmentation fault was occurring with the new translator routines when a print function (TAB or SPC) was used incorrectly in an expression.  The problem was in the existing process final operand routine, which contained a check for print-only internal functions.  For these functions, it set flags for the current command on the command stack.  However, the command stack is not used for the new translator routines and was empty causing the crash.  This section of code will be removed once the old translator routines are removed, but to temporarily correct the problem with the new translator, a check was added to make sure the command stack is not empty.

The second problem occurred in the existing find code routine.  The change above allowed the print functions with a none data type to now be placed on the done stack.  When the the data type of the done stack top item was checked with the expected data type, the conversion code table did not have any entries for the none data type.  The missing entries were added set to the invalid code.

Two additional statements with print functions used incorrectly in expressions were added to translator test #3.  The old translator routines report the error incorrectly, the entire function through the closing parentheses should be reported, not just the function token.  The new translator routines do report the error correctly.

[commit 250b6ffd15]

Multiple String Assignments – With Sub-Strings

Multiple string assignments will be handled by the AssignListStr code.  At run-time, this code (as the other assign list codes) will pop the value to be assigned from the stack and then begin popping variable references from the stack, assign the value, and continue until the stack is empty.  This will not work if any of the references to assign is a sub-string.  With the old design, mix-string assignments were handled with the AssignListMixStr code.  This was detailed in the post on May 22, 2010, but no details were given how this would be handled at run-time.

For the new design, if a multiple string assignment contains at least one sub-string, then there will be a specific assign code for each assignment instead of a single assign list code.  The specific assign codes will keep the value being assigned on the stack for the next assign code.  Only the last code will be a regular assign code.  Consider this mixed string assignment and its translation (note color coding showing the tokens that the codes process):

A$, LEFT$(B$,5), RIGHT$(C$,2) = D$
A$ B$ 5 C$ 2 D$ AssignKeepRight AssignKeepLeft AssignStr

The assign keep codes will pop the value to be assigned from the stack, pop the reference to assign, assign the value to the reference and push the value back to the stack for the next assign code.  The final regular assign code will not push the value to be assigned back to the stack leaving the stack empty.

There will be five assign keep codes: AssignKeepStr, AssignKeepLeft, AssignKeepMid2, AssignKeepMid3 and AssignKeepRight.  In the table, these codes will be the second associated code for the AssignStr, Left, Mid2, Mid3 and Right code entries.

Sub-Strings – New Design

Previously with the original String class, as an optimization, sub-string functions were handled differently than the other string functions.  Instead of returning a temporary string, they simply adjusted what part of the string they referred to, and would therefore work for either temporary strings (results of other string functions or operators) or reference strings (from variables).  Sub-strings assignments would be handled with an assign sub-string code that would work with the result of a sub-string of a variable reference.  This was detailed by the series of posts in May, 2010.

With the change to the QString class, this optimization will not work, and is not necessary.  The sub-string functions (LEFT$, MID$, and RIGHT$) will work like the other string functions and operators where they will return a temporary string.  However, sub-string assignments will need to be handled differently.  There will need to be specific new codes for handling sub-string assignments, to be named AssignLeft, AssignMid2, AssignMid3, and AssignRight.

Consider the following sub-string assign along with the old translation and the proposed new translation:

LEFT$(A$,5)=B$
Old: A$<ref> 5 LEFT$(<ref> B$ AssignSub$
New: A$<ref> 5 B$ AssignLeft

With the old translation, the sub-string reference would be on the stack (along with the value to assign) for the generic AssignSub$ to process.  The new translation is simpler where the new AssignLeft code will expect a regular variable reference, the length argument of the LEFT$ function, and the value to assign.  Internally all the sub-string assignment codes will use the QString::replace() function.

New Translator – Testing and a Correction

Now that all four expression tests along with the first three translator tests are working with the new translator routines, it is becoming somewhat time consuming to test each individually including checking for memory errors.

Therefore, temporarily a new memory test script (memtestn) was added that is basically identical to the current memory test script (memtest) except that the new translator is used and only the first three translator tests are run.  As more of the new translator is implemented and more tests are working, the script will be updated.  This change was put into its own commit, so that it can be reverted once the new translator implementation is complete and the old translator routines are removed.

While using the new memory test script, a problem was discovered with translator test #3 on one of the error tests, which was causing a segmentation fault, but only when compiled for Release.  When compiled for Debug (as used for development), the segmentation fault did not occur.  Finding the problem was difficult because the segmentation fault did not occur when compiled for Debug.

The debugging method used was the insertion of qDebug() calls until the location of the crash was found.  The problem occurred in the new outputLastToken() access function added to the Translator class so that the command translate routines can access the last token added to the RPN output list.  The problem was that this function did not actually have a return keyword.  When compiled for debug, the correct pointer gets returned, but when compiled for release, this is optimized out and a null gets returned.

[commit 06bd286162] [commit 8c872f68c6]

LET Command – Single/Multiple Assignments

The design of each command will be reconsidered with the new translator design.  Several designs for the LET command were considered, but in the end, the current design seems to the most efficient at run time with the minor exception of multiple sub-string assignments.  Excluding sub-string assignments, LET statements are translated as follows:

A = 5.0              A<ref> 5.0 Assign
A,B,C = 5.0          A<ref> B<ref> C<ref> 5.0 AssignList

For multiple assignments, all variables being assigned must be the same data type.  The data type of the value being assigned must match the variables being assigned, however, for numeric types, an appropriate hidden conversion code will be added as needed.  If the optional LET keyword was specified, the hidden LET sub-code is set in the final assignment token.

Click Continue... For details of the implementation of the LET translation.  See the commit log for other minor changes made.  Translator tests #1 through #3 (various assignment tests) now pass with the new translator routines.

[commit f965e0f649]

New Translator – LET Translation (Begin)

Before beginning the implementation of the LET command translation routine, so thought was given on how the project should be organized.  The old token centric translator design had the translation of commands embedded throughout the translator, specifically in the token handling functions.  With the new translator design being command centric, the various routines (translate, recreate and execute) for each command can be organized into their own files.

A decision was also made not to clutter up the main project directory with all the various command source files, so these files will be put into a sub-directory.  The name "basic" was chosen for this sub-directory since all the sources files will be related to the BASIC language.  This sub-directory will also contain the execute routines for all of the operators and internal functions of the BASIC language.  The various command function prototypes (and any other command related definitions needed) will be put into the commands.h header file in this sub-directory.

The let.cpp source file was created in the basic sub-directory to hold LET command routines.  An initial translate routine was implemented for the LET command.  For now, this routine just checks if the LET keyword was specified and returns two different BUG Debug statuses to distinguish between the two forms.  Finally, a function prototype for this routine was added to the commands.h header file.

Some changes were also needed to the CMake build configuration file starting with added the let.cpp source file to the list of source files.  So that the various files in the basic sub-directory can access the various header files, the main project source directory was added to the list of include directories.  It turns out that it is not necessary to make the executable dependent on the list of project header files as CMake automatically figures out all the dependent header files for each source file, so this list was removed.

The initial LET function was tested with translator test #5 (LET command tests) to make sure the temporary BUG Debug errors were reported correctly.

[commit 3330b28c9e]

New Translator – Command Translation

To start the translation of a command, a new get command routine was implemented and starts by getting a token taking into account an assignment statement that does not have the LET keyword.  If the first token has an error, the "expected command" error is returned since it does not matter what type of parser error was detected.

If the token obtained is a command token, the pointer to the translate function for the command is obtained from the table.  Otherwise an assignment statement is assumed and the pointer to the translate function for the LET command is obtained.  The token will be passed to the LET translate function.

The interface of the translate functions contains a reference to the translator (so the command can access the various translator routine like the get token and get expression routines), a pointer to the command token (so the command can add it to the output list), and a reference to a token pointer to be used to return the token the terminated the command or where an error was detected (and will be used to pass the first token to the LET translator function for an implied assignment statement).  The token status is returned.

If the translate function pointer is not set, the token is marked unused and a "not yet implemented" error is returned.  For now, no translate function pointers have been set in the table (none have been implemented).  The translate functions will replace the command handlers.  The token handlers are not needed with the new translator.

The new translator routine was modified to call either the get expression or the new get command routine depending on the expression mode argument.  The expression mode argument does not need to be saved with the new translator routines.  A temporary '-nt' test option was added to access the new translator routines for statements, and similarly the '-n' test option was expanded to support translator test files.  Obviously none of the translator tests succeed with the new translator routines.

[commit de01f48ebb]

New Translator – Parentheses Expressions (Tagged)

The implementation of expressions for all tokens with parentheses including open and closing parentheses is now complete.  The new translator routines now fully support expressions and version v0.4.1 has been tagged.  Note that expression test #1 currently still fails with the regression and memory test scripts because of problems with the expression mode in the old translator routines.  The new translator routines run all expression tests successfully.  Implementation of command translation can now commence in the new translator.

[commit ac6658f16f]

New Translator – Arrays/User Functions

There are four token types used for arrays and functions, which include identifiers with and without parentheses (could be either a variable, an array or a user function to be determined by the encoder), and a defined function with and without parentheses.  A define function without parentheses simply gets added to the output list and push to the done stack like constants, identifiers with no parentheses,  and internals functions with no parentheses.  To support the other two types (identifiers and define functions with parentheses) the get operand routine was updated to call the newly implemented get parentheses token routine.

The new get parentheses token routine starts by pushing the parentheses token to the hold stack, which being of low precedence, will create a border as the expressions of the arguments are processed.  A number of operands counter is initialized and a loop is entered for each operand by first calling the get expression routine.  If the parentheses token is an identifier with parentheses, it could be a user function.  Arguments of user functions are passed by reference, so any operand that could be an variable or array element has its reference flag set.

The terminating token is then checked.  For a comma, the token is deleted and the operand is counted.  For a closing parentheses, the existing process final operand function is called, which upon success attaches all the operands, appends the token with parentheses to the RPN output list, and pushes the token to the done stack.  The token with parentheses is then dropped from the hold stack.  For other terminating tokens, the appropriate error is returned.

One expression in test #3 had a different result from the old translator routines due to two issues.  First, an array in the expression incorrectly had its reference flag set by the old routines (the new routines correctly did not).  Second, the argument of a define function along with the define function incorrectly had their reference flags set.  For a defined function, the old routines assumed the define functions would be passed by reference (and set the reference flags of arguments).  This is no longer the case (see last post).   These minor issues were corrected in the old routines and results for expression test #3 were updated.

[commit 847c5d078e]

New Translator – User Functions

User functions can take two forms, a full user function (using the FUNCTION syntax) or a simple define function (using the DEF FN syntax).  A full user function will pass arguments by reference.  This only applies to variables and array elements.  Results of expressions will be passed by value.  To keep things consistent internally, a temporary value will be allocated and a pointer to it will be passed, so all arguments will be a reference.  To force a single variable or array element to be passed by value, it can be surrounded by a set of parentheses.

Define functions will have two forms, a single line form that looks like an assignment and a multiple line form (that will end with an END DEF statement have one or more assignments for the return value).  Unlike full user functions, arguments to define functions will only be passed by value.  This make sense for the single line form since the arguments can't be assigned inside the function, and to keep the internal code consistent between the two forms, the multiple line form will also have arguments passed by value.  Arguments will be local variables, so any assignments will not affect the original variables.

This is the same method of argument passing used by QBASIC, and seems to be a reasonable design choice, so will also be used for this project.  Subroutines (the SUB syntax) will use the same pass by reference scheme as full user functions.  The passing of entire arrays will be dealt with later.  QBASIC allows this by listing the array name followed by an opening and closing parentheses with no subscripts.  This same syntax may be used for this project.

To handle function calls in the translator, the reference flag of an operand is set if it is an identifier with or without parentheses.  These tokens could end up being function calls, but this will be handled by the encoder.  Identifiers with parentheses could also end up being an array, where its arguments are integer subscripts.  Again this will be handled by the encoder, which will add any needed internal convert to integer codes (for double subscripts) or report errors for string subscripts.  The reference flag in any subscripts will be ignored.  Regardless of whether the identifier is an array or function, the translator will attach a pointer to each argument for the encoder.

New Translator – Internal Functions

To support internal functions, both with arguments (parentheses) and without (no parentheses), the get operand function was updated.  Internal functions without arguments are treated the same as constants and identifiers with no parentheses where the tokens are simply added to the RPN output list and pushed to the done stack.

For internal functions with parentheses, a new get internal function routine was implemented and called.  This function starts by pushing the internal function token to the hold stack, which being of low precedence, will create a border as the expressions of the arguments are processed.  After getting the number arguments, it loops for each argument by first calling the get expression routine and then checking the terminating token for a comma or closing parentheses.

For a comma, if at the last argument, then an error is returned if the internal function code does not have multiple entries (for examine, the MID$ function with two or three arguments).  Otherwise the code is changed to the code with an additional argument.  The comma token is deleted and the existing find code routine is called to process and check the argument.

For a closing parentheses, if not at the last argument, then an error is returned.  Otherwise the existing process final operand function is called to process the final argument, which appends the internal function token to the RPN output list upon success.  The internal function token is then dropped from the hold stack.

If the terminating token is neither a comma nor a closing parentheses, then the appropriate error is returned depending which argument it is at taking into account whether the internal function has multiple entries.

Expression test #3 now passes with the new translator routines except for the three lines that contain identifiers with parentheses (an array or a user function), which has not yet been implemented in the new translator.  Expression test #4 also now passes successfully.  Since none of the expression tests contained a function with no arguments, a new expression was added to test #4 containing the RND function.

[commit 81d3137531]

New Translator – Checking Token Codes

A minor coding issue was discovered in code that needs to check the code in a token.  This is more an issue with the new translator routines because they will be checking tokens that could be of any token type.  The old translator routines already knew the token type before checking the code.

Since it is desirable to make the code as easy to implement as possible, the Token class isCode() function was modified to make sure the token is the correct type before checking the code.  The correct type is any type that has a table entry.  If the calling code already knows the token is a type that has a valid code, then it can use the Token class code() access function and compare to the code directly.

[commit 9bbd1e319f]

New Translator – Parentheses

To support simple parenthetical expressions, the get expression routine was modified to look for an open parentheses token just before checking for a unary operator.  The open parentheses token is pushed to the hold stack so that when the expression is processed by recursively calling get expression, only operators from inside the expression will be popped from the hold stack (open parentheses has a very low precedence).  After processing the closing parentheses, get expression continues on by getting and processing a binary operator of end-of-expression token.

The recursive get expression will terminate upon a closing parentheses token.  In the table, the end expression flag had to be added to the closing parentheses code to cause this termination.  Upon return, the terminating token is checked for a closing parentheses, otherwise an "expected operator or closing parentheses" error is returned.  The open parentheses token is popped from the hold stack and the open and closing parentheses tokens are assigned to the first and last tokens of the item on top of the done stack (used for error reporting; details of this design start with posts on January 16, 2011).

The closing parentheses is saved as a pending parentheses token for later checking.  This token is marked as being used as a last operand and as the pending parentheses (to prevent it from being prematurely deleted).  The precedence of the last token added to the output list (which is on top of the done stack) is saved for when the pending parentheses token is later checked.  If this token is an operator, the precedence is obtained from the table, otherwise it is set to the highest precedence (parentheses around not operators are never necessary).

A new check pending parentheses routine was added starting with the existing do pending parentheses routine (renamed more appropriately instead of adding a '2', but will still eventually replace the old routine).  These routines check if parentheses entered are not necessary, in which case, a flag is set in the translated so that these will be reproduced.  Necessary parentheses are implied by the translated RPN format.  Parentheses necessity is determined by the precedence of the operators (see posts starting on March 21, 2010 for details).

The new check pending parentheses routine is called from two locations in the new process operator routine, one when an operator is popped from the hold stack before its final operand is processed and one for a new operator before it is checked for the end-of-expression or its first operand is processed.  The check pending parentheses routine determines if parentheses are not necessary by checking if the precedence of last token added is greater than the current operator or of equal precedence if th operator was just popped from the hold stack.  There is a new popped argument to determine from which (the old routine used the current translator state).

Expression test #2 (parentheses tests) now passes with the new translator routines, but one change was needed for the unexpected closing parentheses test that reported the "expected operator or end-of-expression" error (internally named NoOpenParen).  When the wording of this error was changed, it should been been removed and the "expected operator or end-of-statement" error used instead.  This error needs to be replaced by the caller as appropriate for the command in which it is detected (see post from Tuesday).

[commit d3d57f2235]

New Translator – Unary Operator Issue

An issue was discovered where if the token after an operand was a unary operator (this token should be a binary operator or a token terminating the expression), the process operator routine incorrectly processed the unary operator assuming it was a binary operator causing the code to malfunction.

A check was added after getting this token where if it is a unary operator, the "expected binary operator or end-of-statement" error is returned.  This error, like the "expected operator or end-of-statement" error, will also need to be changed to an appropriate error by the caller.  A test for this was added to expression test #1.

[commit 261c9647df]

New Translator – Parser Errors

There are currently two types of parser errors, an unrecognizable character and an incorrect number constant (of which there are five different ones).  The number constant errors may have an alternate column (for example, when there is an error in exponent of a floating point number, the alternate column points to the error and the column points to the beginning of the number) or a length more than one (for example, two consecutive decimal points at the beginning).

When the translator is expecting an operand, the number constant errors should be reported as is (for example, pointing to the bad exponent or both decimal points).  However, when the translator is expecting some other token (like an operator), a different error appropriate for the situation (for example, an "expecting an operator or end-of-statement" error) should be reported and it should be pointing to the only the first character of the bad token.

The get token routine was modified to take a desired data type argument instead of an operand flag.  A none data type indicates that the token desired is not an operand.  If the token obtained has an error, then it is marked as unused (it needs to be deleted).  If not getting an operand token and the data type of the token is double (indicating a number constant error), then the token length is set to one (the error will point to only the first character of the token).  For the error to return, if getting an operand token and the data type of the token is not double (indicating a parser error), the error is set to the appropriate "expecting XX expression" error for the desired data type argument, otherwise a generic parser error is returned (so the caller will used the number constant error message in the token).

The loop in the get expression routine was modified to eliminate duplicate code before the loop and at the end of the loop.  At the end of the loop, the token pointer is set to null to force getting a new token at the beginning of the loop (I wish there were a way to prevent having this extra check, but there isn't without resorting to a goto statement and a label).  Each of the return statements were changed to break statements with the final return at the end outside of the loop.

Also in this routine, the statement for getting the operand and next token was broken into two so that the error can be changed to an "expecting operator or end-of-statement" error when get token returns an error (any parser error).  This error will need to be changed to the appropriate error by the caller.  For example, in an IF statement when it calls to get its expression, if this error occurs, the error needs to be changed to an "expected operator or THEN" error.

In the new translate routine, when setting the error message in the RPN output list, the error string is obtained from the token for a parser error instead of from the error status.  This also needed to be done before an unused token is deleted (tokens with parser errors are marked as unused).

Finally, the set operand state function of the parser was removed.  This was always and only called before calling the token routine, so an operand state argument was added to this function.  Several parser error tests were added to expression test #1.  It was also discovered that the autoenums.h include file was not always being regenerated when the Token class source file was modified, which turned out to be a wrong dependency listed in the CMake build file.

[commit a194cdbfb6] [commit 348ec2230b]

New Translator – Simple Expressions (Tagged)

The implementation for simple expressions is now complete in the new translator and version v0.4.0 has been tagged.  Note that expression test #1 currently fails with the regression and memory test scripts because of problems with the expression mode in the old translator routines.  The new translator routines run this test successfully, but should not yet be used on any of the other expression tests.  Also, parser errors are not yet handled correctly by the new translator routines, but this is going to require a major change to how token errors are handled.

[commit 552aa6176a]