LET Command – Single/Multiple Assignments

The design of each command will be reconsidered with the new translator design.  Several designs for the LET command were considered, but in the end, the current design seems to the most efficient at run time with the minor exception of multiple sub-string assignments.  Excluding sub-string assignments, LET statements are translated as follows:

A = 5.0              A<ref> 5.0 Assign
A,B,C = 5.0          A<ref> B<ref> C<ref> 5.0 AssignList

For multiple assignments, all variables being assigned must be the same data type.  The data type of the value being assigned must match the variables being assigned, however, for numeric types, an appropriate hidden conversion code will be added as needed.  If the optional LET keyword was specified, the hidden LET sub-code is set in the final assignment token.

Click Continue... For details of the implementation of the LET translation.  See the commit log for other minor changes made.  Translator tests #1 through #3 (various assignment tests) now pass with the new translator routines.

[commit f965e0f649]

New Translator – LET Translation (Begin)

Before beginning the implementation of the LET command translation routine, so thought was given on how the project should be organized.  The old token centric translator design had the translation of commands embedded throughout the translator, specifically in the token handling functions.  With the new translator design being command centric, the various routines (translate, recreate and execute) for each command can be organized into their own files.

A decision was also made not to clutter up the main project directory with all the various command source files, so these files will be put into a sub-directory.  The name "basic" was chosen for this sub-directory since all the sources files will be related to the BASIC language.  This sub-directory will also contain the execute routines for all of the operators and internal functions of the BASIC language.  The various command function prototypes (and any other command related definitions needed) will be put into the commands.h header file in this sub-directory.

The let.cpp source file was created in the basic sub-directory to hold LET command routines.  An initial translate routine was implemented for the LET command.  For now, this routine just checks if the LET keyword was specified and returns two different BUG Debug statuses to distinguish between the two forms.  Finally, a function prototype for this routine was added to the commands.h header file.

Some changes were also needed to the CMake build configuration file starting with added the let.cpp source file to the list of source files.  So that the various files in the basic sub-directory can access the various header files, the main project source directory was added to the list of include directories.  It turns out that it is not necessary to make the executable dependent on the list of project header files as CMake automatically figures out all the dependent header files for each source file, so this list was removed.

The initial LET function was tested with translator test #5 (LET command tests) to make sure the temporary BUG Debug errors were reported correctly.

[commit 3330b28c9e]

New Translator – Command Translation

To start the translation of a command, a new get command routine was implemented and starts by getting a token taking into account an assignment statement that does not have the LET keyword.  If the first token has an error, the "expected command" error is returned since it does not matter what type of parser error was detected.

If the token obtained is a command token, the pointer to the translate function for the command is obtained from the table.  Otherwise an assignment statement is assumed and the pointer to the translate function for the LET command is obtained.  The token will be passed to the LET translate function.

The interface of the translate functions contains a reference to the translator (so the command can access the various translator routine like the get token and get expression routines), a pointer to the command token (so the command can add it to the output list), and a reference to a token pointer to be used to return the token the terminated the command or where an error was detected (and will be used to pass the first token to the LET translator function for an implied assignment statement).  The token status is returned.

If the translate function pointer is not set, the token is marked unused and a "not yet implemented" error is returned.  For now, no translate function pointers have been set in the table (none have been implemented).  The translate functions will replace the command handlers.  The token handlers are not needed with the new translator.

The new translator routine was modified to call either the get expression or the new get command routine depending on the expression mode argument.  The expression mode argument does not need to be saved with the new translator routines.  A temporary '-nt' test option was added to access the new translator routines for statements, and similarly the '-n' test option was expanded to support translator test files.  Obviously none of the translator tests succeed with the new translator routines.

[commit de01f48ebb]

New Translator – Parentheses Expressions (Tagged)

The implementation of expressions for all tokens with parentheses including open and closing parentheses is now complete.  The new translator routines now fully support expressions and version v0.4.1 has been tagged.  Note that expression test #1 currently still fails with the regression and memory test scripts because of problems with the expression mode in the old translator routines.  The new translator routines run all expression tests successfully.  Implementation of command translation can now commence in the new translator.

[commit ac6658f16f]

New Translator – Arrays/User Functions

There are four token types used for arrays and functions, which include identifiers with and without parentheses (could be either a variable, an array or a user function to be determined by the encoder), and a defined function with and without parentheses.  A define function without parentheses simply gets added to the output list and push to the done stack like constants, identifiers with no parentheses,  and internals functions with no parentheses.  To support the other two types (identifiers and define functions with parentheses) the get operand routine was updated to call the newly implemented get parentheses token routine.

The new get parentheses token routine starts by pushing the parentheses token to the hold stack, which being of low precedence, will create a border as the expressions of the arguments are processed.  A number of operands counter is initialized and a loop is entered for each operand by first calling the get expression routine.  If the parentheses token is an identifier with parentheses, it could be a user function.  Arguments of user functions are passed by reference, so any operand that could be an variable or array element has its reference flag set.

The terminating token is then checked.  For a comma, the token is deleted and the operand is counted.  For a closing parentheses, the existing process final operand function is called, which upon success attaches all the operands, appends the token with parentheses to the RPN output list, and pushes the token to the done stack.  The token with parentheses is then dropped from the hold stack.  For other terminating tokens, the appropriate error is returned.

One expression in test #3 had a different result from the old translator routines due to two issues.  First, an array in the expression incorrectly had its reference flag set by the old routines (the new routines correctly did not).  Second, the argument of a define function along with the define function incorrectly had their reference flags set.  For a defined function, the old routines assumed the define functions would be passed by reference (and set the reference flags of arguments).  This is no longer the case (see last post).   These minor issues were corrected in the old routines and results for expression test #3 were updated.

[commit 847c5d078e]

New Translator – User Functions

User functions can take two forms, a full user function (using the FUNCTION syntax) or a simple define function (using the DEF FN syntax).  A full user function will pass arguments by reference.  This only applies to variables and array elements.  Results of expressions will be passed by value.  To keep things consistent internally, a temporary value will be allocated and a pointer to it will be passed, so all arguments will be a reference.  To force a single variable or array element to be passed by value, it can be surrounded by a set of parentheses.

Define functions will have two forms, a single line form that looks like an assignment and a multiple line form (that will end with an END DEF statement have one or more assignments for the return value).  Unlike full user functions, arguments to define functions will only be passed by value.  This make sense for the single line form since the arguments can't be assigned inside the function, and to keep the internal code consistent between the two forms, the multiple line form will also have arguments passed by value.  Arguments will be local variables, so any assignments will not affect the original variables.

This is the same method of argument passing used by QBASIC, and seems to be a reasonable design choice, so will also be used for this project.  Subroutines (the SUB syntax) will use the same pass by reference scheme as full user functions.  The passing of entire arrays will be dealt with later.  QBASIC allows this by listing the array name followed by an opening and closing parentheses with no subscripts.  This same syntax may be used for this project.

To handle function calls in the translator, the reference flag of an operand is set if it is an identifier with or without parentheses.  These tokens could end up being function calls, but this will be handled by the encoder.  Identifiers with parentheses could also end up being an array, where its arguments are integer subscripts.  Again this will be handled by the encoder, which will add any needed internal convert to integer codes (for double subscripts) or report errors for string subscripts.  The reference flag in any subscripts will be ignored.  Regardless of whether the identifier is an array or function, the translator will attach a pointer to each argument for the encoder.

New Translator – Internal Functions

To support internal functions, both with arguments (parentheses) and without (no parentheses), the get operand function was updated.  Internal functions without arguments are treated the same as constants and identifiers with no parentheses where the tokens are simply added to the RPN output list and pushed to the done stack.

For internal functions with parentheses, a new get internal function routine was implemented and called.  This function starts by pushing the internal function token to the hold stack, which being of low precedence, will create a border as the expressions of the arguments are processed.  After getting the number arguments, it loops for each argument by first calling the get expression routine and then checking the terminating token for a comma or closing parentheses.

For a comma, if at the last argument, then an error is returned if the internal function code does not have multiple entries (for examine, the MID$ function with two or three arguments).  Otherwise the code is changed to the code with an additional argument.  The comma token is deleted and the existing find code routine is called to process and check the argument.

For a closing parentheses, if not at the last argument, then an error is returned.  Otherwise the existing process final operand function is called to process the final argument, which appends the internal function token to the RPN output list upon success.  The internal function token is then dropped from the hold stack.

If the terminating token is neither a comma nor a closing parentheses, then the appropriate error is returned depending which argument it is at taking into account whether the internal function has multiple entries.

Expression test #3 now passes with the new translator routines except for the three lines that contain identifiers with parentheses (an array or a user function), which has not yet been implemented in the new translator.  Expression test #4 also now passes successfully.  Since none of the expression tests contained a function with no arguments, a new expression was added to test #4 containing the RND function.

[commit 81d3137531]

New Translator – Checking Token Codes

A minor coding issue was discovered in code that needs to check the code in a token.  This is more an issue with the new translator routines because they will be checking tokens that could be of any token type.  The old translator routines already knew the token type before checking the code.

Since it is desirable to make the code as easy to implement as possible, the Token class isCode() function was modified to make sure the token is the correct type before checking the code.  The correct type is any type that has a table entry.  If the calling code already knows the token is a type that has a valid code, then it can use the Token class code() access function and compare to the code directly.

[commit 9bbd1e319f]

New Translator – Parentheses

To support simple parenthetical expressions, the get expression routine was modified to look for an open parentheses token just before checking for a unary operator.  The open parentheses token is pushed to the hold stack so that when the expression is processed by recursively calling get expression, only operators from inside the expression will be popped from the hold stack (open parentheses has a very low precedence).  After processing the closing parentheses, get expression continues on by getting and processing a binary operator of end-of-expression token.

The recursive get expression will terminate upon a closing parentheses token.  In the table, the end expression flag had to be added to the closing parentheses code to cause this termination.  Upon return, the terminating token is checked for a closing parentheses, otherwise an "expected operator or closing parentheses" error is returned.  The open parentheses token is popped from the hold stack and the open and closing parentheses tokens are assigned to the first and last tokens of the item on top of the done stack (used for error reporting; details of this design start with posts on January 16, 2011).

The closing parentheses is saved as a pending parentheses token for later checking.  This token is marked as being used as a last operand and as the pending parentheses (to prevent it from being prematurely deleted).  The precedence of the last token added to the output list (which is on top of the done stack) is saved for when the pending parentheses token is later checked.  If this token is an operator, the precedence is obtained from the table, otherwise it is set to the highest precedence (parentheses around not operators are never necessary).

A new check pending parentheses routine was added starting with the existing do pending parentheses routine (renamed more appropriately instead of adding a '2', but will still eventually replace the old routine).  These routines check if parentheses entered are not necessary, in which case, a flag is set in the translated so that these will be reproduced.  Necessary parentheses are implied by the translated RPN format.  Parentheses necessity is determined by the precedence of the operators (see posts starting on March 21, 2010 for details).

The new check pending parentheses routine is called from two locations in the new process operator routine, one when an operator is popped from the hold stack before its final operand is processed and one for a new operator before it is checked for the end-of-expression or its first operand is processed.  The check pending parentheses routine determines if parentheses are not necessary by checking if the precedence of last token added is greater than the current operator or of equal precedence if th operator was just popped from the hold stack.  There is a new popped argument to determine from which (the old routine used the current translator state).

Expression test #2 (parentheses tests) now passes with the new translator routines, but one change was needed for the unexpected closing parentheses test that reported the "expected operator or end-of-expression" error (internally named NoOpenParen).  When the wording of this error was changed, it should been been removed and the "expected operator or end-of-statement" error used instead.  This error needs to be replaced by the caller as appropriate for the command in which it is detected (see post from Tuesday).

[commit d3d57f2235]

New Translator – Unary Operator Issue

An issue was discovered where if the token after an operand was a unary operator (this token should be a binary operator or a token terminating the expression), the process operator routine incorrectly processed the unary operator assuming it was a binary operator causing the code to malfunction.

A check was added after getting this token where if it is a unary operator, the "expected binary operator or end-of-statement" error is returned.  This error, like the "expected operator or end-of-statement" error, will also need to be changed to an appropriate error by the caller.  A test for this was added to expression test #1.

[commit 261c9647df]

New Translator – Parser Errors

There are currently two types of parser errors, an unrecognizable character and an incorrect number constant (of which there are five different ones).  The number constant errors may have an alternate column (for example, when there is an error in exponent of a floating point number, the alternate column points to the error and the column points to the beginning of the number) or a length more than one (for example, two consecutive decimal points at the beginning).

When the translator is expecting an operand, the number constant errors should be reported as is (for example, pointing to the bad exponent or both decimal points).  However, when the translator is expecting some other token (like an operator), a different error appropriate for the situation (for example, an "expecting an operator or end-of-statement" error) should be reported and it should be pointing to the only the first character of the bad token.

The get token routine was modified to take a desired data type argument instead of an operand flag.  A none data type indicates that the token desired is not an operand.  If the token obtained has an error, then it is marked as unused (it needs to be deleted).  If not getting an operand token and the data type of the token is double (indicating a number constant error), then the token length is set to one (the error will point to only the first character of the token).  For the error to return, if getting an operand token and the data type of the token is not double (indicating a parser error), the error is set to the appropriate "expecting XX expression" error for the desired data type argument, otherwise a generic parser error is returned (so the caller will used the number constant error message in the token).

The loop in the get expression routine was modified to eliminate duplicate code before the loop and at the end of the loop.  At the end of the loop, the token pointer is set to null to force getting a new token at the beginning of the loop (I wish there were a way to prevent having this extra check, but there isn't without resorting to a goto statement and a label).  Each of the return statements were changed to break statements with the final return at the end outside of the loop.

Also in this routine, the statement for getting the operand and next token was broken into two so that the error can be changed to an "expecting operator or end-of-statement" error when get token returns an error (any parser error).  This error will need to be changed to the appropriate error by the caller.  For example, in an IF statement when it calls to get its expression, if this error occurs, the error needs to be changed to an "expected operator or THEN" error.

In the new translate routine, when setting the error message in the RPN output list, the error string is obtained from the token for a parser error instead of from the error status.  This also needed to be done before an unused token is deleted (tokens with parser errors are marked as unused).

Finally, the set operand state function of the parser was removed.  This was always and only called before calling the token routine, so an operand state argument was added to this function.  Several parser error tests were added to expression test #1.  It was also discovered that the autoenums.h include file was not always being regenerated when the Token class source file was modified, which turned out to be a wrong dependency listed in the CMake build file.

[commit a194cdbfb6] [commit 348ec2230b]

New Translator – Simple Expressions (Tagged)

The implementation for simple expressions is now complete in the new translator and version v0.4.0 has been tagged.  Note that expression test #1 currently fails with the regression and memory test scripts because of problems with the expression mode in the old translator routines.  The new translator routines run this test successfully, but should not yet be used on any of the other expression tests.  Also, parser errors are not yet handled correctly by the new translator routines, but this is going to require a major change to how token errors are handled.

[commit 552aa6176a]

New Translator – Memory Leaks

Several of the error conditions caused a memory leak because the token at which the error occurred was not deleted.  The error token can't just be deleted because it may be in the RPN output list.  Tokens in the output list will be deleted when the list is clears when an error is detected.  A method was needed to determine when an error token should be deleted.

This was accomplished by added a new UnUsed sub-code.  At the locations where an error detected, the routine setting the error needs to set this sub-code if the token has not been added to the output list.  This sub-code was set in two places, one in the get operand routine when there is a command or operator token, and the other in the translate routine that called the get expression routine when the terminating token is not the end-of-line token.

One other problem that caused a uninitialized variable used error from valgrind was also in the translate routine when the terminating token is checked for the end-of-line token.  The check also needed to test if the token has a table entry before checking the token for the end-of-line code (non-table entry tokens don't have a code).

[commit a9b45327d6]

New Translator – Expression Error Checks

There were several error conditions that were not being caught correctly by the new translator routines.  The get expression routine was also rewritten with a loop instead of recursively calling itself, which should be slightly more efficient and use slightly less stack space for complex expressions.

The first error condition occurred when a command or operator token is found when getting an operand.  When this occurs, the proper "expected XX expression" error needed to be return, which is why the get operand routine contains the data type argument.

The second error condition occurred with the token after an operand in the end of expression token check.  This check tested whether the token was an operator, however, both operators and command type tokens are reported as operators.  The test was changed to specifically check for the operator token type.

The third error condition occurred when the hold stack is cleared of higher precedence operators.  The check included obtaining the precedence of the current token and whether the token was a unary operator.  However, these assumed the token had a valid code (was in the table), but the token could be any type.  A different precedence function was used (taking a token instead of a code argument and handles any token type), and a new is unary operator function was implemented that takes a token argument that handles any token types.  Precedences for all the token types also needed to be assigned (only some of them were assigned precedences previously).

Several more tests were added to expression test #1 for testing many of the error conditions.  Note that the old translator expression mode has trouble with some of these error tests because the routine is not as robust as the new routines.  The problems only occur in the expression mode.  These problems will not be corrected, so for the time being expressions test #1 will fail when using the old translator (including the regression and memory test scripts).

[commit fab9d675a6] [commit fc7398879f]

New Translator – Simple Expressions

To be able to use the new translator routines, two new temporary command lines options were added, namely ‑n and ‑ne, which do the same thing as ‑t and ‑te except that the new translator routine is called.  These will be removed once the new translator has been fully implemented.

Several new expression tests were added to expression test #1 (simple expressions) including a test of the "‑E" expression (which previously reported an error), several more unary operator tests (including multiple unary operators), many with mixed data types (double and integer) and some with invalid mixed data type types (number and string).

When running the old translator, it was discovered that a negative constant at the beginning of the expression was interpreted as a negate and a positive constant instead of a negative constant.  This occurred because the parser was not set to the operand state being that the translator was in its initial state, so a check was added to set the parser operand state if in the initial state and expression mode is selected.

Currently for simple expressions in the new translator routines, parser errors are not being handled correctly and therefore not reported correctly.  Expression test #1 does not contain any expressions demonstrating this (some will be added once this issue is resolved).  The old translator routines crash on these when using expression mode (command mode does not have a problem).  The new translator is also not handling invalid operands and operators correctly.

[commit 58a03cb51b]

New Translator – Get Expression Design

Five new functions were implemented to support simple expressions (simple operands, unary and binary operators) in the new translator and two existing translator functions are utilized.  The names of new translator routines that conflict with existing functions are being temporarily suffixed with a '2' character.  This functions will replace the existing functions once the new translator is fully implemented and working.

The new routines consist of the top level translate routine (currently only supports expression mode); the get expression routine for getting an expression (to be utilized by any command needing an expression), which will return the token terminating the expression; the process operator routine for handling the precedence of operators; the get operand routine for getting an operand (simple operands to start); and the get token routine for getting a token from the parser.  Click Continue... for some more details of each of these new routines.