Translator – Debugging - Sub-String Assignments (II)

The next issue was more difficult to resolve and was a strange problem because statements that appeared to cause the program to crash, worked by themselves. Using the process of elimination, the statement below was finally found to be causing the crash:

RIGHT$(A$+B$,2) = C$

When the comma token is received, the +$ token is added to the output list and is pushed on the done stack. The RIGHT$ token is on top the hold stack with its reference flag set because it is a sub-string assignment statement. Being a sub-string assignment, the first operand must have the reference flag set, which for the +$ token was not, hence an error.

This error is detected in the find code routine where it is checking for a reference of the first operand when the token passed in has its reference flag set. If the reference flag of the operand is not set, an error is returned. The passed in token is first deleted and the error token is set to the token of the first operand.

The problem in this case was that the passed in token, the sub-string function, that gets deleted, is still on top of the hold stack. When an error occurs on a line, one of the things that the cleanup routine does is delete all the tokens on the hold stack. The sub-string function token gets deleted twice, which causes a crash later on – allocated memory must not be freed more than one.

The solution upon these reference errors is to delete the token in the find code routine only if the token is not an internal (sub-string) function. The find code is only called from two places where the token will have its reference flag set: internal sub-string functions in sub-string assignment statements (which are on the hold stack and should not be deleted) and assignment tokens (which should be deleted).

The next problem is that currently, the error reported against the + in the statement above is “item can not be assigned” but should be “expected comma” instead. This problem affects several other statement all related to the fact that a string variable is expected and the error message should reflect this...

Translator – Debugging - Sub-String Assignments

There were no issues with the seventh (Data Type Assignment) Translator tests other than the expected text file needed to be updated for all the new operator codes. The eighth (Sub-String Assignment) tests were all failing with a bug message indicating the done stack was empty when it was expecting operands and one of the tests were crashing the program.

The done stack empty errors were caused because the string within the sub-string that was being assigned was already popped off of the done stack by the assignment (variables being assigned do not need to be attached to the assignment token). But when the sub-string function was processed, it expected to find the first operand, which is a string, on the done stack. A check needed to be added to the process final operand function to not count string operands for functions that have the reference flag set (which is only set for sub-string functions on the left side of an assignment statement).

The next issue for the sub-string assignment tests, was that the Translator was incorrectly adding an AssignStr code instead of the correct AssignSubStr code. This problem was caused because the find code routine was using the wrong condition to check for an exact data type match when looking at the operands of the main and associated codes to find the proper code for the operand. The code obtained from the conversion code table was used – the exact match check was if the conversion code was the Null code. This allowed the Sub-String data type to match the AssignStr that was expecting a String data type instead of continuing to check the AssignSubStr, which expects the Sub-string data type.

To correct this problem, the exact match check was changed to compare the actual data type to the code's operand's data type. This is the check the old find code routine used. Also, the section that inserts the conversion code was modified to only add the conversion code from the conversion table if it is not the Null code. In other words, the Null code indicates that the data type can be converted, but no actual conversion code is needed.

Translator – Debugging - String Assignments

Debugging has continued; problems with the third (Array/Function Parentheses Expressions), fourth (Internal Functions), and fifth (Assignments) Translator tests were corrected. There were some done stack not empty errors from the sixth (Data Type) test with the string assignment tests.

The problem turned out to be that the string variable being assigned was not popped from the done stack because it was a string. (Strings that are not determined to be temporary strings are left on the done stack so that they can be popped later and attached to the item that uses them – like an operator, function or an array.) However, these string variables were not being popped and attached to the assignment command/operator.

This was by design, the thinking was that items being assigned are definitely variables or arrays and not function calls. (Functions can also be assigned, but for these assignments, the function names will not have arguments and are only permitted within the function body.)  In any case, variables being assigned do not need to be attached to the assignment command, including string variables.

To solve this issue, in the find code routine where non-string values are popped from the done stack – an additional check was added that if the token needed to be a reference, then it will be popped from the done stack regardless of its data type. The reference flag is checked when the calling token has its reference flag set (which is set only for assignment operators).

Translator – New Operand Code (Debugging II)

The Translator was not processing the equal operator properly (the test expression was “not A < 5 = B > 2”) where the equal was translated to a =%2 (EqI2) and it should have been =% (EqInt). The problem was that the Equal token handler needs to process first operand like operators when expression (equal operator), so a call to the find code routine was added.

The Translator was not generating the correct output for strings (the test expression was “a$ = "this" + "test"”) where the attached operands was just strange. The problem was found in the process final operand routine, which was not popping enough string operands from the done stack to attach to the + (CatStr) and = (EqStr) tokens because the loop prematurely ended before popping the last string operand. The loop needed to be terminated with a  “––i >= 0” instead of a “––i > 0”.

From this last test expression, I realized that there is no need to save string operands that are constant strings. Constant strings will be coded the same as variables (in other words, string constants will have dictionary entries just like variables) and are Strings and not Temporary Strings (they don't need to be deleted during execution). For now, it will be assumed that the Encoder will take of this issue.

So far, the first (Simple Expressions) and second (Parentheses Expressions) Translator tests are working correctly though some of the expected results needed to be updated for the new associated codes. However, the third (Array/Function Parentheses Expressions) Translator test is crashing on the very first test expression. Debugging continues...

Translator – New Operand Code (Debugging)

There were several problems in the Table that needed to be fixed, all caught by the check code during initialization (so it is a good thing to have the initialization checks to significantly cut down on debugging time). Some of the problems were in the check code itself, but these were easily corrected.

The first problem was the check for the index to the second operand associated codes – to make sure that the index is not greater than the number of associated codes in the array. This check can only performed if the index is greater than zero because otherwise the code either does not have a second set of associated codes or the index is zero (which does not need to be checked). Several entries had the wrong index to the second operand associated codes the needed to be corrected.

The last problem was the checks added for multiple entries (e.g. MID$) where an entry with the multiple flag set, the name of the function of the next table entry must be the same and the next entry must have expression information present – these checks were reversed (reporting an error when the entries were correct).

Once the Table initialization succeeded, the regression tests were run, where all of them failed including the Parser tests. The Parser tests failures were due to the function name array in the test code was not updated for all the new associated codes that were added to the table. To prevent having to update this array in the future (which is a pain), the process was somewhat automated.

A new awk script was created (similar to the existing codes.awk) that processes the ibcp.h include file scanning for the xx_Code enumeration values and creating a list of quoted strings separated by commas (except the last) with the name of the code, i.e. the “xx” in the enumeration value. The output of this file is put into an include file and an include statement was added into the array to get these strings.

Without using a more sophisticated make system, the awk scripts still need to be run manually when the Code enumeration is modified.  The Parser tests now all succeed so debugging continues with Translator tests...

Translator – New Operand Processing

Once it appeared all the changes were made for the new operand processing (new find code and process final operand routines), the process of getting it compile began. Several corrections were made. Once most of the errors were corrected, one problem remained.

Previously there was an operand array in the Translator class that held pointers to the output list items. This array was filled in the old find code routine and used by the callers of find code to attached to a newly created PRN output item. The array is not necessary for the new find code and so was removed. However, the array was still be used by the Assign command handler, which filled the array with the value being assigned to be used when creating the output item.

The operand array was only used locally so the Assign command handler was updated to have a local variable for holding the operand of the value being assigned. If this operand is a string, then a one element operand pointer is allocated to attach to the RPN output item. Now that the code compiles successfully, debugging can begin...

Translator – Process Final Operand

The rewrite of the find code routine and the Translator has been a major undertaking. In addition, another project has taken priority (and this project is for my actual paying job). Fortunately I'm getting to the end of the modifications for the new find code and will be able to start testing soon.

There is some code that calls the new find code that was very similar in the add operator routine and the close parentheses token handler and so a new function was implemented, which will process the final operand.

This new process final operand routine will call the find code routine for the last or only operand of either an operator or an internal function token. The non-temporary string operands for the operator or internal function will be counted. These are the operands that need to be saved with the operator or internal function token in the output list.

This new routine will also be called for arrays and non-internal function tokens from the close parentheses token handler. All the operands for these tokens need to be saved with the token (the Encoder will sort out what to do with these operands). To identify these tokens, the number of operands will be passed as an argument (the value will be zero for operators and internal functions).

This routine will allocate an array of operand pointers for the number of operands to save. The pointers to the operands will be popped from the done stack and put into the array. A new RPN list item will be created and the operands will be attached to it, and the RPN item will be appended to the output list. The output list item will be pushed to the done stack unless the token was an internal print function.

Translator – New Find Code (Multiple Assignment)

The last issue for the new find code implementation is the most tricky and requires the elimination of the C-like multiple equal assignment (for example, A=B=C=0). The problem occurs when the items being assigned are array elements, which can be shown in this example statement:

A(I) = B(I) = 5

In this example, the A can be assumed to be an array because a function (with arguments) cannot be assigned. The subscripts must be numeric expressions. However, after the first equal, this statement can be interpreted two ways depending on whether B is an array or a function call:

I CvtInt A(<ref> I CvtInt B(<ref> 5 AssignList
I CvtInt A(<ref> I B([I] 5 = CvtDbl Assign

The first interpretation is a multiple list assignment of two array elements and the second interpretation is a single assignment to the result of an equality comparison between the result of a function call and a constant. The two resulting translations are radically difference. The Encoder can't be expected to change one translation into the other once it determines whether B is an array or a function.

A similar problem can occur with multiple sub-string assignments. These problems occur because an equal can be one of two different operators, assignment and equality. C resolves this issue by having difference operators for assignment (=) and equality (==). If this C-like multiple equal assignment is eliminated, then the rules are greatly simplified.

Therefore, in an expression, any equals will be equality operators. In a statement, only the first equal is the assignment operator. After this equal, an expression follows so any equals will be equality operators. Multiple assignments can still be performed by using the comma to separate the items being assigned.

Translator – New Find Code (Multiple Flag)

The next issue for the new find code implementation is with internal functions that have different number of arguments (MID$, INSTR and ASC). Currently the number of arguments is checked at the closing parentheses before the call to find code to check the data types of the arguments. Before calling find code, if the number of arguments was not correct, but the Multiple Flag was set, then a search for another code was made where the number of arguments did match.

For the new implementation, the data types of the internal function arguments will be checked as each argument is processed – in other words, at the comma token (or close parentheses token for the last argument). The checking for an alternate form of the internal function will now take place in the comma token handler.

Previously to support checking if a comma was valid in an internal function call (as opposed to a closing parentheses), the internal function table entry with the most number of arguments had to first in the table because otherwise an error would occur since a closing parentheses would be expected if the table entry with the smaller number of arguments was listed first. The number of arguments would be checked, and the code changed if necessary, in the closing parentheses token handler.

Now with the comma token handler taking care of changing the code when needed, the smaller number of arguments table entry needs to be first. If a comma is received where a parentheses is expected (at the last argument), a check will be made and if the Multiple Flag is set then it will move to the next table entry with the next number of argument (otherwise an “expected closing parentheses” error occurs).

This method implies that the table entries for the number of arguments must be in order by number of arguments. If there are three forms of an internal function (there are currently none), then the Multiple Flag should be in each table entry except the last. To insure the table entries are placed in the table correctly, a check will be added to the table initialization along with the other current table checks.

Translator – New Find Code (Reference Flag)

here has been a struggle with a number of issues that have turned up with the new find code implementation, including reference checking, internal functions with multiple forms (different number of arguments), and multiple assignments (specifically multiple equal assignments, which may need to be removed from the language).

The current find code routine checks if the first argument is a reference if the table entry for the token has the Reference Flag. This was used for the assignment operators when called from the set assign command routine, which is called from the comma and equal token handlers (assignment operators have the Reference Flag). If this would be the only location that needs to check for a reference flag, the checking could be moved from the find code routine to the set assign command routine.

However, there is another situation that needs to check for a reference flag - the first argument to a sub-string assignment. This is currently handled by an additional check in the find code routine. With the new find code, this will occur at the first comma for the sub-string function (from the comma token handler). So the reference check should still be done in the new find code routine.

The reference flag can't be added to the sub-string table entries since not all instances of the sub-string functions are assignments. The token being checked (internal function, operator, assignment code or print code) is passed to the find code routine. A simply method would be to set the reference flag in this token argument if it's operand needs to be a reference. The find code would check for a reference if the token's reference flag is set.

Therefore, in the set assign command, the reference flag of the assignment token will be set. Also, when internal functions are first pushed to the hold stack, a check is made to see if the mode is currently one of the assignment modes and only a sub-string function is permitted. At this time, the reference flag of the sub-string function can be set.

There will be no problem leaving the reference flag set in the internal function or operator token since nothing downstream will be checking for a reference flag on these type of tokens.

Translator – Find Code (New Design)

The current find code routine pops all of the operands for an operator or internal function from the done stack, checks the data type of each changing the code to an associated code as needed, and inserts hidden conversion codes as needed.  The find code routine is currently called from these locations:

add operator – to get the appropriate code for unary and binary operators
add print code – to get the data type specific print code
set assign command – to get the data type specific assignment code
close parentheses handler – to check the arguments for internal functions

The Translator will be modified to find the appropriate binary operator code at the first operand, and find the appropriate binary operator code again at the second operand. Also for internal functions, each argument will be checked as it is processed, in other words, in the comma handler.

The new find code will only check one operand where the index of the operand to check will be passed as an argument. The callers of find code will be modified:

operator handler – call for first operand of a binary operator
add operator – call for second operand of a binary (or operand of a unary) operator
comma handler – call for current argument of an internal function
close parentheses handler – call for current argument of an internal function

Translator – Operators (New Codes)

Table entries for the new codes for the Double/Integer (I2) and Integer/Double (I1) were added for the operators. It is not necessary to have all the operand combinations for the logical operators (AND, OR, etc.) that produce an integer result for integer operands. These should be used with integers, but double operands will be allowed and the Translator will continue to insert hidden CvtInt codes for these.

Likewise for the integer division operator (\) that's meant to take double operands and produce an integer result. There's no reason to have separate versions for integer operands as the regular division operator can be used for these. So hidden CvtDbl codes will be inserted for integer operands.

Translator – Operators (New Design)

The new design for operators will be that each data type combination will have it's own code. There will be associated codes for the first operand and associated codes for the second operand. For efficiency, there will be just one associated codes array with the second operand associated codes after the first operand associated codes. In addition to the number of associated codes value, there will be a new secondary associated codes index that will point to the first second operand associated, which can also be used to determine the end of the first operand associated codes.

Again using plus as an example, the main codes along with associated codes are list below. The convention will still be that the default operator has double arguments. The “I1” and “I2” codes represent the first or second operand being an integer, and the “Int” code for two integer operands. Similarly for strings where the “T1” and “T2” codes represent the first or second operand being a temporary string, and the “TT” code will for two temporary string operands.

Add  (4, 3)  AddI1, CatStr, CatStrT1, AddI2
AddI1  (1, 0)  AddInt
CatStr  (1, 0)  CatStrT2
CatStrT1  (1, 0)  CatStrTT

The second operand associated codes are underlined. The first number in parentheses is the number of associated codes and the second underlined number is the start index of the second operand associated codes. The decision for this design was arrived at by looking at the assembly language generated for approximately what the run-time will look like. Information about the assembly language research after the Continued break...

Translator – Operators (Change Needed)

A problem has surfaced as the new table entries were being added for temporary strings. The problem is unrelated to strings, but is related to integers. The plan was to make use of a associated codes as each operand is processed. Using plus operator as an example, there would have been these codes (with their return and operand data types):

Add  (Dbl)  Dbl Dbl  (also handles Int Dbl and Dbl Int with CvtDbl)
AddInt  (Int)  Int Int
CatStr    (Tmp)  Str Str
CatStrT1  (Tmp)  Tmp Str
CatStrT2  (Tmp)  Str Tmp
CatStrTT  (Tmp)  Tmp Tmp

The Add code would have the associated codes of AddInt, CatStr and CatStrT1. In other words, one code for each of the four data types of the first operand. For handling the second operand for strings, the CatStr code would have an associated code of CatStrT2, and the CatStrT1 code would have an associated code of CatStrTT. The main code would be selected if the second operand was a string (i.e. could not be determined to be a temporary string), and the associated code would be selected if the second operand was a temporary string.

With this scheme, a problem occurs when the first operand is an integer, where the AddInt code would be selected. But what happens when the second operand is a double? The code needs to revert to the Add code since integers need to be promoted to doubles (doubles do not demote to integers).

This could be handled by making the Add code an associated code of the AddInt code. But a CvtDbl needs to be added for the first operand, but the first operand is no longer being saved (except for strings that are not temporary). Which means that integer first operands would need to be saved. Anyway, this all started to get rather complicated. I have some ideas on how to resolved this, but some test code needs to be tried to see which idea is better...

Translator – String Processing (Change)

The planned changes for operator and internal functions where their operands (arguments) will no longer be kept on the done stack conflicts with the way strings are handled.

If an operator or internal function has string operands, all the operands are attached to the operator or internal function. This is necessary for string operands because the Translator has insufficient information to determine if generic string tokens (with and without a parentheses) are  temporary or not. Temporary strings need different codes at run-time because the temporary strings are deleted when no longer needed. The Encoder will determine if these tokens refer to variables and arrays (not temporary) or functions (temporary).

As mentioned, all operands are attached, which is unnecessary because only the string operands are actually needed. Further, the Translator can determine if some strings are temporary, like the result of the concatenate operator and most of the internal functions that return a string. The sub-string functions only return a temporary string if their argument is a temporary string.

To resolve this conflict between saving the string operands and the new operand processing, if a string operand (or argument) cannot be determined to be a temporary string, it will be left on the done stack. When the operator or internal function is appended to the output, it's specific code for the data type has already been determined. If the code has one or more string data type operands (not temporary string operands), the string operands will be counted to determine how many strings were left on the done stack. These strings will be popped and attached to the operator or internal function token.

Translator – Operand Processing (Change)

Previously, operands were processed (checking types and inserting conversion codes) after all of the operands were received. The operands are pushed to the done stack. Operators are temporarily pushed to the hold stack and are removed before a lower precedence operator is pushed to the hold stack (the method of rearranging operators according to their precedence).

Similarly for functions and arrays that are not processed until the matching closing parentheses token is received, at which time all the arguments or sub-scripts are on the done stack. No type checking can be performed for non-internal functions and arrays, so their operands are saved with (attached to) the array or function token.

A change has already been implemented for multiple list assignments where each assignment item (variable or sub-string) is checked as received, at either comma or equal tokens. Something similar can also be performed for internal functions, in fact, some logic has already been added that determines if a comma or closing parentheses token is valid after each argument. Logic can be added to also check the data type of the argument, adding conversion operators when necessary.

It has also been determined that for a binary operator, it data type specific code is needed on the hold stack based on the first operand of an operator (for possible error reporting). If necessary, a conversion code will be added at the same time. And it is now not necessary to keep the first operand on the done stack, so it can be popped. When the operator is emptied from the hold stack, only the second operand needs to be checked.

It is possible that the operator code may need to be changed from the code found based on the first operand. Currently there is only one operator that has more than one code with the same first operand data type but different data types for the second operand (Power and PowerMul, discussed here). The multiple table flag can be used to identify these type of operators (this is the same flag used to identify internal functions with different codes for the number of arguments). When set, the Translator will look for the correct code instead of inserting a conversion code.