Translator – Sub-String Implementation

Implementation of the transfer of operands of sub-strings will mostly take place in the process final operand function. But first, some information needs to be added to the Table, namely the number of string arguments each code has. This value could be calculated on the fly as needed, but it will make for a simpler design to just put these values in the table. The table initialization code (Table constructor) was modified to calculate these values automatically.

In the process final operand routine, the internal function and operator section, when the number of operands passed in is zero, needs to set the number of operands to the number of string arguments that are on the done stack. These operands will be popped from the done stack and attached to the internal function or operand token.  Only string operands were left on the done stack (other numeric operands were popped by the find code routine as they do not need to be attached to any token).

However, for sub-string functions, which have only one string operand, the string operand will be left on the done stack, to be attached to the next token. If the next token is another sub-string function, then the string operand argument will continue to be carried forward. Therefore, the number of (string) operands to pop from the done stack will only be set to the code's number of string arguments if the code is not a sub-string. For sub-string function, the number of operands will be left set to zero, so nothing will be popped and attached to the sub-string function token.

Translator – Operands of Sub-Strings

One final issue was noticed with the eighth (Sub-String Assignments) Translator test that needed to be corrected, which can be seen with these sample statements (and their current translations):

A$ = LEFT$(B$, 1)  A$<ref> B$ 1 LEFT$([B$] Assign$
A$ = LEFT$(B$+C$, 2)    A$<ref> B$ C$ +$[B$,C$] 2 LEFT$([+$] Assign$

In these examples, remember that the Translator cannot determine if B$ and C$ are variables or functions, therefore these operands are attached to the code they belong to so that the Encoder can adjust the operators and functions accordingly with associated codes with the appropriate arguments. However, sub-string functions add a wrinkle to this scheme.

During run-time, sub-string functions work the same whether their string operand is a reference or a temporary string – they will simply modify the string pointer and length of the string on top of the evaluation stack. Temporary strings are not deleted since the sub-string refers to the temporary string – whichever code has the sub-string function as an operand will take care of deleting the temporary strings. Only be one code for each sub-string function is needed (not two, one with a reference string and one with a temporary string as an operand). The correct translations of the statements above should be:

A$ = LEFT$(B$, 1)  A$<ref> B$ 1 LEFT$( Assign$[B$]
A$ = LEFT$(B$+C$, 2)  A$<ref> B$ C$ +$[B$,C$] 2 LEFT$( Assign$[+$]

In the first statement, the B$ should be attached to the Assign$ token so that the Encoder can change it to an AssignStrTmp code if B$ turns out to be a function (results of functions are  temporary strings) or leave it the AssignStr code if it is a variable. In the second statement, the Translator can change it to an AssignStrTmp code with no attached token, but until temporary strings are fully support, it will attach the +$ to an AssignStr code.

Since the result of a sub-string function is the same as its operand (reference or temporary string), the operands of the sub-string function simply get transferred to the token that has the sub-string function as an operand.

Translator – Debugging - Sub-String Assignments (III)

The next issue is the output of the correct error messages. Consider these expressions and the current error messages (underlined tokens indicate where the error is pointing):

RIGHT$(LEFT$(A$,1),2) = B$  item cannot be assigned
RIGHT$(A,2) = B$  expected string expression
RIGHT$(A%,2) = B$  expected string expression
RIGHT$(A$+B$,2) = C$  item cannot be assigned

The desired error message is “expected string variable” for the first three statement and “expected comma” for the fourth statement. The section that receives an internal function token and checks for a sub-string function in an assignment was previously corrected to accept sub-string functions and set their reference flag (so that the reference flag of the first operand is checked).

An additional check was needed to catch a second sub-string function. Checking if there is a token on top of the hold stack with its reference flag set is sufficient. This condition only occurs when there is a sub-string assignment. However, this check is only be performed when on the first operand.

When the find code routine found an operand with an incorrect data type, it was returning “expected <datatype> expression” errors. This was modified that if the reference flag of the token is set, then an expected variable error is returned. However, for strings, the “expected string item for assignment” error was returned to indicate sub-string assignments are allowed. But if already inside the sub-string function, this needed to be the “expected string variable” error.

For the last statement, a check was added to the operator section in the add token routine, the same if the token on top of the hold stack has its reference flag set on the first operand check, if the token is not a comma, then an “expected comma” error is returned.

Translator – Debugging - Sub-String Assignments (II)

The next issue was more difficult to resolve and was a strange problem because statements that appeared to cause the program to crash, worked by themselves. Using the process of elimination, the statement below was finally found to be causing the crash:

RIGHT$(A$+B$,2) = C$

When the comma token is received, the +$ token is added to the output list and is pushed on the done stack. The RIGHT$ token is on top the hold stack with its reference flag set because it is a sub-string assignment statement. Being a sub-string assignment, the first operand must have the reference flag set, which for the +$ token was not, hence an error.

This error is detected in the find code routine where it is checking for a reference of the first operand when the token passed in has its reference flag set. If the reference flag of the operand is not set, an error is returned. The passed in token is first deleted and the error token is set to the token of the first operand.

The problem in this case was that the passed in token, the sub-string function, that gets deleted, is still on top of the hold stack. When an error occurs on a line, one of the things that the cleanup routine does is delete all the tokens on the hold stack. The sub-string function token gets deleted twice, which causes a crash later on – allocated memory must not be freed more than one.

The solution upon these reference errors is to delete the token in the find code routine only if the token is not an internal (sub-string) function. The find code is only called from two places where the token will have its reference flag set: internal sub-string functions in sub-string assignment statements (which are on the hold stack and should not be deleted) and assignment tokens (which should be deleted).

The next problem is that currently, the error reported against the + in the statement above is “item can not be assigned” but should be “expected comma” instead. This problem affects several other statement all related to the fact that a string variable is expected and the error message should reflect this...

Translator – Debugging - Sub-String Assignments

There were no issues with the seventh (Data Type Assignment) Translator tests other than the expected text file needed to be updated for all the new operator codes. The eighth (Sub-String Assignment) tests were all failing with a bug message indicating the done stack was empty when it was expecting operands and one of the tests were crashing the program.

The done stack empty errors were caused because the string within the sub-string that was being assigned was already popped off of the done stack by the assignment (variables being assigned do not need to be attached to the assignment token). But when the sub-string function was processed, it expected to find the first operand, which is a string, on the done stack. A check needed to be added to the process final operand function to not count string operands for functions that have the reference flag set (which is only set for sub-string functions on the left side of an assignment statement).

The next issue for the sub-string assignment tests, was that the Translator was incorrectly adding an AssignStr code instead of the correct AssignSubStr code. This problem was caused because the find code routine was using the wrong condition to check for an exact data type match when looking at the operands of the main and associated codes to find the proper code for the operand. The code obtained from the conversion code table was used – the exact match check was if the conversion code was the Null code. This allowed the Sub-String data type to match the AssignStr that was expecting a String data type instead of continuing to check the AssignSubStr, which expects the Sub-string data type.

To correct this problem, the exact match check was changed to compare the actual data type to the code's operand's data type. This is the check the old find code routine used. Also, the section that inserts the conversion code was modified to only add the conversion code from the conversion table if it is not the Null code. In other words, the Null code indicates that the data type can be converted, but no actual conversion code is needed.

Translator – Debugging - String Assignments

Debugging has continued; problems with the third (Array/Function Parentheses Expressions), fourth (Internal Functions), and fifth (Assignments) Translator tests were corrected. There were some done stack not empty errors from the sixth (Data Type) test with the string assignment tests.

The problem turned out to be that the string variable being assigned was not popped from the done stack because it was a string. (Strings that are not determined to be temporary strings are left on the done stack so that they can be popped later and attached to the item that uses them – like an operator, function or an array.) However, these string variables were not being popped and attached to the assignment command/operator.

This was by design, the thinking was that items being assigned are definitely variables or arrays and not function calls. (Functions can also be assigned, but for these assignments, the function names will not have arguments and are only permitted within the function body.)  In any case, variables being assigned do not need to be attached to the assignment command, including string variables.

To solve this issue, in the find code routine where non-string values are popped from the done stack – an additional check was added that if the token needed to be a reference, then it will be popped from the done stack regardless of its data type. The reference flag is checked when the calling token has its reference flag set (which is set only for assignment operators).

Translator – New Operand Code (Debugging II)

The Translator was not processing the equal operator properly (the test expression was “not A < 5 = B > 2”) where the equal was translated to a =%2 (EqI2) and it should have been =% (EqInt). The problem was that the Equal token handler needs to process first operand like operators when expression (equal operator), so a call to the find code routine was added.

The Translator was not generating the correct output for strings (the test expression was “a$ = "this" + "test"”) where the attached operands was just strange. The problem was found in the process final operand routine, which was not popping enough string operands from the done stack to attach to the + (CatStr) and = (EqStr) tokens because the loop prematurely ended before popping the last string operand. The loop needed to be terminated with a  “––i >= 0” instead of a “––i > 0”.

From this last test expression, I realized that there is no need to save string operands that are constant strings. Constant strings will be coded the same as variables (in other words, string constants will have dictionary entries just like variables) and are Strings and not Temporary Strings (they don't need to be deleted during execution). For now, it will be assumed that the Encoder will take of this issue.

So far, the first (Simple Expressions) and second (Parentheses Expressions) Translator tests are working correctly though some of the expected results needed to be updated for the new associated codes. However, the third (Array/Function Parentheses Expressions) Translator test is crashing on the very first test expression. Debugging continues...

Translator – New Operand Code (Debugging)

There were several problems in the Table that needed to be fixed, all caught by the check code during initialization (so it is a good thing to have the initialization checks to significantly cut down on debugging time). Some of the problems were in the check code itself, but these were easily corrected.

The first problem was the check for the index to the second operand associated codes – to make sure that the index is not greater than the number of associated codes in the array. This check can only performed if the index is greater than zero because otherwise the code either does not have a second set of associated codes or the index is zero (which does not need to be checked). Several entries had the wrong index to the second operand associated codes the needed to be corrected.

The last problem was the checks added for multiple entries (e.g. MID$) where an entry with the multiple flag set, the name of the function of the next table entry must be the same and the next entry must have expression information present – these checks were reversed (reporting an error when the entries were correct).

Once the Table initialization succeeded, the regression tests were run, where all of them failed including the Parser tests. The Parser tests failures were due to the function name array in the test code was not updated for all the new associated codes that were added to the table. To prevent having to update this array in the future (which is a pain), the process was somewhat automated.

A new awk script was created (similar to the existing codes.awk) that processes the ibcp.h include file scanning for the xx_Code enumeration values and creating a list of quoted strings separated by commas (except the last) with the name of the code, i.e. the “xx” in the enumeration value. The output of this file is put into an include file and an include statement was added into the array to get these strings.

Without using a more sophisticated make system, the awk scripts still need to be run manually when the Code enumeration is modified.  The Parser tests now all succeed so debugging continues with Translator tests...

Translator – New Operand Processing

Once it appeared all the changes were made for the new operand processing (new find code and process final operand routines), the process of getting it compile began. Several corrections were made. Once most of the errors were corrected, one problem remained.

Previously there was an operand array in the Translator class that held pointers to the output list items. This array was filled in the old find code routine and used by the callers of find code to attached to a newly created PRN output item. The array is not necessary for the new find code and so was removed. However, the array was still be used by the Assign command handler, which filled the array with the value being assigned to be used when creating the output item.

The operand array was only used locally so the Assign command handler was updated to have a local variable for holding the operand of the value being assigned. If this operand is a string, then a one element operand pointer is allocated to attach to the RPN output item. Now that the code compiles successfully, debugging can begin...

Translator – Process Final Operand

The rewrite of the find code routine and the Translator has been a major undertaking. In addition, another project has taken priority (and this project is for my actual paying job). Fortunately I'm getting to the end of the modifications for the new find code and will be able to start testing soon.

There is some code that calls the new find code that was very similar in the add operator routine and the close parentheses token handler and so a new function was implemented, which will process the final operand.

This new process final operand routine will call the find code routine for the last or only operand of either an operator or an internal function token. The non-temporary string operands for the operator or internal function will be counted. These are the operands that need to be saved with the operator or internal function token in the output list.

This new routine will also be called for arrays and non-internal function tokens from the close parentheses token handler. All the operands for these tokens need to be saved with the token (the Encoder will sort out what to do with these operands). To identify these tokens, the number of operands will be passed as an argument (the value will be zero for operators and internal functions).

This routine will allocate an array of operand pointers for the number of operands to save. The pointers to the operands will be popped from the done stack and put into the array. A new RPN list item will be created and the operands will be attached to it, and the RPN item will be appended to the output list. The output list item will be pushed to the done stack unless the token was an internal print function.

Translator – New Find Code (Multiple Assignment)

The last issue for the new find code implementation is the most tricky and requires the elimination of the C-like multiple equal assignment (for example, A=B=C=0). The problem occurs when the items being assigned are array elements, which can be shown in this example statement:

A(I) = B(I) = 5

In this example, the A can be assumed to be an array because a function (with arguments) cannot be assigned. The subscripts must be numeric expressions. However, after the first equal, this statement can be interpreted two ways depending on whether B is an array or a function call:

I CvtInt A(<ref> I CvtInt B(<ref> 5 AssignList
I CvtInt A(<ref> I B([I] 5 = CvtDbl Assign

The first interpretation is a multiple list assignment of two array elements and the second interpretation is a single assignment to the result of an equality comparison between the result of a function call and a constant. The two resulting translations are radically difference. The Encoder can't be expected to change one translation into the other once it determines whether B is an array or a function.

A similar problem can occur with multiple sub-string assignments. These problems occur because an equal can be one of two different operators, assignment and equality. C resolves this issue by having difference operators for assignment (=) and equality (==). If this C-like multiple equal assignment is eliminated, then the rules are greatly simplified.

Therefore, in an expression, any equals will be equality operators. In a statement, only the first equal is the assignment operator. After this equal, an expression follows so any equals will be equality operators. Multiple assignments can still be performed by using the comma to separate the items being assigned.

Translator – New Find Code (Multiple Flag)

The next issue for the new find code implementation is with internal functions that have different number of arguments (MID$, INSTR and ASC). Currently the number of arguments is checked at the closing parentheses before the call to find code to check the data types of the arguments. Before calling find code, if the number of arguments was not correct, but the Multiple Flag was set, then a search for another code was made where the number of arguments did match.

For the new implementation, the data types of the internal function arguments will be checked as each argument is processed – in other words, at the comma token (or close parentheses token for the last argument). The checking for an alternate form of the internal function will now take place in the comma token handler.

Previously to support checking if a comma was valid in an internal function call (as opposed to a closing parentheses), the internal function table entry with the most number of arguments had to first in the table because otherwise an error would occur since a closing parentheses would be expected if the table entry with the smaller number of arguments was listed first. The number of arguments would be checked, and the code changed if necessary, in the closing parentheses token handler.

Now with the comma token handler taking care of changing the code when needed, the smaller number of arguments table entry needs to be first. If a comma is received where a parentheses is expected (at the last argument), a check will be made and if the Multiple Flag is set then it will move to the next table entry with the next number of argument (otherwise an “expected closing parentheses” error occurs).

This method implies that the table entries for the number of arguments must be in order by number of arguments. If there are three forms of an internal function (there are currently none), then the Multiple Flag should be in each table entry except the last. To insure the table entries are placed in the table correctly, a check will be added to the table initialization along with the other current table checks.

Translator – New Find Code (Reference Flag)

here has been a struggle with a number of issues that have turned up with the new find code implementation, including reference checking, internal functions with multiple forms (different number of arguments), and multiple assignments (specifically multiple equal assignments, which may need to be removed from the language).

The current find code routine checks if the first argument is a reference if the table entry for the token has the Reference Flag. This was used for the assignment operators when called from the set assign command routine, which is called from the comma and equal token handlers (assignment operators have the Reference Flag). If this would be the only location that needs to check for a reference flag, the checking could be moved from the find code routine to the set assign command routine.

However, there is another situation that needs to check for a reference flag - the first argument to a sub-string assignment. This is currently handled by an additional check in the find code routine. With the new find code, this will occur at the first comma for the sub-string function (from the comma token handler). So the reference check should still be done in the new find code routine.

The reference flag can't be added to the sub-string table entries since not all instances of the sub-string functions are assignments. The token being checked (internal function, operator, assignment code or print code) is passed to the find code routine. A simply method would be to set the reference flag in this token argument if it's operand needs to be a reference. The find code would check for a reference if the token's reference flag is set.

Therefore, in the set assign command, the reference flag of the assignment token will be set. Also, when internal functions are first pushed to the hold stack, a check is made to see if the mode is currently one of the assignment modes and only a sub-string function is permitted. At this time, the reference flag of the sub-string function can be set.

There will be no problem leaving the reference flag set in the internal function or operator token since nothing downstream will be checking for a reference flag on these type of tokens.

Translator – Find Code (New Design)

The current find code routine pops all of the operands for an operator or internal function from the done stack, checks the data type of each changing the code to an associated code as needed, and inserts hidden conversion codes as needed.  The find code routine is currently called from these locations:

add operator – to get the appropriate code for unary and binary operators
add print code – to get the data type specific print code
set assign command – to get the data type specific assignment code
close parentheses handler – to check the arguments for internal functions

The Translator will be modified to find the appropriate binary operator code at the first operand, and find the appropriate binary operator code again at the second operand. Also for internal functions, each argument will be checked as it is processed, in other words, in the comma handler.

The new find code will only check one operand where the index of the operand to check will be passed as an argument. The callers of find code will be modified:

operator handler – call for first operand of a binary operator
add operator – call for second operand of a binary (or operand of a unary) operator
comma handler – call for current argument of an internal function
close parentheses handler – call for current argument of an internal function

Translator – Operators (New Codes)

Table entries for the new codes for the Double/Integer (I2) and Integer/Double (I1) were added for the operators. It is not necessary to have all the operand combinations for the logical operators (AND, OR, etc.) that produce an integer result for integer operands. These should be used with integers, but double operands will be allowed and the Translator will continue to insert hidden CvtInt codes for these.

Likewise for the integer division operator (\) that's meant to take double operands and produce an integer result. There's no reason to have separate versions for integer operands as the regular division operator can be used for these. So hidden CvtDbl codes will be inserted for integer operands.

Translator – Operators (New Design)

The new design for operators will be that each data type combination will have it's own code. There will be associated codes for the first operand and associated codes for the second operand. For efficiency, there will be just one associated codes array with the second operand associated codes after the first operand associated codes. In addition to the number of associated codes value, there will be a new secondary associated codes index that will point to the first second operand associated, which can also be used to determine the end of the first operand associated codes.

Again using plus as an example, the main codes along with associated codes are list below. The convention will still be that the default operator has double arguments. The “I1” and “I2” codes represent the first or second operand being an integer, and the “Int” code for two integer operands. Similarly for strings where the “T1” and “T2” codes represent the first or second operand being a temporary string, and the “TT” code will for two temporary string operands.

Add  (4, 3)  AddI1, CatStr, CatStrT1, AddI2
AddI1  (1, 0)  AddInt
CatStr  (1, 0)  CatStrT2
CatStrT1  (1, 0)  CatStrTT

The second operand associated codes are underlined. The first number in parentheses is the number of associated codes and the second underlined number is the start index of the second operand associated codes. The decision for this design was arrived at by looking at the assembly language generated for approximately what the run-time will look like. Information about the assembly language research after the Continued break...