The SimpleStack class is used for the new command stack, which will hold a CmdItem structure consisting of a token pointer and a code. The code will initially be set from the token's index, but may be changed to other associated codes as commands are processed by the Translator. Since the hold and done stacks are also simple stacks, these were changed from the List class to the SimpleStack class. This only required minor changes to push and pop calls for these stacks.
As previously hinted, the Translator status enumeration needed to be moved outside the Translator class before the TableEntry structure. The enumeration was renamed TokenStatus as that seemed appropriate for the return value of the Translator's add token function. Each of the enumeration values were also renamed except for the BUG statuses, which were left alone.
A previously mentioned, the mode must be changed from Command to Assignment upon receiving the LET command token. Some commands will need to change the mode from Command to Expression (PRINT, IF, WHILE, etc.). Some commands will need to change the mode from Command to a new End-of-Statement mode, since nothing is expected after command keyword (END IF, DO, LOOP, etc.).
Having an every growing switch statement of the command code is not efficient. Therefore, a next token mode value was added to the TableEntry structure. When a command token is received, if the mode is currently Command, then the mode will be changed to the command's table entry next token mode value. The command will be pushed onto the command stack along with it's current code. No further action needs to be taken until the rest of the statement is processed.
Since there will be mode values in the TableEntry structure, the Translator mode enumeration was also moved before TableEntry and renamed TokenMode (again an appropriate name). The enumeration values were also renamed to reflect this. For now, if the next token mode is not set for a command (a new Null token mode value), then a “not yet implemented” bug error occurs. If the current mode is not Command, then a new “unexpected command” message occurs.
For the LET command, the next token mode value is Assignment. When an assignment operator is added to the output list, if there is a LET command on the command stack, it is popped and the assignment operator's token's LET sub-code flag is set. For now, since the command stack should be empty, the end-of-line processing only needs to check if the command stack is empty. Later it will need to process commands on the stack.
Sunday, May 30, 2010
Saturday, May 29, 2010
Parser – Bug / Test Updates
During the testing of the LET command implementation, some problems were discovered. The first problem was in the Parser's get number token function. The simple single 0 digit caused an “invalid leading zero in number constant” error. This error check was to prevent a constant like 01 from being accepted, but it did allow a leading zero when followed by a decimal point like in 0.1. This fix was to check if the next character is a digit, and if it's not, then it terminates looking for more characters, and then otherwise cause the error.
Due to this error, it was a good idea to re-run all of the parser tests since it has been a while since these were checked. Low and behold, they all failed or miscompared. The print token function used by these tests hadn't been updated for the changes to the code and data type enumerations, so these were updated. Once corrected, there were still miscompares, but these were due to either the decimal code values changing (because of all the new codes that have been added) or because the data type of many operators and internal functions were changed from None to their appropriate data type. Additional 0 constant test inputs were added to the Parser number test inputs.
The regression test scripts were updated to also include the parser tests. The Windows batch file uses the comp command, which has a nice feature that a wild card can be used for both files names and it is able to compare all sets of files with one command (like all 8 translator test output files). However, it also has an irritating feature where after it's done comparing files, it asks if there are more files to compare. No has to be entered to continue. There is not option to prevent this. Since there are two compares (one for the parser files, one for the translator files), no has to be entered after the comparing the parser file before the translator files are compared.
One last problem was discovered that affects both the print token and print small token functions. Numeric constants were not being output as intended. The code was outputting an integer using the C %d format specifier and doubles using the %g specifier. The problem is that if the constant 1.0 (a double) was entered, it would be output as 1 making it impossible to know if it was an integer or a double. The raw strings entered for the numbers were suppose to be output - the reason these strings were saved in the first place, to preserve the original string for later output by the Recreator.
Due to this error, it was a good idea to re-run all of the parser tests since it has been a while since these were checked. Low and behold, they all failed or miscompared. The print token function used by these tests hadn't been updated for the changes to the code and data type enumerations, so these were updated. Once corrected, there were still miscompares, but these were due to either the decimal code values changing (because of all the new codes that have been added) or because the data type of many operators and internal functions were changed from None to their appropriate data type. Additional 0 constant test inputs were added to the Parser number test inputs.
The regression test scripts were updated to also include the parser tests. The Windows batch file uses the comp command, which has a nice feature that a wild card can be used for both files names and it is able to compare all sets of files with one command (like all 8 translator test output files). However, it also has an irritating feature where after it's done comparing files, it asks if there are more files to compare. No has to be entered to continue. There is not option to prevent this. Since there are two compares (one for the parser files, one for the translator files), no has to be entered after the comparing the parser file before the translator files are compared.
One last problem was discovered that affects both the print token and print small token functions. Numeric constants were not being output as intended. The code was outputting an integer using the C %d format specifier and doubles using the %g specifier. The problem is that if the constant 1.0 (a double) was entered, it would be output as 1 making it impossible to know if it was an integer or a double. The raw strings entered for the numbers were suppose to be output - the reason these strings were saved in the first place, to preserve the original string for later output by the Recreator.
Translator – Token Sub-Codes (Implementation)
The sub-code flag was implemented, which consisted of adding the sub-code memory to the Token class, adding the sub-code flag value definitions and modifying the print token test routine to output the flags.
The setting of the parentheses sub-code flag was handled in the do pending parentheses function, which checks if unnecessary parentheses were entered and appended a dummy parentheses token. The code was changed to set the parentheses sub-code flag of the last token appended to the output. Two issues were discovered.
The first issue found was if two sets of unnecessary parentheses are entered, for example, A=((B)), then the parentheses sub-code flag can only be set once. Upon reproducing the original source, the Recreator will not know that there were two sets of unnecessary parentheses. So for this case, a dummy parentheses token will still be appending for each additional set of unnecessary parentheses entered.
Curiously, with this change, if three (or and odd number of) unnecessary sets of parentheses are entered, for the third (fifth, etc.) set, the parentheses sub-code flag gets set in the second (fourth, etc.) set's dummy parentheses token. Something the Recreator will need to handle.
The second issue found was if the last token appended was a hidden conversion code, the conversion code's token parentheses sub-code flag gets set. It is anticipated that this would cause a problem for the Recreator. The Recreator should be able to safely ignore the conversion codes, but if it needs to look for sub-code flags, these can't be ignored. To avoid this, the code was modified that if token's table entry has the new Hidden flag set, then the item previous to the conversion code has its parentheses sub-code set.
The setting of the comma sub-code flag was handled in the new equal token handler function. When an equal token is received, if the mode was a multiple comma assignment, then the comma sub-code flag is set when the token's code is set to the assign list operator. The comma sub-code flag will not be set of the mode was a multiple equal assignment.
The setting of the parentheses sub-code flag was handled in the do pending parentheses function, which checks if unnecessary parentheses were entered and appended a dummy parentheses token. The code was changed to set the parentheses sub-code flag of the last token appended to the output. Two issues were discovered.
The first issue found was if two sets of unnecessary parentheses are entered, for example, A=((B)), then the parentheses sub-code flag can only be set once. Upon reproducing the original source, the Recreator will not know that there were two sets of unnecessary parentheses. So for this case, a dummy parentheses token will still be appending for each additional set of unnecessary parentheses entered.
Curiously, with this change, if three (or and odd number of) unnecessary sets of parentheses are entered, for the third (fifth, etc.) set, the parentheses sub-code flag gets set in the second (fourth, etc.) set's dummy parentheses token. Something the Recreator will need to handle.
The second issue found was if the last token appended was a hidden conversion code, the conversion code's token parentheses sub-code flag gets set. It is anticipated that this would cause a problem for the Recreator. The Recreator should be able to safely ignore the conversion codes, but if it needs to look for sub-code flags, these can't be ignored. To avoid this, the code was modified that if token's table entry has the new Hidden flag set, then the item previous to the conversion code has its parentheses sub-code set.
The setting of the comma sub-code flag was handled in the new equal token handler function. When an equal token is received, if the mode was a multiple comma assignment, then the comma sub-code flag is set when the token's code is set to the assign list operator. The comma sub-code flag will not be set of the mode was a multiple equal assignment.
Translator – Token Handlers (Implementation)
A TokenHandler typedef was needed to define the pointer to token handler function. I was not able to define the function pointer directly on the variables. This always proves difficult with more complex types, especially involving pointers. Fortunately, using typedef simplifies the issue. Here is the definition that was inserted before the TableEntry class:
The code that handles operators (which was after the switch statement for the special codes) was also put into a handler function. This greatly simplified the code at the end of the add token function. For processing the token, the temporary token handler function pointer is set to the code's table entry value. If the value is not set (is NULL), then the temporary pointer is set to the default operand token handler function. The function is called using the temporary pointer and it's status return value is immediately returned.
The program was compiled several times during the making of the changes. I got tired of running of the test cases (to output is redirected to a file in the base directory) and comparing to the official test output files (in the test directory), so I wrote a MSYS (bash) script to do it automatically and check all the test cases. An equivalent Windows batch file was also written, but does not work near as nice. Both will be included in the next release.
class Translator; // forward reference to Translator classThe TokenHandler type definition is then used for defining an token handler function pointers. Each of the token handlers were created using the existing code in the switch statement, where the code was modified to add the Translator pointer in front of all the Translator variables and the Translator scope (Translator::) was added to the Translator enumeration values.
typedef TokenStatus (*TokenHandler)(Translator &p, Token *&token);
The code that handles operators (which was after the switch statement for the special codes) was also put into a handler function. This greatly simplified the code at the end of the add token function. For processing the token, the temporary token handler function pointer is set to the code's table entry value. If the value is not set (is NULL), then the temporary pointer is set to the default operand token handler function. The function is called using the temporary pointer and it's status return value is immediately returned.
The program was compiled several times during the making of the changes. I got tired of running of the test cases (to output is redirected to a file in the base directory) and comparing to the official test output files (in the test directory), so I wrote a MSYS (bash) script to do it automatically and check all the test cases. An equivalent Windows batch file was also written, but does not work near as nice. Both will be included in the next release.
Friday, May 28, 2010
Translator – Token Handlers
The Translator's add token function is becoming quite large and at the current rate will become much larger as the different commands are implemented. It's never a good idea to have giant functions. Therefore, this routine needs to be broken up into separate functions. The code that contains the special token code handling (for equal, comma, closing parentheses, and end-of-line) will be separate functions. These functions will be called token handlers. Most of the commands will also have token handlers.
Having a switch statement on the token code where each case calls a token handler function is not the most efficient implementation. The concern is not execution time, but the amount of code lines required, in other words, a large switch statement. A better implementation is to store a function pointer in each table entry where a code requires a token handler. The Translator will check if there is a function pointer and then call the function to process the token.
And here is where it gets complicated. Pointers to member functions are not allowed. So the token handlers can't be Translator member function. But they need access to all of the Translator data members. The solution was to make them C++ friend functions of the Translator class with an reference argument to the Translator instance (a Translator function will pass *this).
The token handlers functions will also require a reference to the current token pointer (a reference so that it can be changed for an error) and will return the status of the operation. The Status enumeration needs to be moved out of the Translator because in order to define the return value for the token handler function pointer, it needs to be defined before TableEntry (and Table). But Table needs to be defined before TableEntry. Catch 22.
A forward reference for the Translator class can be added before TableEntry (needed for the Translator reference argument, which is simply class followed by the class name and a semicolon), but not for enumerations inside the Translator. Therefore the status enumeration will be moved out of Translator and renamed to TokenStatus (appropriate for a token handler, and the word Token is shorter than Translator).
Having a switch statement on the token code where each case calls a token handler function is not the most efficient implementation. The concern is not execution time, but the amount of code lines required, in other words, a large switch statement. A better implementation is to store a function pointer in each table entry where a code requires a token handler. The Translator will check if there is a function pointer and then call the function to process the token.
And here is where it gets complicated. Pointers to member functions are not allowed. So the token handlers can't be Translator member function. But they need access to all of the Translator data members. The solution was to make them C++ friend functions of the Translator class with an reference argument to the Translator instance (a Translator function will pass *this).
The token handlers functions will also require a reference to the current token pointer (a reference so that it can be changed for an error) and will return the status of the operation. The Status enumeration needs to be moved out of the Translator because in order to define the return value for the token handler function pointer, it needs to be defined before TableEntry (and Table). But Table needs to be defined before TableEntry. Catch 22.
A forward reference for the Translator class can be added before TableEntry (needed for the Translator reference argument, which is simply class followed by the class name and a semicolon), but not for enumerations inside the Translator. Therefore the status enumeration will be moved out of Translator and renamed to TokenStatus (appropriate for a token handler, and the word Token is shorter than Translator).
Thursday, May 27, 2010
Translator – LET Command
There is nothing special to say about the LET command – the format of assignment statements, less the optional LET keyword, has already been defined and implemented. When the Translator receives the LET command token, it will be pushed on the new command stack. The mode will be Command when the LET is received and, as the Translator is currently implemented, needs to left set to Command mode for the assignment statement to be processed as currently implemented.
When the assignment token is added to the output list, it will first check if there is a LET command token on top of the command stack, and if there is, then the LET sub-code flag will be set in the assignment token, the LET command will be popped from the command stack, and the LET token will be deleted.
There is a problem. The mode must be set to Command for the LET token to be accepted. The mode must be set to Command for the assignment statement. Some detection is necessary to prevent double LET keywords. To solve this problem, a new mode is needed, an Assignment mode. For most of the Translator, both Command and Assignment mode will be equivalent except for the processing of command tokens.
When the assignment token is added to the output list, it will first check if there is a LET command token on top of the command stack, and if there is, then the LET sub-code flag will be set in the assignment token, the LET command will be popped from the command stack, and the LET token will be deleted.
There is a problem. The mode must be set to Command for the LET token to be accepted. The mode must be set to Command for the assignment statement. Some detection is necessary to prevent double LET keywords. To solve this problem, a new mode is needed, an Assignment mode. For most of the Translator, both Command and Assignment mode will be equivalent except for the processing of command tokens.
Wednesday, May 26, 2010
Translator – Command Processing
When processing tokens in a statement, it is necessary to know what command is currently being processed. For example, the SPC and TAB functions are only valid in the PRINT statement. If the command is pushed onto the hold stack, there will be other tokens on top of the command token. Therefore the current command can't be determined by looking at the top of the hold stack. This means another variable would be needed to store the current command.
Commands can also be nested within a line (for example, an assignment or PRINT inside an IF-THEN-ELSE statement), so a stack is still the ideal solution. The command tokens need to be pushed onto a separate stack from the hold stack, a command stack. The precedence of a command token will still be used to empty the hold stack, but the command token will be pushed onto the command stack instead. The current command can be accessed by looking at the top of the command stack.
Before getting started with designing and implementing the commands, it's important to mention that there is no attempt being made to follow any particular standard or existing BASIC for this project. Being a heavy GW-Basic user in the 80's and early 90's, a lot of the inspiration for this project comes from that experience.
Multiple statements will be supported – as way to fit more code on the screen. In GW-Basic, the is a general lack of any rigid multiple line control structures. There are limited ones like FOR-NEXT, WHILE-WEND, DEF-ENDDEF, but these are easily abused (mainly due to the interpreted nature of GW-Basic). Factor in the GOTO, ON-GOTO GOSUB and ON-GOSUB commands (not being supported here), and GW-Basic code could look very ugly.
For this project, there will be new modern control structures including some structures that don't exist in other BASIC or other languages. It is time to get started on the commands, which will be defined before implemented. One command has already been implemented, the assignment statement, less the optional LET keyword, so it's now time to implement this optional keyword...
Commands can also be nested within a line (for example, an assignment or PRINT inside an IF-THEN-ELSE statement), so a stack is still the ideal solution. The command tokens need to be pushed onto a separate stack from the hold stack, a command stack. The precedence of a command token will still be used to empty the hold stack, but the command token will be pushed onto the command stack instead. The current command can be accessed by looking at the top of the command stack.
Before getting started with designing and implementing the commands, it's important to mention that there is no attempt being made to follow any particular standard or existing BASIC for this project. Being a heavy GW-Basic user in the 80's and early 90's, a lot of the inspiration for this project comes from that experience.
Multiple statements will be supported – as way to fit more code on the screen. In GW-Basic, the is a general lack of any rigid multiple line control structures. There are limited ones like FOR-NEXT, WHILE-WEND, DEF-ENDDEF, but these are easily abused (mainly due to the interpreted nature of GW-Basic). Factor in the GOTO, ON-GOTO GOSUB and ON-GOSUB commands (not being supported here), and GW-Basic code could look very ugly.
For this project, there will be new modern control structures including some structures that don't exist in other BASIC or other languages. It is time to get started on the commands, which will be defined before implemented. One command has already been implemented, the assignment statement, less the optional LET keyword, so it's now time to implement this optional keyword...
Tuesday, May 25, 2010
Translator – Commands and Precedence
In the translated RPN list, the command token will generally end up at the end of the statement. To accomplish this, the command tokens need to be saved until the rest of the statement is translated. The best place to save the command tokens is on the hold stack with a very low precedence to keep them on the stack until the end of statement.
The appropriate precedence of commands would appear to be the same as the EOL token, so that when the EOL token is processed, the command tokens will be emptied from the stack. This implies that commands could empty other commands, but generally, this will not occur except for multiple keyword commands (like the IF-THEN-ELSE and FOR-TO-STEP commands).
The EOL token currently has the same precedence as assignment, closing parentheses, and comma tokens. It makes sense for assignment and commands to be the same precedence because assignment is technically a command. An assignment will never empty a command from the hold stack because the Translator never receives an assignment operator. An equals token is received as an equality operator, which has a higher precedence, so it won't empty any lower precedence commands. It's only after the stack emptying that an equals token may be changed to an assignment token.
Closing parentheses and comma tokens, being the same precedence, will empty all tokens with higher or same precedence, which is a problem, since command tokens will be the same precedence. This is currently not an issue because:
The appropriate precedence of commands would appear to be the same as the EOL token, so that when the EOL token is processed, the command tokens will be emptied from the stack. This implies that commands could empty other commands, but generally, this will not occur except for multiple keyword commands (like the IF-THEN-ELSE and FOR-TO-STEP commands).
The EOL token currently has the same precedence as assignment, closing parentheses, and comma tokens. It makes sense for assignment and commands to be the same precedence because assignment is technically a command. An assignment will never empty a command from the hold stack because the Translator never receives an assignment operator. An equals token is received as an equality operator, which has a higher precedence, so it won't empty any lower precedence commands. It's only after the stack emptying that an equals token may be changed to an assignment token.
Closing parentheses and comma tokens, being the same precedence, will empty all tokens with higher or same precedence, which is a problem, since command tokens will be the same precedence. This is currently not an issue because:
- An EOL token is not pushed onto the hold stack and therefore can't be emptied by a closing parentheses or comma.
- An assignment operator is not on the hold stack when a multiple assignment comma is processed, so the comma won't empty an assignment token.
- An assignment token is on the hold stack when a comma is processed within an array or function – the lower precedence array or function token will be on the stack before the assignment token, so it won't empty the assignment token.
- An assignment token is on the hold stack when a comma is processed not within an array or function (though possibly after an opening parentheses) – this is an unexpected comma and an error occurs.
- A closing parentheses may empty an assignment token from the stack, but since no opening parentheses, array or function token is found on the stack, an error occurs.
Monday, May 24, 2010
Translator – Token Sub-Code
The internal code will contain sub-codes that will be used by the Recreator, thus eliminating the need to put dummy codes into the program. The Translator knows when information will be needed by the Recreator in reproducing the original source. A sub-code member will be added to the token class so that the Translator can pass this information to the Encoder. The Encoder will set the sub-codes in the internal code words.
Instead of adding the dummy parentheses token to the output list, the Translator will now set the parentheses sub-code in the last token appended to the output list. The closing parentheses token used for the dummy token is no longer needed, and will be deleted. The token output routine in the test code will output a ')' at the end of a token when the parentheses sub-code is present.
Similarly, if the LET keyword is present in front of an assignment, then the LET sub-code will be set in the assignment operator token. The token output routine in the test code will output a 'LET' at the end of a token when the LET sub-code is present. Before knowing exactly how this will be accomplished, it is necessary to defined how commands will be processed...
Instead of adding the dummy parentheses token to the output list, the Translator will now set the parentheses sub-code in the last token appended to the output list. The closing parentheses token used for the dummy token is no longer needed, and will be deleted. The token output routine in the test code will output a ')' at the end of a token when the parentheses sub-code is present.
Similarly, if the LET keyword is present in front of an assignment, then the LET sub-code will be set in the assignment operator token. The token output routine in the test code will output a 'LET' at the end of a token when the LET sub-code is present. Before knowing exactly how this will be accomplished, it is necessary to defined how commands will be processed...
Sunday, May 23, 2010
Internal Code Format
The encoded BASIC program will be stored in memory in a very compact form. It will not be any kind of linked list of tokens like come out of the Translator. A linked list would have way too much overhead during execution and would waste a lot of memory. Instead, the program will be very simple format where each code will take up a single 16-bit word.
A 16-bit word has plenty of space to support all the codes, which currently stands at 137 (there will be lot more, but far less than the 65,536 numbers possible in a 16‑bit word). This handles the actual codes (operators, internal functions, and commands), but what about entered identifiers like variables, arrays, define and user functions, constants, and so on?
Entered identifiers will have an index value into a table, known as the Dictionary, which will contain the information about each identifier (like name, data type, number of array dimensions, size of array dimensions, number of arguments, argument data types, constant values, etc.). The index to the Dictionary entry will be stored in the program. During run-time, much of the information in the Dictionary will not be used.
These indexes will be preceded by a code. For example, one of these codes will push a double variable value onto the evaluation stack. The routine for this code will know to get the next 16-bit word that will contain the Dictionary index. Using a 16-bit word for the index implies a maximum of 65,536 Dictionary entries. This should be sufficient since subroutines and functions will each have their own Dictionary, with each having a maximum 65,536 Dictionary entries.
The 16-bit code word has more bits than are needed for the code value. The extra space can be user for other things. One use would be for information like the presence of unnecessary parentheses or the optional LET keyword. This information will be called a sub-code and ignored by run-time module, but would be used by the Recreator. The detail of the internal program doesn't need to defined right now, just the knowledge that there will be a sub-code present in the internal code. Next, what this means for the Translator...
A 16-bit word has plenty of space to support all the codes, which currently stands at 137 (there will be lot more, but far less than the 65,536 numbers possible in a 16‑bit word). This handles the actual codes (operators, internal functions, and commands), but what about entered identifiers like variables, arrays, define and user functions, constants, and so on?
Entered identifiers will have an index value into a table, known as the Dictionary, which will contain the information about each identifier (like name, data type, number of array dimensions, size of array dimensions, number of arguments, argument data types, constant values, etc.). The index to the Dictionary entry will be stored in the program. During run-time, much of the information in the Dictionary will not be used.
These indexes will be preceded by a code. For example, one of these codes will push a double variable value onto the evaluation stack. The routine for this code will know to get the next 16-bit word that will contain the Dictionary index. Using a 16-bit word for the index implies a maximum of 65,536 Dictionary entries. This should be sufficient since subroutines and functions will each have their own Dictionary, with each having a maximum 65,536 Dictionary entries.
The 16-bit code word has more bits than are needed for the code value. The extra space can be user for other things. One use would be for information like the presence of unnecessary parentheses or the optional LET keyword. This information will be called a sub-code and ignored by run-time module, but would be used by the Recreator. The detail of the internal program doesn't need to defined right now, just the knowledge that there will be a sub-code present in the internal code. Next, what this means for the Translator...
Translator – Commands (Introduction)
Now that expressions have been fully implemented in the Translator (at least until more language features are added), it's time to start implementing the BASIC commands. In fact, one command has already been implemented, the assignment statement without the optional LET keyword.
It was mentioned some time ago that there was a way to eliminate the need for the dummy close parentheses codes. These dummy codes are put into the translated output where unnecessary parentheses were entered into an expression, so that the Recreator would know to reproduce them. This will prevent confusion from having entered parentheses just disappear. During run-time, these dummy codes would be skipped.
It turns the same method to get rid of these dummy parentheses codes will also be used for the LET command, first command to be implemented. There are already 8 assignment codes, and there will be 4 more once temporary strings are fully implemented in the Encoder. All of these assignment codes are for without the optional LET. Either another dummy code is needed for the LET or each of the assignment operator needs to duplicated when the optional LET is included – that's 12 more codes.
Neither of these two alternatives is desirable. There is a third alternative that will help eliminate the need for these dummy codes, but some look ahead planning is required into the design of the format for how the BASIC program code will be stored internally. In fact, as each BASIC command is implemented, so forward planning is necessary into how the command will be stored in memory and executed during run-time. Next a preliminary design of the internal code...
It was mentioned some time ago that there was a way to eliminate the need for the dummy close parentheses codes. These dummy codes are put into the translated output where unnecessary parentheses were entered into an expression, so that the Recreator would know to reproduce them. This will prevent confusion from having entered parentheses just disappear. During run-time, these dummy codes would be skipped.
It turns the same method to get rid of these dummy parentheses codes will also be used for the LET command, first command to be implemented. There are already 8 assignment codes, and there will be 4 more once temporary strings are fully implemented in the Encoder. All of these assignment codes are for without the optional LET. Either another dummy code is needed for the LET or each of the assignment operator needs to duplicated when the optional LET is included – that's 12 more codes.
Neither of these two alternatives is desirable. There is a third alternative that will help eliminate the need for these dummy codes, but some look ahead planning is required into the design of the format for how the BASIC program code will be stored internally. In fact, as each BASIC command is implemented, so forward planning is necessary into how the command will be stored in memory and executed during run-time. Next a preliminary design of the internal code...
Saturday, May 22, 2010
Translator – Multiple String Assignments (Release)
Upon contemplating the string flags in the table entries, I realized that care must be taken to make sure the string flag is set in the table entries when an operator or function code has a string operand. A better design is to set the string flag automatically during table initialization if any of the operands of the string data type. Therefore, the string flags were removed from the table entries and the Table constructor was modified set the flags in the table entries automatically.
To correct the issue of saving all the operands for string list assignments, a temporary simple stack is used to save the operands as they are popped off of the done stack and processed. Since the last two operands (the value being assigned and the last item in list) have already been popped and processed by the find code routine, these are pushed to the simple stack before processing the rest of the operands. Each additional operand processed is also pushed to this stack.
After the list operands are processed for a string list assignment a new array needs is allocated and filled from the temporary simple stack. In order to determine the size of this array, either the operands need to be counted as they are processed, or the number of items in the stack needs to be accessed. Since the simple stack already knows how many items it has, a new access function was added to the SimpleStack class to return the number of items.
It had been decided previously that mixed strings and sub-strings would be allowed in a multiple list assignment statement. Having a mix string list assignment replaces the need for a separate sub-string list assignment code, the mix-string list assignment will handle this case. So, a new AssignListMixStr associated code was added to AssignList (this new code has a sub-string as the first operand, the value being assigned). The list assignment handling code was modified to detect if the list contains both reference strings and sub-strings. If it does, then the token is changed to this new AssignListMixStr code.
Several new sub-string and mix-string assignments were added to test inputs. This completes string handling in the Translator. The code now handles expressions and assignment statements and ibcp_0.1.11-src.zip has been uploaded at Sourceforge IBCP Project along with the binary for the program. Next the real meat of the Translator begins, translating actual BASIC commands...
To correct the issue of saving all the operands for string list assignments, a temporary simple stack is used to save the operands as they are popped off of the done stack and processed. Since the last two operands (the value being assigned and the last item in list) have already been popped and processed by the find code routine, these are pushed to the simple stack before processing the rest of the operands. Each additional operand processed is also pushed to this stack.
After the list operands are processed for a string list assignment a new array needs is allocated and filled from the temporary simple stack. In order to determine the size of this array, either the operands need to be counted as they are processed, or the number of items in the stack needs to be accessed. Since the simple stack already knows how many items it has, a new access function was added to the SimpleStack class to return the number of items.
It had been decided previously that mixed strings and sub-strings would be allowed in a multiple list assignment statement. Having a mix string list assignment replaces the need for a separate sub-string list assignment code, the mix-string list assignment will handle this case. So, a new AssignListMixStr associated code was added to AssignList (this new code has a sub-string as the first operand, the value being assigned). The list assignment handling code was modified to detect if the list contains both reference strings and sub-strings. If it does, then the token is changed to this new AssignListMixStr code.
Several new sub-string and mix-string assignments were added to test inputs. This completes string handling in the Translator. The code now handles expressions and assignment statements and ibcp_0.1.11-src.zip has been uploaded at Sourceforge IBCP Project along with the binary for the program. Next the real meat of the Translator begins, translating actual BASIC commands...
Friday, May 21, 2010
Translator – Sub-String Assignments (Release)
After testing the Table initialization code that checks maximum number of operands and maximum number of associated codes, several sub-string assignments statements were added and tested. Then some sub-string assignments with error were added.
One of the types of errors being tested were the assignment of temporary strings like MID$(A$+B$,2)=C$. There error was correctly detected, but the error was pointing to the MID$, which could be confusing. Therefore, a change was made that for an assignment, if the expected reference flag is not set, then if the token without the reference flag is a sub-string function (its data type is sub-string), then the error token is set to the token if the first operand of the sub-string function. So for this statement, the error will be pointing to the + of the A$+B$.
During testing of the sub-string assignments, one of tests tried was the mixing of regular strings and sub-string, for example A$,LEFT$(B$,1),C$=D$. This did not work correctly (this type of statement will be allowed). Also noticed that for lists, the assignment operator output list item only had two operands (the last variable being assigned and the operand being assigned). It should contain all of the operands being assigned. So these are two more issues that need to corrected.
In any case, sub-string assignments are working so another developmental release is being made and ibcp_0.1.11-dev-3-src.zip has been uploaded at Sourceforge IBCP Project along with the binary for the program. Now to fix assignment list operands and mixing of string types...
One of the types of errors being tested were the assignment of temporary strings like MID$(A$+B$,2)=C$. There error was correctly detected, but the error was pointing to the MID$, which could be confusing. Therefore, a change was made that for an assignment, if the expected reference flag is not set, then if the token without the reference flag is a sub-string function (its data type is sub-string), then the error token is set to the token if the first operand of the sub-string function. So for this statement, the error will be pointing to the + of the A$+B$.
During testing of the sub-string assignments, one of tests tried was the mixing of regular strings and sub-string, for example A$,LEFT$(B$,1),C$=D$. This did not work correctly (this type of statement will be allowed). Also noticed that for lists, the assignment operator output list item only had two operands (the last variable being assigned and the operand being assigned). It should contain all of the operands being assigned. So these are two more issues that need to corrected.
In any case, sub-string assignments are working so another developmental release is being made and ibcp_0.1.11-dev-3-src.zip has been uploaded at Sourceforge IBCP Project along with the binary for the program. Now to fix assignment list operands and mixing of string types...
Thursday, May 20, 2010
Translator – Sub-String Assignments (Testing)
The changes to support sub-string assignment were implemented, which included adding the SubStr data type entries to the conversion code array in the match code routine; changing the data type to SubStr for the LEFT$, MID$ and RIGHT$ table entries; and added the sub-string reference checking. Initial testing started with the existing test inputs – no problems were discovered.
Upon trying a sub-string assignment, discovered that it didn't work because there was no AssignSubStr associated code, so this code was added along with its table entry. Now, many of the existing test inputs were failing. Next discovered that the maximum associated codes needed to be changed from 2 to 3 because Assign now had three associated codes. The wrong value was causing the find code routine to malfunction.
I thought it would be best to calculate both the maximum operands and maximum associated codes automatically during the Table initialization, so moving these constants to members of the Table class seemed to be the best solution. Unfortunately, these values need to be constants because they are used to define the sizes of several arrays.
It would still be prudent to have the the Table initialization at least check to make sure these constants agree with what was in the table entries. So code was added to the Table initialization to find the maximum operands and associated codes as it was scanning the entries for code checking. Two new table error types were added for these errors, which are reported by exceptions from the Table constructor. Testing continues...
Upon trying a sub-string assignment, discovered that it didn't work because there was no AssignSubStr associated code, so this code was added along with its table entry. Now, many of the existing test inputs were failing. Next discovered that the maximum associated codes needed to be changed from 2 to 3 because Assign now had three associated codes. The wrong value was causing the find code routine to malfunction.
I thought it would be best to calculate both the maximum operands and maximum associated codes automatically during the Table initialization, so moving these constants to members of the Table class seemed to be the best solution. Unfortunately, these values need to be constants because they are used to define the sizes of several arrays.
It would still be prudent to have the the Table initialization at least check to make sure these constants agree with what was in the table entries. So code was added to the Table initialization to find the maximum operands and associated codes as it was scanning the entries for code checking. Two new table error types were added for these errors, which are reported by exceptions from the Table constructor. Testing continues...
Wednesday, May 19, 2010
Translator – Sub-String Assignments
There will be several associated codes for assigning strings from a reference string or temporary string to a reference string or to a sub-string. The Encoder will determine which code to use once the string type for each operand is determined. For the Encoder to know if a sub-string is present, a new SubStr data type is needed, which only the LEFT$, MID$ and RIGHT$ functions return.
The Translator will check to make sure that sub-string assignments are valid by making sure the string argument is a reference string, at least if it is a Paren or NoParen token type. If it turns out to be a user function, then the Encoder will report the error. The Translator will also make sure that a compound sub-string assignment like LEFT$(RIGHT$(B$,3),2)="AB" is not entered.
The find code routine needs to handle sub-strings where the operands are popped off of the done stack and the reference flag is checked. For the first operand (last operand popped), if the data type of the token is SubStr (indicating a sub-string function) and the operand is a String, then the token's reference flag is set to operand's reference flag (the reference flag is transferred). The operand's reference flag is then cleared as it is currently implemented. This assumes that the first argument of the sub-string function is the string being assigned.
If the operand's data type is anything but String (SubStr or TmpStr), then the reference flag is not transferred. For the invalid compound sub-string assignment, since the data type of the operand will be SubStr, the reference flag of the function's token will not be set. When the reference flag is checked for an assignment operator, the reference flag will only be set if the string operand was reference, otherwise if won't be set and an error will be returned.
The Translator will check to make sure that sub-string assignments are valid by making sure the string argument is a reference string, at least if it is a Paren or NoParen token type. If it turns out to be a user function, then the Encoder will report the error. The Translator will also make sure that a compound sub-string assignment like LEFT$(RIGHT$(B$,3),2)="AB" is not entered.
The find code routine needs to handle sub-strings where the operands are popped off of the done stack and the reference flag is checked. For the first operand (last operand popped), if the data type of the token is SubStr (indicating a sub-string function) and the operand is a String, then the token's reference flag is set to operand's reference flag (the reference flag is transferred). The operand's reference flag is then cleared as it is currently implemented. This assumes that the first argument of the sub-string function is the string being assigned.
If the operand's data type is anything but String (SubStr or TmpStr), then the reference flag is not transferred. For the invalid compound sub-string assignment, since the data type of the operand will be SubStr, the reference flag of the function's token will not be set. When the reference flag is checked for an assignment operator, the reference flag will only be set if the string operand was reference, otherwise if won't be set and an error will be returned.
Tuesday, May 18, 2010
Sub-String Assignments
Many BASICs support sub-string assignments, the syntax that will be supported is the same used as in GW-Basic, QuickBASIC, FreeBASIC, etc.. Here is an example sub-string assignment along with it's translation:
At the A$, a reference string is pushed on the evaluation stack, that is, its pointer and length are copied to the stack. When the MID3$ is processed, it will pop the 2 and 5 off of the stack. The pointer on top of the stack will be changed to point to the fifth character in A$ and the length is set to 2 (assuming that A$ is at least 6 characters long).
When the AssignSubStr is processed, the "AB" string constant will be popped off of the stack. Two characters of this value are copied directly to the pointer that is on top of the stack (which is pointing to the fifth character in A$). A different AssignSubStr vs. AssignStr code is required because no allocation occurs, only a copy to an existing character array. An AssignSubStrTmp code is also required for the case that the value being assigned is a temporary string, which needs to be deleted after the characters are copied.
Since LEFT$ and RIGHT$ are also sub-string functions, there is no reason these functions can't also be used to assign sub-strings – and as it turns out, no extra code is required. Next, how sub-string assignments will be handled by the Translator...
MID$(A$,5,2) = "AB" A$ 5 2 MID3$ "AB" AssignSubStrNotice the new code AssignSubStr and the lack of <ref> on A$. When the MID$ is processed by the Translator, the reference flags of its operands are cleared, hence no A$<ref>. However, the MID3$ needs to have its token reference flag set when the assign operator checks for a reference, since that MID3$ token will be on the done stack. Consider how this statement is processed at run-time.
At the A$, a reference string is pushed on the evaluation stack, that is, its pointer and length are copied to the stack. When the MID3$ is processed, it will pop the 2 and 5 off of the stack. The pointer on top of the stack will be changed to point to the fifth character in A$ and the length is set to 2 (assuming that A$ is at least 6 characters long).
When the AssignSubStr is processed, the "AB" string constant will be popped off of the stack. Two characters of this value are copied directly to the pointer that is on top of the stack (which is pointing to the fifth character in A$). A different AssignSubStr vs. AssignStr code is required because no allocation occurs, only a copy to an existing character array. An AssignSubStrTmp code is also required for the case that the value being assigned is a temporary string, which needs to be deleted after the characters are copied.
Since LEFT$ and RIGHT$ are also sub-string functions, there is no reason these functions can't also be used to assign sub-strings – and as it turns out, no extra code is required. Next, how sub-string assignments will be handled by the Translator...
Subscribe to:
Posts (Atom)