The code that handles checking the done stack and appending the necessary data type specific code was also put into a new add print code function so that it can be called from the comma token handler, semicolon token handler and new print command handler. Once the changes were implemented and made to compile, testing was started with the new regression test script.
The first problem encountered was that at the end of the line for the expression tests reported that the done stack was not empty, which was true because the expression only mode leaves the result on the done stack. The code was modified to save the expression mode from the start function and the end-of-line token handler checks for the expression only mode and makes sure there is one item on the done stack.
The next problem encountered was that the single PRINT command (the last test of the current test inputs) reported an “expected operand” at the end of the line. This occurred because after the PRINT command token was received, the mode was set to Expression and the state was set to Operand. The EOL token received next was considered an operator and therefore caused the error.
To correct this, a new EndExpr table entry flag was added and set in the EOL code table entry. This flag is also needed for the Comma and SemiColon codes and will be used for other codes that end expressions (for example, Colon, THEN, ELSE, END IF, etc.). If a token is received that has this flag set, then the normal operand/operator sections are skipped. The stack emptying section follows next where these codes will empty most of the operators from the hold stack. All of these codes will have token handlers.
Once all the existing test inputs worked, several PRINT test inputs were added. The next issue discovered was that both the comma and semicolon token handlers must switch Translator state back to Operand from Operator (an “expected operator” error was occurring). The are many other bugs in the PRINT statement code; debugging continues...
Sunday, June 6, 2010
Saturday, June 5, 2010
Translator – Command Handlers
The command handlers will be called at the end of the statement, which the end-of-line token handler is current performing. This will change when the end-of-statement is implemented (soon with the colon operator). For now, the end-of-line token handler will call the command handler for the command on top of the command stack.
The command handler functions are friends of the Translator class, like the token handler functions; necessary so that pointers to these functions can be put into the table entries. The interface to these functions are similar to the token handlers except a pointer to the command stack item is passed for the second argument instead of a reference to a token pointer. This gives the command handler access to the code and flag information in the command stack item in addition to the token pointer of the command. If an error occurs and a different token needs to be pointed to for the error, the token pointer can be changed (the original token needs to be deleted first).
The command handler functions are friends of the Translator class, like the token handler functions; necessary so that pointers to these functions can be put into the table entries. The interface to these functions are similar to the token handlers except a pointer to the command stack item is passed for the second argument instead of a reference to a token pointer. This gives the command handler access to the code and flag information in the command stack item in addition to the token pointer of the command. If an error occurs and a different token needs to be pointed to for the error, the token pointer can be changed (the original token needs to be deleted first).
- Command handlers will now be in charge of checking if an expression has been ended correctly and will make sure the done stack has been emptied. The end-of-line handler is responsible for:
- Checking if the command stack is not empty (otherwise a “command stack empty” bug error occurs).
- Checking if the command on top of the command stack has a command handler (otherwise a “not yet implemented” bug error occurs).
- Popping the command item from command stack and calling its command handler.
- Popping the initial null token from the hold stack.
- Checking if the done stack is empty.
- Checking if the command stack is empty.
- Deleting the EOL token and return a done status.
Friday, June 4, 2010
Translator – End of Statement Processing
The end-of-line processing currently performs the check if an expression was ended correctly, that is, checks if the hold stack contains only the initial null token (meaning all operator have been processed). The comma and semicolon should empty all operators from the hold stack, except lower precedence opening parentheses, functions, and array token. Except for the null token which acts as a buffer, if any of these are still on the stack, there is a missing closing parentheses. Finally, any pending parentheses needs to be processed.
This end of expression code needs to called by the comma and semicolon token handlers, so it will be moved into its own function (duplicating code is a bad programming practice). This function will also be called by future token handlers. In other words, it needs to be called at the end of a statement. The end-of-line is also the end of a statement, which currently contains this code.
However, the end of expression should not be called by the end-of-line token handler, which should instead be calling the command handler for the command that is currently being processed (on top of the command stack). The command handler will then do the end of expression call if required (some commands don't have expressions). The end-of-line token handler will pop the command from on top of the command stack and call the command's command handler.
Currently, when an assignment operator is appended to the output, the pointer to the output item is pushed to the done stack. There is no reason to push this to the done stack because it is not needed there (it is just popped at the end of the statement). The reason for pushing onto the done stack is so that the output item can be referenced by another token (for example, operands of functions for checking data types).
Remember the LET command was popped from the command stack (if there) by the assignment operator. The assignment operator, should instead be pushed onto the command stack since it is a command. It will be more clear why once the compound commands are implemented (like IF-THEN-ELSE). The assignment, being a command, will not get popped from the command stack (like all commands). The assignment command doesn't need a command handler, so it will have a NULL command handler function pointer.
This end of expression code needs to called by the comma and semicolon token handlers, so it will be moved into its own function (duplicating code is a bad programming practice). This function will also be called by future token handlers. In other words, it needs to be called at the end of a statement. The end-of-line is also the end of a statement, which currently contains this code.
However, the end of expression should not be called by the end-of-line token handler, which should instead be calling the command handler for the command that is currently being processed (on top of the command stack). The command handler will then do the end of expression call if required (some commands don't have expressions). The end-of-line token handler will pop the command from on top of the command stack and call the command's command handler.
Currently, when an assignment operator is appended to the output, the pointer to the output item is pushed to the done stack. There is no reason to push this to the done stack because it is not needed there (it is just popped at the end of the statement). The reason for pushing onto the done stack is so that the output item can be referenced by another token (for example, operands of functions for checking data types).
Remember the LET command was popped from the command stack (if there) by the assignment operator. The assignment operator, should instead be pushed onto the command stack since it is a command. It will be more clear why once the compound commands are implemented (like IF-THEN-ELSE). The assignment, being a command, will not get popped from the command stack (like all commands). The assignment command doesn't need a command handler, so it will have a NULL command handler function pointer.
Thursday, June 3, 2010
Translator – Commas with PRINT
Currently a comma is not valid within an expression if the comma is not inside an array or function. The comma has a low precedence, below all operators, but above end-of-line, close parentheses and commands, so it will empty all other operators from the hold stack. No lower precedence tokens will be on the hold stack except for an open parentheses, identifier with parentheses (function or array), function (internal or defined), or an assignment operator.
The comma token handler will be updated to check for a PRINT command on top of the command stack. Any other command will still cause an “unexpected comma in expression” error to occur. The hold stack needs to be checked to make sure it is empty (has the initial Null token on top), otherwise a “missing closing parentheses” error occurs. This check is the same that is performed for in semicolon token handler and also in the end-of-line token handler, so this code will be put into its own function for all to call. An assignment won't be on the hold stack if there is a PRINT on the command stack.
For a PRINT command, the handler will process any expression on top of the done stack. The expression on the done stack will be handled by the find code function as previously described. The flag for the PRINT command item on top of the done stack needs to be set in case this is the last comma in the PRINT statement, to tell the PRINT command handler not to append the Print code to the output (to keep the cursor on the same line).
If the done stack is empty, then there was no expression before the comma (allowed), which means to skip to the next column. The comma token will be appended to the output. For now, the PRINT command is the only reason a comma token is put into the translated output, therefore the execution routine called during run-time for the comma will be to advance to the next column. There is no need for a PrintComma code unless there will be another reason to put a comma in the program.
The comma token handler will be updated to check for a PRINT command on top of the command stack. Any other command will still cause an “unexpected comma in expression” error to occur. The hold stack needs to be checked to make sure it is empty (has the initial Null token on top), otherwise a “missing closing parentheses” error occurs. This check is the same that is performed for in semicolon token handler and also in the end-of-line token handler, so this code will be put into its own function for all to call. An assignment won't be on the hold stack if there is a PRINT on the command stack.
For a PRINT command, the handler will process any expression on top of the done stack. The expression on the done stack will be handled by the find code function as previously described. The flag for the PRINT command item on top of the done stack needs to be set in case this is the last comma in the PRINT statement, to tell the PRINT command handler not to append the Print code to the output (to keep the cursor on the same line).
If the done stack is empty, then there was no expression before the comma (allowed), which means to skip to the next column. The comma token will be appended to the output. For now, the PRINT command is the only reason a comma token is put into the translated output, therefore the execution routine called during run-time for the comma will be to advance to the next column. There is no need for a PrintComma code unless there will be another reason to put a comma in the program.
Wednesday, June 2, 2010
Translator – Semicolons with PRINT
Up to now, the Semicolon (an operator) has not been implemented (in fact, causes the program to crash if entered in an expression because its table entry has no expression information structure). The semicolon will have a low precedence, the same as the comma since it is also an expression separator, so it will empty all other operators from the hold stack. No lower precedence tokens will be on the hold stack except for an open parentheses, identifier with parentheses (function or array), function (internal or defined), or an assignment operator.
The Semicolon will have a token handler function, which is called after the hold stack is processed. The handler will check the command on top of the command stack. If the command stack is empty, then an “unexpected semicolon” error will occur. For now, only a PRINT command needs to be processed. Any other command will also cause an “unexpected semicolon” error to occur. The hold stack needs to be checked to make sure it is empty (has the initial Null token on top), otherwise a “missing closing parentheses” error occurs. An assignment won't be on the hold stack if there is a PRINT on the command stack.
For a PRINT command, the handler needs to process any expression on top of the done stack. If the done stack is empty, then there was no expression before the semicolon (allowed). In this case, a dummy semicolon token is appended to the output so that it can be reproduced. Adding a sub-code for this situation is a wasted effort (because there may be no token before the semicolon in the output, which would complicate matters in using a sub-code).
The expression on the done stack will be handled by the find code function as previously described. The flag for the PRINT command item on top of the done stack needs to be set in case this is the last semicolon in the PRINT statement – to tell the PRINT command not to append the Print code to the output (to keep the cursor on the same line).
The PrintTmpStr (which will be named PrintTmp to stick with the 3 character data type naming convention for code names) is not being added at this time. PrintStr is a code that takes a string operand, and as such, whether it's operand is a String or a TmpStr can't be determined by the Translator, so it will have it's operand saved for later determination by the Encoder.
The Semicolon will have a token handler function, which is called after the hold stack is processed. The handler will check the command on top of the command stack. If the command stack is empty, then an “unexpected semicolon” error will occur. For now, only a PRINT command needs to be processed. Any other command will also cause an “unexpected semicolon” error to occur. The hold stack needs to be checked to make sure it is empty (has the initial Null token on top), otherwise a “missing closing parentheses” error occurs. An assignment won't be on the hold stack if there is a PRINT on the command stack.
For a PRINT command, the handler needs to process any expression on top of the done stack. If the done stack is empty, then there was no expression before the semicolon (allowed). In this case, a dummy semicolon token is appended to the output so that it can be reproduced. Adding a sub-code for this situation is a wasted effort (because there may be no token before the semicolon in the output, which would complicate matters in using a sub-code).
The expression on the done stack will be handled by the find code function as previously described. The flag for the PRINT command item on top of the done stack needs to be set in case this is the last semicolon in the PRINT statement – to tell the PRINT command not to append the Print code to the output (to keep the cursor on the same line).
The PrintTmpStr (which will be named PrintTmp to stick with the 3 character data type naming convention for code names) is not being added at this time. PrintStr is a code that takes a string operand, and as such, whether it's operand is a String or a TmpStr can't be determined by the Translator, so it will have it's operand saved for later determination by the Encoder.
Tuesday, June 1, 2010
Translator – Print-Only Functions
There are currently two print-only functions, SPC and TAB. Both functions may only be used in a PRINT statement and are not valid in any other expression. There needs to be a new table entry Print flag that will be used for these print-only functions, which are defined as internal functions.
When internal functions are processed (arguments checked and necessary hidden conversion codes inserted), a check is needed that if a function has the Print flag, then the current command on top of the command stack must be a PRINT command otherwise an “invalid use of print function” error occurs. In additional, print-only functions do not have return values, so no result value is pushed onto the done stack. These print-only functions also need to set the flag in the PRINT command item on top of the command stack in case these are the last item in the PRINT statement (to prevent the PRINT code from being appended to the output, which will keep the cursor on the same line at run-time).
During execution at run-time, these functions perform their action directly to the output, which is the reason no result needs to be pushed back on the evaluation stack (the result is outputted instead).
Bug Fix:Upon making the changes for print-only functions, I realized there was an unrelated bug where a command would be accepted after receiving an operand (for example: A LET ...) because the mode was still set to Command. Therefore, when an operand token is processed, if the mode is Command then the mode is set to Assignment – this will prevent a command token from being accepted after an operand.
When internal functions are processed (arguments checked and necessary hidden conversion codes inserted), a check is needed that if a function has the Print flag, then the current command on top of the command stack must be a PRINT command otherwise an “invalid use of print function” error occurs. In additional, print-only functions do not have return values, so no result value is pushed onto the done stack. These print-only functions also need to set the flag in the PRINT command item on top of the command stack in case these are the last item in the PRINT statement (to prevent the PRINT code from being appended to the output, which will keep the cursor on the same line at run-time).
During execution at run-time, these functions perform their action directly to the output, which is the reason no result needs to be pushed back on the evaluation stack (the result is outputted instead).
Bug Fix:Upon making the changes for print-only functions, I realized there was an unrelated bug where a command would be accepted after receiving an operand (for example: A LET ...) because the mode was still set to Command. Therefore, when an operand token is processed, if the mode is Command then the mode is set to Assignment – this will prevent a command token from being accepted after an operand.
Monday, May 31, 2010
Translator – PRINT Command (Translating)
Currently, command tokens are pushed onto the command stack, so will a PRINT command token when it is received. The PRINT's next token mode value will be set to Expression. The Semicolon, Comma and End-of-statement will trigger the insertion of the value specific print code.
The existing find code function can be used to determine which specific print value code to append to the output. The PrintDbl code will serve as the main code, which will have PrintInt, PrintStr and PrintTmpStr as associated codes, each with one operand of the appropriate data type. A new token will be created with the PrintDbl code and passed to the find code function, which will return the correct code for the expression. The token is then be appended to the output.
At the end of the statement, the PRINT command token will be on top of the command stack. The end of statement processing needs to pop this token off from the stack and perform whatever handling is necessary. There needs to be a command handler that performs a command's specific needs. For reasons that will be clear later, the token handler function cannot be used for this purpose, so a new command handler function pointer is required in the table entries.
The PRINT command handler needs to determine when the PRINT token should be appended to the output (to go to a new line), or not (stay on the same line). If the done stack is not empty, then there is an unprocessed expression that needs a specific print value code. This means there was no semicolon or comma at the end of the statement, so the PRINT token can be appended. However, an empty done stack does not indicate no PRINT token is needed, consider the PRINT command by itself.
Therefore, a flag is needed, to be set when an print-only internal function, semicolon or comma is processed. If the done stack was empty, the command handler will check if this flag is set, to determine not to append the PRINT command token (in which case the PRINT token is deleted). This flag will be added to the command stack's item structure item, which is cleared when the command is initially pushed onto the stack.
The existing find code function can be used to determine which specific print value code to append to the output. The PrintDbl code will serve as the main code, which will have PrintInt, PrintStr and PrintTmpStr as associated codes, each with one operand of the appropriate data type. A new token will be created with the PrintDbl code and passed to the find code function, which will return the correct code for the expression. The token is then be appended to the output.
At the end of the statement, the PRINT command token will be on top of the command stack. The end of statement processing needs to pop this token off from the stack and perform whatever handling is necessary. There needs to be a command handler that performs a command's specific needs. For reasons that will be clear later, the token handler function cannot be used for this purpose, so a new command handler function pointer is required in the table entries.
The PRINT command handler needs to determine when the PRINT token should be appended to the output (to go to a new line), or not (stay on the same line). If the done stack is not empty, then there is an unprocessed expression that needs a specific print value code. This means there was no semicolon or comma at the end of the statement, so the PRINT token can be appended. However, an empty done stack does not indicate no PRINT token is needed, consider the PRINT command by itself.
Therefore, a flag is needed, to be set when an print-only internal function, semicolon or comma is processed. If the done stack was empty, the command handler will check if this flag is set, to determine not to append the PRINT command token (in which case the PRINT token is deleted). This flag will be added to the command stack's item structure item, which is cleared when the command is initially pushed onto the stack.
Translator – PRINT Command (Translation)
The translated statement PRINT will consist of a number of different printing codes, which include PrintDbl, PrintInt, PrintStr, PrintTmpStr, Comma, Semicolon, and Print. The print only Spc and Tab internal function codes will also print. Before explaining further, here are some examples with their translations:
At run-time, the Comma code will advance the cursor to the next column by outputting spaces. The Tab and Spc codes perform there action by outputting spaces. The Semicolon code does not perform any action during run-time, it is there only so that it can't reproduced during recreation of the source. Next, how the PRINT statement will be translated...
PRINT PrintNote that when the cursor is to go to a new line, there is a Print code at the end of the translation, otherwise it is absent. The Print code at run-time preforms the new line action, so in it's absence, the cursor remains on the same line. The PrintDbl, PrintInt, PrintStr and PrintTmpStr will output the value that is on top of the evaluation stack, with PrintTmpStr deleting the temporary string when done.
PRINT A;B% A PrintDbl B PrintInt Print
PRINT A$+B$,C$; A$ B$ + PrintTmpStr Comma C$ PrintStr
PRINT TAB(A+1);"Hello" A 1 + Tab "Hello" PrintStr Print
PRINT "Start";SPC(10) "Start" PrintStr 10 Spc
PRINT ,,A Comma Comma A PrintDbl Print
PRINT ; Semicolon
At run-time, the Comma code will advance the cursor to the next column by outputting spaces. The Tab and Spc codes perform there action by outputting spaces. The Semicolon code does not perform any action during run-time, it is there only so that it can't reproduced during recreation of the source. Next, how the PRINT statement will be translated...
Translator – PRINT Command
It is necessary to look at how PRINT statements will be executed to determine how they need to be encoded and thus translated. But first, here is the general syntax of the PRINT statement:
The SPC function outputs the number of spaces of the integer expression (less than 1 outputs no spaces). The TAB function advances the cursor to the column of the integer expression, where the first column is defined as column 1 (less than 1 is taken as column 1). If the cursor is already beyond the column specified, then PRINT goes to the next line, and then to the specified column. Using TAB(1) in the middle of a PRINT statement has the effect of doing a new line (an '\n' in C).
A semicolon is used to separate expressions, but does not affect what is output. The syntax above implies multiple semicolons may be entered. This is allowed, but will cause dummy semicolon tokens to be inserted into the code. A comma advances to the next column. A column will initially be defined as one-fifth of the screen width (at 80 columns, this is 5 columns of 16 characters). Eventually there will be a way for this width to be defined to any value.
PRINT [<expression>][;|, [<expression>]...]Zero or more expressions may be included. The expressions can result with a double, integer or string value. Numbers are output with a trailing space. The expression may be one of print only functions SPC and TAB, both taking an integer expression. After the expressions are output, the PRINT will move to a new line unless the line ends with a semicolon, comma, SPC or TAB. (More options will be added later like PRINT USING and PRINT to a file.)
The SPC function outputs the number of spaces of the integer expression (less than 1 outputs no spaces). The TAB function advances the cursor to the column of the integer expression, where the first column is defined as column 1 (less than 1 is taken as column 1). If the cursor is already beyond the column specified, then PRINT goes to the next line, and then to the specified column. Using TAB(1) in the middle of a PRINT statement has the effect of doing a new line (an '\n' in C).
A semicolon is used to separate expressions, but does not affect what is output. The syntax above implies multiple semicolons may be entered. This is allowed, but will cause dummy semicolon tokens to be inserted into the code. A comma advances to the next column. A column will initially be defined as one-fifth of the screen width (at 80 columns, this is 5 columns of 16 characters). Eventually there will be a way for this width to be defined to any value.
Sunday, May 30, 2010
Translator – LET Command (Release)
The LET command is implemented and working. A new set of translator test inputs were added for commands, with LET statements for each possible assignment (each data type, non-list and list) with the LET keyword were included. Three error test inputs were added also, one with a two LET keywords, one with a LET keyword in the middle, and a PRINT (to test the “not yet implemented” error).
The plan was to release a whole series of development releases (0.1.12-dev-X) starting with the LET command, but some many changes were made to the code (including the parser correction, parser test updates, token handlers, etc.) from the 0.1.11 release, that this release will be an official Translator development release (0.1.12). This will probably continue for each command added to the Translator. Sub-developmental releases may still be used for the more complex commands.
This release also contains regression test scripts that were used during the latest round of changes. The scripts saved a lot of command line typing due to all the reorganization of the code (token handlers, token status and token mode changes). A Windows batch file equivalent is also included, but does not work near as nice because of the limitations of the Window compare utility. Both scripts delete any current output files, runs the tests and then compares to the files in the test directory.
The Translator now has initial support for commands with just the LET/Assignment statement supported and ibcp_0.1.12-src.zip has been uploaded at Sourceforge IBCP Project along with the binary for the program. Next, implementing the PRINT command...
The plan was to release a whole series of development releases (0.1.12-dev-X) starting with the LET command, but some many changes were made to the code (including the parser correction, parser test updates, token handlers, etc.) from the 0.1.11 release, that this release will be an official Translator development release (0.1.12). This will probably continue for each command added to the Translator. Sub-developmental releases may still be used for the more complex commands.
This release also contains regression test scripts that were used during the latest round of changes. The scripts saved a lot of command line typing due to all the reorganization of the code (token handlers, token status and token mode changes). A Windows batch file equivalent is also included, but does not work near as nice because of the limitations of the Window compare utility. Both scripts delete any current output files, runs the tests and then compares to the files in the test directory.
The Translator now has initial support for commands with just the LET/Assignment statement supported and ibcp_0.1.12-src.zip has been uploaded at Sourceforge IBCP Project along with the binary for the program. Next, implementing the PRINT command...
Translator – LET Command (Implementation)
The SimpleStack class is used for the new command stack, which will hold a CmdItem structure consisting of a token pointer and a code. The code will initially be set from the token's index, but may be changed to other associated codes as commands are processed by the Translator. Since the hold and done stacks are also simple stacks, these were changed from the List class to the SimpleStack class. This only required minor changes to push and pop calls for these stacks.
As previously hinted, the Translator status enumeration needed to be moved outside the Translator class before the TableEntry structure. The enumeration was renamed TokenStatus as that seemed appropriate for the return value of the Translator's add token function. Each of the enumeration values were also renamed except for the BUG statuses, which were left alone.
A previously mentioned, the mode must be changed from Command to Assignment upon receiving the LET command token. Some commands will need to change the mode from Command to Expression (PRINT, IF, WHILE, etc.). Some commands will need to change the mode from Command to a new End-of-Statement mode, since nothing is expected after command keyword (END IF, DO, LOOP, etc.).
Having an every growing switch statement of the command code is not efficient. Therefore, a next token mode value was added to the TableEntry structure. When a command token is received, if the mode is currently Command, then the mode will be changed to the command's table entry next token mode value. The command will be pushed onto the command stack along with it's current code. No further action needs to be taken until the rest of the statement is processed.
Since there will be mode values in the TableEntry structure, the Translator mode enumeration was also moved before TableEntry and renamed TokenMode (again an appropriate name). The enumeration values were also renamed to reflect this. For now, if the next token mode is not set for a command (a new Null token mode value), then a “not yet implemented” bug error occurs. If the current mode is not Command, then a new “unexpected command” message occurs.
For the LET command, the next token mode value is Assignment. When an assignment operator is added to the output list, if there is a LET command on the command stack, it is popped and the assignment operator's token's LET sub-code flag is set. For now, since the command stack should be empty, the end-of-line processing only needs to check if the command stack is empty. Later it will need to process commands on the stack.
As previously hinted, the Translator status enumeration needed to be moved outside the Translator class before the TableEntry structure. The enumeration was renamed TokenStatus as that seemed appropriate for the return value of the Translator's add token function. Each of the enumeration values were also renamed except for the BUG statuses, which were left alone.
A previously mentioned, the mode must be changed from Command to Assignment upon receiving the LET command token. Some commands will need to change the mode from Command to Expression (PRINT, IF, WHILE, etc.). Some commands will need to change the mode from Command to a new End-of-Statement mode, since nothing is expected after command keyword (END IF, DO, LOOP, etc.).
Having an every growing switch statement of the command code is not efficient. Therefore, a next token mode value was added to the TableEntry structure. When a command token is received, if the mode is currently Command, then the mode will be changed to the command's table entry next token mode value. The command will be pushed onto the command stack along with it's current code. No further action needs to be taken until the rest of the statement is processed.
Since there will be mode values in the TableEntry structure, the Translator mode enumeration was also moved before TableEntry and renamed TokenMode (again an appropriate name). The enumeration values were also renamed to reflect this. For now, if the next token mode is not set for a command (a new Null token mode value), then a “not yet implemented” bug error occurs. If the current mode is not Command, then a new “unexpected command” message occurs.
For the LET command, the next token mode value is Assignment. When an assignment operator is added to the output list, if there is a LET command on the command stack, it is popped and the assignment operator's token's LET sub-code flag is set. For now, since the command stack should be empty, the end-of-line processing only needs to check if the command stack is empty. Later it will need to process commands on the stack.
Saturday, May 29, 2010
Parser – Bug / Test Updates
During the testing of the LET command implementation, some problems were discovered. The first problem was in the Parser's get number token function. The simple single 0 digit caused an “invalid leading zero in number constant” error. This error check was to prevent a constant like 01 from being accepted, but it did allow a leading zero when followed by a decimal point like in 0.1. This fix was to check if the next character is a digit, and if it's not, then it terminates looking for more characters, and then otherwise cause the error.
Due to this error, it was a good idea to re-run all of the parser tests since it has been a while since these were checked. Low and behold, they all failed or miscompared. The print token function used by these tests hadn't been updated for the changes to the code and data type enumerations, so these were updated. Once corrected, there were still miscompares, but these were due to either the decimal code values changing (because of all the new codes that have been added) or because the data type of many operators and internal functions were changed from None to their appropriate data type. Additional 0 constant test inputs were added to the Parser number test inputs.
The regression test scripts were updated to also include the parser tests. The Windows batch file uses the comp command, which has a nice feature that a wild card can be used for both files names and it is able to compare all sets of files with one command (like all 8 translator test output files). However, it also has an irritating feature where after it's done comparing files, it asks if there are more files to compare. No has to be entered to continue. There is not option to prevent this. Since there are two compares (one for the parser files, one for the translator files), no has to be entered after the comparing the parser file before the translator files are compared.
One last problem was discovered that affects both the print token and print small token functions. Numeric constants were not being output as intended. The code was outputting an integer using the C %d format specifier and doubles using the %g specifier. The problem is that if the constant 1.0 (a double) was entered, it would be output as 1 making it impossible to know if it was an integer or a double. The raw strings entered for the numbers were suppose to be output - the reason these strings were saved in the first place, to preserve the original string for later output by the Recreator.
Due to this error, it was a good idea to re-run all of the parser tests since it has been a while since these were checked. Low and behold, they all failed or miscompared. The print token function used by these tests hadn't been updated for the changes to the code and data type enumerations, so these were updated. Once corrected, there were still miscompares, but these were due to either the decimal code values changing (because of all the new codes that have been added) or because the data type of many operators and internal functions were changed from None to their appropriate data type. Additional 0 constant test inputs were added to the Parser number test inputs.
The regression test scripts were updated to also include the parser tests. The Windows batch file uses the comp command, which has a nice feature that a wild card can be used for both files names and it is able to compare all sets of files with one command (like all 8 translator test output files). However, it also has an irritating feature where after it's done comparing files, it asks if there are more files to compare. No has to be entered to continue. There is not option to prevent this. Since there are two compares (one for the parser files, one for the translator files), no has to be entered after the comparing the parser file before the translator files are compared.
One last problem was discovered that affects both the print token and print small token functions. Numeric constants were not being output as intended. The code was outputting an integer using the C %d format specifier and doubles using the %g specifier. The problem is that if the constant 1.0 (a double) was entered, it would be output as 1 making it impossible to know if it was an integer or a double. The raw strings entered for the numbers were suppose to be output - the reason these strings were saved in the first place, to preserve the original string for later output by the Recreator.
Translator – Token Sub-Codes (Implementation)
The sub-code flag was implemented, which consisted of adding the sub-code memory to the Token class, adding the sub-code flag value definitions and modifying the print token test routine to output the flags.
The setting of the parentheses sub-code flag was handled in the do pending parentheses function, which checks if unnecessary parentheses were entered and appended a dummy parentheses token. The code was changed to set the parentheses sub-code flag of the last token appended to the output. Two issues were discovered.
The first issue found was if two sets of unnecessary parentheses are entered, for example, A=((B)), then the parentheses sub-code flag can only be set once. Upon reproducing the original source, the Recreator will not know that there were two sets of unnecessary parentheses. So for this case, a dummy parentheses token will still be appending for each additional set of unnecessary parentheses entered.
Curiously, with this change, if three (or and odd number of) unnecessary sets of parentheses are entered, for the third (fifth, etc.) set, the parentheses sub-code flag gets set in the second (fourth, etc.) set's dummy parentheses token. Something the Recreator will need to handle.
The second issue found was if the last token appended was a hidden conversion code, the conversion code's token parentheses sub-code flag gets set. It is anticipated that this would cause a problem for the Recreator. The Recreator should be able to safely ignore the conversion codes, but if it needs to look for sub-code flags, these can't be ignored. To avoid this, the code was modified that if token's table entry has the new Hidden flag set, then the item previous to the conversion code has its parentheses sub-code set.
The setting of the comma sub-code flag was handled in the new equal token handler function. When an equal token is received, if the mode was a multiple comma assignment, then the comma sub-code flag is set when the token's code is set to the assign list operator. The comma sub-code flag will not be set of the mode was a multiple equal assignment.
The setting of the parentheses sub-code flag was handled in the do pending parentheses function, which checks if unnecessary parentheses were entered and appended a dummy parentheses token. The code was changed to set the parentheses sub-code flag of the last token appended to the output. Two issues were discovered.
The first issue found was if two sets of unnecessary parentheses are entered, for example, A=((B)), then the parentheses sub-code flag can only be set once. Upon reproducing the original source, the Recreator will not know that there were two sets of unnecessary parentheses. So for this case, a dummy parentheses token will still be appending for each additional set of unnecessary parentheses entered.
Curiously, with this change, if three (or and odd number of) unnecessary sets of parentheses are entered, for the third (fifth, etc.) set, the parentheses sub-code flag gets set in the second (fourth, etc.) set's dummy parentheses token. Something the Recreator will need to handle.
The second issue found was if the last token appended was a hidden conversion code, the conversion code's token parentheses sub-code flag gets set. It is anticipated that this would cause a problem for the Recreator. The Recreator should be able to safely ignore the conversion codes, but if it needs to look for sub-code flags, these can't be ignored. To avoid this, the code was modified that if token's table entry has the new Hidden flag set, then the item previous to the conversion code has its parentheses sub-code set.
The setting of the comma sub-code flag was handled in the new equal token handler function. When an equal token is received, if the mode was a multiple comma assignment, then the comma sub-code flag is set when the token's code is set to the assign list operator. The comma sub-code flag will not be set of the mode was a multiple equal assignment.
Translator – Token Handlers (Implementation)
A TokenHandler typedef was needed to define the pointer to token handler function. I was not able to define the function pointer directly on the variables. This always proves difficult with more complex types, especially involving pointers. Fortunately, using typedef simplifies the issue. Here is the definition that was inserted before the TableEntry class:
The code that handles operators (which was after the switch statement for the special codes) was also put into a handler function. This greatly simplified the code at the end of the add token function. For processing the token, the temporary token handler function pointer is set to the code's table entry value. If the value is not set (is NULL), then the temporary pointer is set to the default operand token handler function. The function is called using the temporary pointer and it's status return value is immediately returned.
The program was compiled several times during the making of the changes. I got tired of running of the test cases (to output is redirected to a file in the base directory) and comparing to the official test output files (in the test directory), so I wrote a MSYS (bash) script to do it automatically and check all the test cases. An equivalent Windows batch file was also written, but does not work near as nice. Both will be included in the next release.
class Translator; // forward reference to Translator classThe TokenHandler type definition is then used for defining an token handler function pointers. Each of the token handlers were created using the existing code in the switch statement, where the code was modified to add the Translator pointer in front of all the Translator variables and the Translator scope (Translator::) was added to the Translator enumeration values.
typedef TokenStatus (*TokenHandler)(Translator &p, Token *&token);
The code that handles operators (which was after the switch statement for the special codes) was also put into a handler function. This greatly simplified the code at the end of the add token function. For processing the token, the temporary token handler function pointer is set to the code's table entry value. If the value is not set (is NULL), then the temporary pointer is set to the default operand token handler function. The function is called using the temporary pointer and it's status return value is immediately returned.
The program was compiled several times during the making of the changes. I got tired of running of the test cases (to output is redirected to a file in the base directory) and comparing to the official test output files (in the test directory), so I wrote a MSYS (bash) script to do it automatically and check all the test cases. An equivalent Windows batch file was also written, but does not work near as nice. Both will be included in the next release.
Friday, May 28, 2010
Translator – Token Handlers
The Translator's add token function is becoming quite large and at the current rate will become much larger as the different commands are implemented. It's never a good idea to have giant functions. Therefore, this routine needs to be broken up into separate functions. The code that contains the special token code handling (for equal, comma, closing parentheses, and end-of-line) will be separate functions. These functions will be called token handlers. Most of the commands will also have token handlers.
Having a switch statement on the token code where each case calls a token handler function is not the most efficient implementation. The concern is not execution time, but the amount of code lines required, in other words, a large switch statement. A better implementation is to store a function pointer in each table entry where a code requires a token handler. The Translator will check if there is a function pointer and then call the function to process the token.
And here is where it gets complicated. Pointers to member functions are not allowed. So the token handlers can't be Translator member function. But they need access to all of the Translator data members. The solution was to make them C++ friend functions of the Translator class with an reference argument to the Translator instance (a Translator function will pass *this).
The token handlers functions will also require a reference to the current token pointer (a reference so that it can be changed for an error) and will return the status of the operation. The Status enumeration needs to be moved out of the Translator because in order to define the return value for the token handler function pointer, it needs to be defined before TableEntry (and Table). But Table needs to be defined before TableEntry. Catch 22.
A forward reference for the Translator class can be added before TableEntry (needed for the Translator reference argument, which is simply class followed by the class name and a semicolon), but not for enumerations inside the Translator. Therefore the status enumeration will be moved out of Translator and renamed to TokenStatus (appropriate for a token handler, and the word Token is shorter than Translator).
Having a switch statement on the token code where each case calls a token handler function is not the most efficient implementation. The concern is not execution time, but the amount of code lines required, in other words, a large switch statement. A better implementation is to store a function pointer in each table entry where a code requires a token handler. The Translator will check if there is a function pointer and then call the function to process the token.
And here is where it gets complicated. Pointers to member functions are not allowed. So the token handlers can't be Translator member function. But they need access to all of the Translator data members. The solution was to make them C++ friend functions of the Translator class with an reference argument to the Translator instance (a Translator function will pass *this).
The token handlers functions will also require a reference to the current token pointer (a reference so that it can be changed for an error) and will return the status of the operation. The Status enumeration needs to be moved out of the Translator because in order to define the return value for the token handler function pointer, it needs to be defined before TableEntry (and Table). But Table needs to be defined before TableEntry. Catch 22.
A forward reference for the Translator class can be added before TableEntry (needed for the Translator reference argument, which is simply class followed by the class name and a semicolon), but not for enumerations inside the Translator. Therefore the status enumeration will be moved out of Translator and renamed to TokenStatus (appropriate for a token handler, and the word Token is shorter than Translator).
Thursday, May 27, 2010
Translator – LET Command
There is nothing special to say about the LET command – the format of assignment statements, less the optional LET keyword, has already been defined and implemented. When the Translator receives the LET command token, it will be pushed on the new command stack. The mode will be Command when the LET is received and, as the Translator is currently implemented, needs to left set to Command mode for the assignment statement to be processed as currently implemented.
When the assignment token is added to the output list, it will first check if there is a LET command token on top of the command stack, and if there is, then the LET sub-code flag will be set in the assignment token, the LET command will be popped from the command stack, and the LET token will be deleted.
There is a problem. The mode must be set to Command for the LET token to be accepted. The mode must be set to Command for the assignment statement. Some detection is necessary to prevent double LET keywords. To solve this problem, a new mode is needed, an Assignment mode. For most of the Translator, both Command and Assignment mode will be equivalent except for the processing of command tokens.
When the assignment token is added to the output list, it will first check if there is a LET command token on top of the command stack, and if there is, then the LET sub-code flag will be set in the assignment token, the LET command will be popped from the command stack, and the LET token will be deleted.
There is a problem. The mode must be set to Command for the LET token to be accepted. The mode must be set to Command for the assignment statement. Some detection is necessary to prevent double LET keywords. To solve this problem, a new mode is needed, an Assignment mode. For most of the Translator, both Command and Assignment mode will be equivalent except for the processing of command tokens.
Subscribe to:
Posts (Atom)