There are two codes for remarks, the command code (for the REM command) and the operator code (for the "'" operator). The token contains the string of the remark. Generally, the keyword (REM or "'") along with the remark string is added to the output string. However, there were two issues to be handled.
Unlike all the other commands, no space is required after the REM command when followed by a letter, so a statement like "REMARK A Comment" is valid. The issue is for a REM statement entered as "remark a comment" in all lower case. When recreated, the result would have been the "REMark a comment" statement. So, if the first character of the remark string is lower case, the REM keyword is converted to lower case. This still won't work if something like "Remark A Comment" is entered.
The second issue involves the remark operator. A space is needed before the "'" operator if the command is not at the beginning of the line to provide some separation between the previous statement. To determine if the remark is at the beginning of the line, an is empty access function was added to the recreator class that returns whether the output string is empty (which implies this the beginning of the line if nothing was added yet).
A single rem recreate function was added to handle both remark codes and a pointer to this function was added to the remark code table entries. To test the lower case check, an all lower case remark statement was added to translator test #15 (REM tests). The expected recreated outputs for this test were updated and are recreated correctly.
[commit 03c73c3de7]
Showing posts with label Rem Command. Show all posts
Showing posts with label Rem Command. Show all posts
Saturday, November 23, 2013
Saturday, October 5, 2013
Operand Text / Remark Operands
Next up was to retrieve the text for the operand (comment string) of the remark command or operator from the dictionary using the index that is stored in the program code. The recreator also needs to do this, therefore, a mechanism was implemented to allow for the getting the text for the debug output, which will later be used for the recreator.
A new operand text function pointer was added to the table entry. This function is only needed for codes that have an operand, and an access function was added for this pointer. These functions have arguments for the pointer to the current program unit (to access the dictionaries) and the operand. The rem operand text function was implemented to get the text of the remark for an operand (an index) by passing it to the new dictionary string access function (gets the text from the key list member).
The program line text routine was renamed to debug text since it is only used for generating text for test output. Also, since this routine needs access the program unit for the dictionaries to look index values, it was moved to the program unit class. Since the program line class was only created to add the text routine to a QVector, this class was removed and QVector<ProgramWord> is now used.
The operand debug text (renamed) routine outputs the index of the operand along with the value of the operand, and needed to output the text of the operand. To get the pointer to the operand text function, it would need to be passed the code for the operand and would need the table instance. Instead, this functionality is handled by the debug text routine and the resulting text is passed to this routine to add the word index and operand index values.
The debug text routine also needs the table instance, so a table reference was added to the program unit class. This routine will also be called when encoded debug text is integrated into the program view of the GUI. The test routine called this routine with a program line (now a program word vector), however, when used for the program view, a program word vector would need to be extracted from the program unit code vector. This would require a vector to be copied from part of the total program code vector. To prevent this copy, the debug text was modified to take a program word pointer and a count of the number of words in the line.
Some minor cleanup of the table entries were performed on a separate commit (placed function pointers on their own lines, corrected the camel casing of the rem encode routine, and fixed a comment). The expected results file of encoder test #1 was updated for the remark comments that are now in the output.
[commit f5fe1efa4f] [commit bf9264bafb]
A new operand text function pointer was added to the table entry. This function is only needed for codes that have an operand, and an access function was added for this pointer. These functions have arguments for the pointer to the current program unit (to access the dictionaries) and the operand. The rem operand text function was implemented to get the text of the remark for an operand (an index) by passing it to the new dictionary string access function (gets the text from the key list member).
The program line text routine was renamed to debug text since it is only used for generating text for test output. Also, since this routine needs access the program unit for the dictionaries to look index values, it was moved to the program unit class. Since the program line class was only created to add the text routine to a QVector, this class was removed and QVector<ProgramWord> is now used.
The operand debug text (renamed) routine outputs the index of the operand along with the value of the operand, and needed to output the text of the operand. To get the pointer to the operand text function, it would need to be passed the code for the operand and would need the table instance. Instead, this functionality is handled by the debug text routine and the resulting text is passed to this routine to add the word index and operand index values.
The debug text routine also needs the table instance, so a table reference was added to the program unit class. This routine will also be called when encoded debug text is integrated into the program view of the GUI. The test routine called this routine with a program line (now a program word vector), however, when used for the program view, a program word vector would need to be extracted from the program unit code vector. This would require a vector to be copied from part of the total program code vector. To prevent this copy, the debug text was modified to take a program word pointer and a count of the number of words in the line.
Some minor cleanup of the table entries were performed on a separate commit (placed function pointers on their own lines, corrected the camel casing of the rem encode routine, and fixed a comment). The expected results file of encoder test #1 was updated for the remark comments that are now in the output.
[commit f5fe1efa4f] [commit bf9264bafb]
Thursday, October 3, 2013
Remark Dictionary
The remark dictionary is the first dictionary implemented. This is the simplest dictionary as it only needs to store the string of a comment, so the base dictionary is sufficient for this dictionary. The ProgramUnit class was added to contain the remark dictionary, which is defined as a pointer and allocated in the ProgramUnit constructor. A pointer was used since all the other program units will use the same remark dictionary.
Previously, the encoder encode routine just set the operand to zero and was modified to call the encode function for the code with an operand. For now, if the code does not have an encode function, the operand is still set to zero. The encode functions contain arguments for the pointer to the current program unit and the pointer to the token being encoded. Additional arguments will probably be needed, but these are sufficient for now.
The rem encode function was implemented, which simply passes the token to the add function of the remark dictionary and returns the index of the entry to be set as the operand word. The pointer to an encode function was added to the table entry structure and a table access function was added to return this pointer. A pointer to the rem encode function was added to the REM command and REM operator code table entries.
For testing, a temporary program unit was added to the main tester run routine and a pointer to it is passed to the encode input routine, which passed to the encode routine. Additional remarks were added to encoder test #1 including one set of duplicate comments to test that a single entry is stored in the dictionary. Currently only the index value is output as the string (of the remark) is not yet retrieved, which will be the subject of the next change. For now, the current output of this test (though incomplete) was copied for the expected results. Though this makes the test pass, the purpose for changing this file was detect changes as more of the encode routines are implemented.
[commit d2222df1d8]
Previously, the encoder encode routine just set the operand to zero and was modified to call the encode function for the code with an operand. For now, if the code does not have an encode function, the operand is still set to zero. The encode functions contain arguments for the pointer to the current program unit and the pointer to the token being encoded. Additional arguments will probably be needed, but these are sufficient for now.
The rem encode function was implemented, which simply passes the token to the add function of the remark dictionary and returns the index of the entry to be set as the operand word. The pointer to an encode function was added to the table entry structure and a table access function was added to return this pointer. A pointer to the rem encode function was added to the REM command and REM operator code table entries.
For testing, a temporary program unit was added to the main tester run routine and a pointer to it is passed to the encode input routine, which passed to the encode routine. Additional remarks were added to encoder test #1 including one set of duplicate comments to test that a single entry is stored in the dictionary. Currently only the index value is output as the string (of the remark) is not yet retrieved, which will be the subject of the next change. For now, the current output of this test (though incomplete) was copied for the expected results. Though this makes the test pass, the purpose for changing this file was detect changes as more of the encode routines are implemented.
[commit d2222df1d8]
Thursday, September 26, 2013
REM Command Correction
The operands of program instructions will hold indexes to dictionary entries that will contain the text of the instruction (for example, variable names, the original strings of constants, the strings of remarks, etc.). The remark dictionary will be implemented first since it will just hold the strings of the comments. The other dictionaries will required looking up strings to find if a variable or constant already exists.
But first, a problem was discovered with how REM statements are parsed. The REM command should be recognized regardless of what characters follow the command. The parser for the most part did this already except that a space was required after the command. However, the parser should not require the space, consider these examples:
The parser get identifier was modified to first look for a statement starting with the three characters R-E-M and store all the characters after this before scanning for a word (valid identifier characters up to a invalid identifier character). Some minor code simplification was also done in replacing the sequence of setting the token code, type and data type with a call to the existing set token table routine that performed these steps. Two additional statements were added to translator test #15 (remark tests) similar to the two examples above.
[commit c52d65a479]
But first, a problem was discovered with how REM statements are parsed. The REM command should be recognized regardless of what characters follow the command. The parser for the most part did this already except that a space was required after the command. However, the parser should not require the space, consider these examples:
REMARK this should be a valid commentedThe first statement was rejected because it was assumed to be an assignment of the REMARK variable and expected an equals instead of this. The second statement was valid but all the spaces were removed from the comment string.
REM any number of spaces should be allowed
The parser get identifier was modified to first look for a statement starting with the three characters R-E-M and store all the characters after this before scanning for a word (valid identifier characters up to a invalid identifier character). Some minor code simplification was also done in replacing the sequence of setting the token code, type and data type with a call to the existing set token table routine that performed these steps. Two additional statements were added to translator test #15 (remark tests) similar to the two examples above.
[commit c52d65a479]
Thursday, August 22, 2013
New Translator – Remarks
There are two forms of remarks (comments), the REM command and the remark operator (a single quote). The remark operator can be placed anywhere an end-of-statement token can be put including at the beginning of the line where a command is. There are no commands after a remark since all characters are part of the comment up to the end of the line. Remarks were implemented a couple of different ways and while these all worked, these solutions were not very clean (there were multiple checks for the Rem and RemOp tokens in several routines).
The final solution was to implement a new get commands routine, which as the name implies will be able to handle multiple commands on the line separated by colons (which wasn't implemented yet). The new routine contains a loop, which begins by getting a token. If the token is a Rem or RemOp, the loop is exited. Otherwise the token is processed by calling the process command routine (see below). If the terminating token of the command is a RemOp, the loop is exited. Otherwise, for now, the routine returns the terminating token from the command translator routine and the done status.
When the loop is exited in the new get commands routine, which will be due to a Rem or RemOp token, the token is appended to the RPN output list, and the next token is obtained, which should be an end-of-line token unless there is a bug in the parser. The end-of-line token is returned as the terminating token with the done status.
The original get command routine was renamed to the process command routine, which made more sense and this routine won't be public as only the get commands routine will be the only caller. It was modified to receive the first token from the token argument instead of getting the token itself.
The expected results for translator test #15 (Remark tests) needed to be updated due to the change in translation of the PRINT and INPUT commands (the old results were saved). Now all the translator tests pass with the new translator routines.
[commit a3a71526f9]
The final solution was to implement a new get commands routine, which as the name implies will be able to handle multiple commands on the line separated by colons (which wasn't implemented yet). The new routine contains a loop, which begins by getting a token. If the token is a Rem or RemOp, the loop is exited. Otherwise the token is processed by calling the process command routine (see below). If the terminating token of the command is a RemOp, the loop is exited. Otherwise, for now, the routine returns the terminating token from the command translator routine and the done status.
When the loop is exited in the new get commands routine, which will be due to a Rem or RemOp token, the token is appended to the RPN output list, and the next token is obtained, which should be an end-of-line token unless there is a bug in the parser. The end-of-line token is returned as the terminating token with the done status.
The original get command routine was renamed to the process command routine, which made more sense and this routine won't be public as only the get commands routine will be the only caller. It was modified to receive the first token from the token argument instead of getting the token itself.
The expected results for translator test #15 (Remark tests) needed to be updated due to the change in translation of the PRINT and INPUT commands (the old results were saved). Now all the translator tests pass with the new translator routines.
[commit a3a71526f9]
Saturday, April 6, 2013
REM Operator (For BASIC Comments)
Implementing the REM operator (single quote) turned out to be simpler than anticipated. The REM operator type of REM always occurs at the end of a line, therefore, it can be treated as the end of the line except that the REM operator token (with comment string) will be added to the end of the RPN output list.
A new REM operator token handler was implemented, which first checks if the command stack is not empty or if the current token mode is not command (occurs for assignment statements without the LET keyword). Otherwise the REM operator being processed occurred at the beginning of the line and no current command needs to be processed.
To process the current command, the end of line token handler is called since it contains all the needed functionality for processing the command when the end of a line is reached. A new end of line token is created and passed to the end of line token handler. A new token was needed in case the command changes the end of line token into another token to add to the RPN output list (as the INPUT command does). If the command does not use this token, the end of line token handler will delete it. If an error is returned, then the end of line token is deleted and the error is returned. Otherwise, the REM operator token is appended to the end of the RPN output list.
Since the REM operator token acts as the end of the line, the table entry for the REM operator code entry was changed to include the end expression and end statement flags. The pointer to the new REM operator token handler was also added. Several new REM tests were added to translator test #15 for the REM operator on various types of commands. Some error tests were also added.
[commit da69b3ba07]
A new REM operator token handler was implemented, which first checks if the command stack is not empty or if the current token mode is not command (occurs for assignment statements without the LET keyword). Otherwise the REM operator being processed occurred at the beginning of the line and no current command needs to be processed.
To process the current command, the end of line token handler is called since it contains all the needed functionality for processing the command when the end of a line is reached. A new end of line token is created and passed to the end of line token handler. A new token was needed in case the command changes the end of line token into another token to add to the RPN output list (as the INPUT command does). If the command does not use this token, the end of line token handler will delete it. If an error is returned, then the end of line token is deleted and the error is returned. Otherwise, the REM operator token is appended to the end of the RPN output list.
Since the REM operator token acts as the end of the line, the table entry for the REM operator code entry was changed to include the end expression and end statement flags. The pointer to the new REM operator token handler was also added. Several new REM tests were added to translator test #15 for the REM operator on various types of commands. Some error tests were also added.
[commit da69b3ba07]
REM Command (For BASIC Comments)
Before continuing with highlighting errors in the edit box, I noticed (when implementing the routine that converts the token contents to text for display in the program view) that the Remark token type was not actually being used. The parser is handling two types of remarks, the REM command and the single quote comment method, which is treated as an operator since it can appear at the end of any line and does not need to be preceded by the colon statement separator.
However, these two tokens (REM command and REM operator) were not being handled in the translator. The REM command token was returning a "Not Yet Implemented" error message. For the REM command token to be handled and not return this error message, the REM code in the table needed to be assigned a token mode. It turned out that the specific token mode assigned was not important as long as it wasn't the default NULL token type, which triggers the error message, because the REM command will always be the last command on a line.
To process the REM command token, a new REM command handler was implemented that simply adds the REM command token (which contains the actual comment text in the string of the token) to the RPN output list. A new translator test (#15) was added for various REM command tests. The REM operator token is a little trickier to handle, which will be implemented next.
[commit 258f3a9df0]
However, these two tokens (REM command and REM operator) were not being handled in the translator. The REM command token was returning a "Not Yet Implemented" error message. For the REM command token to be handled and not return this error message, the REM code in the table needed to be assigned a token mode. It turned out that the specific token mode assigned was not important as long as it wasn't the default NULL token type, which triggers the error message, because the REM command will always be the last command on a line.
To process the REM command token, a new REM command handler was implemented that simply adds the REM command token (which contains the actual comment text in the string of the token) to the RPN output list. A new translator test (#15) was added for various REM command tests. The REM operator token is a little trickier to handle, which will be implemented next.
[commit 258f3a9df0]
Subscribe to:
Posts (Atom)