At the moment, the token codes are assigned to the constant and identifiers with no parentheses token types after the translation is complete. This was initially the first step of encoding, but was moved into the translator. Before moving this code assignment earlier in the translation, a simplified routine was needed where it is given a token with a data type and for a base code, find a code for that data type (which will be either the base code or one of its associated codes).
The table find code routine was being called to perform this action, but this routine does a lot more including converting a constant to the expected data type (so no hidden conversion code needs to be added), finding a conversion code if there was no associated code with the desired data type, and returning the expected data type when data type cannot be converted (an error).
The find code routine contained the part for finding an associated code, so this part was moved into the new set token code routine. This new routine just looks up an associated code if the expected data type of base case does not match a specified data type for a given operand of the code. If the base code or an associated code matches, then the token is set to the code and its type and data type is set to that of the code. Otherwise it returns false.
A secondary simplified set token code routine was added for setting the code of a token for a base code for the data type already in the token. The base codes that are used for this routine have only one operand, which currently only includes the Const, Var and VarRef codes. This secondary routine just calls the main set token code with the data type of the token and for the first (and only) operand of the base code.
[commit 32eec6f3c0]
Saturday, September 21, 2013
Saturday, September 14, 2013
Translator Token Code Transition (Begin)
The translator will be transitioned to assign token codes to token types without a code as soon as possible. This includes the constant, identifiers with and without parentheses, and defined functions with and without parentheses token types. The handling of the defined functions types will be delayed if possible until define functions are implemented (much later).
Development of this transition will be done on the tokenCode0.5 sub-branch in case the changes do not work out and need to be abandoned, otherwise when complete, the main branch0.5 branch will be fast-forwarded to this sub-branch (which can then be deleted).
The first change was to give all tokens a code when they are created by the parser. Since the mentioned token types do not have a code, these will be assigned to the Invalid code value. This was accomplished by initializing the code to Invalid in the token constructor. As currently implemented, when the parser finds a command, operator, or internal function, the token is then assigned a valid code.
Since all tokens now have a code (some with the Invalid code), the token code can now be used to determine if the token has a table entry (it doesn't if the code is invalid). The has table entry access function is now longer needed and was removed along with the static table array indexed by token type that was used to determine if the token had a table entry. A new has valid code access function was added in its place. Users of the has table entry access function were updated to use the token code (see the commit for the specifics).
[commit 57d65135de]
Development of this transition will be done on the tokenCode0.5 sub-branch in case the changes do not work out and need to be abandoned, otherwise when complete, the main branch0.5 branch will be fast-forwarded to this sub-branch (which can then be deleted).
The first change was to give all tokens a code when they are created by the parser. Since the mentioned token types do not have a code, these will be assigned to the Invalid code value. This was accomplished by initializing the code to Invalid in the token constructor. As currently implemented, when the parser finds a command, operator, or internal function, the token is then assigned a valid code.
Since all tokens now have a code (some with the Invalid code), the token code can now be used to determine if the token has a table entry (it doesn't if the code is invalid). The has table entry access function is now longer needed and was removed along with the static table array indexed by token type that was used to determine if the token had a table entry. A new has valid code access function was added in its place. Users of the has table entry access function were updated to use the token code (see the commit for the specifics).
[commit 57d65135de]
Friday, September 13, 2013
Encoder/Translator – Design Considerations
In preparing for implementation of the next Encoder step of actually encoding the RPN list of tokens into program code, I realized that were some design issues. The first minor issue was that assigning codes and assigning program code word indexes with word counting can't be in the same loop because later for arrays, integer conversion codes may need to be inserted after subscripts, which will trip up the indexes already sett and the words counted. These indexes and word count could be adjusted as items are inserted, but it would more efficient to have two loops.
In further consideration of the design, I realized that the these first steps should really be in the translator and not the encoder. The code for these steps were moved to the translator (executed after successfully getting the commands for the line). A new test mode option was needed so that this call would not be made when just testing the translator so the expected translator test results would not change.
However, I realized that the translator could assign the codes before waiting until the end of the translation. The codes could be assigned when the tokens are first processed, specifically in the get operand routine. Also, the translator cannot work in a vacuum, meaning it will need access to the existing program, specifically the various dictionaries so that the types of tokens can be determined (variables, arrays, or functions).
Therefore, the translator will be changed to this new scheme of assigned codes to tokens in the get operand routine, but over the next several commits, with the exception that the dictionaries are not implemented yet. The first change made was to move the first two encoder steps into the translator and two separate loops, though this is a temporary step (and the encoder class is now empty, but this is also temporary).
The second change in this transition was to add a temporary check to determine if an identifier with parentheses is an array or function. Eventually this will be accomplished with the array and function dictionaries. To make this determination simple, if the identifier starts with the 'F' letter, then it is considered a function, otherwise it is considered an array. This affected the expected results because array subscripts are checked for integers with integer conversions added as needed.
[commit f9f0cd3de7] [commit 9111d8e0f8]
In further consideration of the design, I realized that the these first steps should really be in the translator and not the encoder. The code for these steps were moved to the translator (executed after successfully getting the commands for the line). A new test mode option was needed so that this call would not be made when just testing the translator so the expected translator test results would not change.
However, I realized that the translator could assign the codes before waiting until the end of the translation. The codes could be assigned when the tokens are first processed, specifically in the get operand routine. Also, the translator cannot work in a vacuum, meaning it will need access to the existing program, specifically the various dictionaries so that the types of tokens can be determined (variables, arrays, or functions).
Therefore, the translator will be changed to this new scheme of assigned codes to tokens in the get operand routine, but over the next several commits, with the exception that the dictionaries are not implemented yet. The first change made was to move the first two encoder steps into the translator and two separate loops, though this is a temporary step (and the encoder class is now empty, but this is also temporary).
The second change in this transition was to add a temporary check to determine if an identifier with parentheses is an array or function. Eventually this will be accomplished with the array and function dictionaries. To make this determination simple, if the identifier starts with the 'F' letter, then it is considered a function, otherwise it is considered an array. This affected the expected results because array subscripts are checked for integers with integer conversions added as needed.
[commit f9f0cd3de7] [commit 9111d8e0f8]
Saturday, September 7, 2013
Encoder – First Phase Complete (Tagged)
The initial implementation of the encode is complete, which includes preparing the tokens for encoding. Version v.0.5.0 has been tagged. Work will now begin in encoding the tokens into the internal program code format.
[commit 6ed93f833c]
[commit 6ed93f833c]
Encoder Test Mode – Blank Lines
When the new line was added to encoder test #1, a blank line was added before the new line, but the line was ignored. The test run routine was modified to allow a blank line only when testing the encoder. The expected results were updated accordingly.
[commit c84fbfa08b]
[commit c84fbfa08b]
Encoder – Assigning Positional Indexes
The next step in preparing the tokens in the RPN list for encoding is assigning a position index to each token. This is a position, or offset, that the token will have in the encoded program code within the line. This index will be used later for calculating offsets of a single line statements like from the IF token to the ELSE, END IF or last token on a line.
For now however, this index is not needed, but after assigning the indexes to all the tokens, the final count or size of the encoded program code for the line will be known. This value will be used for allocating a program word array that will be filled in during the next step of encoding.
It was more efficient to assign the position index after assigning codes to tokens instead of having a separate routine with another loop, the assign codes routine was renamed to the prepare tokens routine. A count variable was added to the routine, which is initialized to zero. After the token type switch, the index of the token is set to the count and the count is incremented. If the token will have an an operand (the token code has the has operand flag), the count is incremented again for the operand word.
The prepare tokens routine was also changed to return the size required for the encoder program line instead of a success/fail boolean status. For errors, a -1 value is returned. The calling encode routine was updated accordingly for the new return value.
The RPN list, item and token text routines were modified to optionally output the index of each token. The token text routine was also modified to output an index for the operand word, to make sure that the sub-codes are output after the code word (not after the operand word), and to treat the comment of remark as the operand word (when index output is selected). Another test line with multiple statements was added to encoder test #1 and the expected results were updated for the indexes now output on each token.
[commit 9e0e442171]
For now however, this index is not needed, but after assigning the indexes to all the tokens, the final count or size of the encoded program code for the line will be known. This value will be used for allocating a program word array that will be filled in during the next step of encoding.
It was more efficient to assign the position index after assigning codes to tokens instead of having a separate routine with another loop, the assign codes routine was renamed to the prepare tokens routine. A count variable was added to the routine, which is initialized to zero. After the token type switch, the index of the token is set to the count and the count is incremented. If the token will have an an operand (the token code has the has operand flag), the count is incremented again for the operand word.
The prepare tokens routine was also changed to return the size required for the encoder program line instead of a success/fail boolean status. For errors, a -1 value is returned. The calling encode routine was updated accordingly for the new return value.
The RPN list, item and token text routines were modified to optionally output the index of each token. The token text routine was also modified to output an index for the operand word, to make sure that the sub-codes are output after the code word (not after the operand word), and to treat the comment of remark as the operand word (when index output is selected). Another test line with multiple statements was added to encoder test #1 and the expected results were updated for the indexes now output on each token.
[commit 9e0e442171]
Encoder – Variable Reference Fix
In testing the next step, a small problem was discovered with variable references where the table entry for the main variable reference had the constant associated code array. This was corrected and the expected results for encoder test #1 was updated.
[commit 88913ed121]
[commit 88913ed121]
Initial Encoder – Assigning Codes
The new Encoder class was implemented initially with just the first step where tokens without codes are assigned codes. For now, only variables (identifier with no parentheses token type) and constants are handled. The other token types will be implemented once the recreator and run-time modules are implemented for all the initial set of commands (LET, PRINT, INPUT and REM). The first phase of the Encoder class implementation (steps 1 and 2 described on August 30) only prepare the RPN output list from the translator (will be referred to as the RPN input list for the encoder). The second phase (step 3) involves the generation of the program code.
The initial Encoder class contains two member variables, a reference to the table instance and a pointer to the RPN input list. The class includes a single public encode routine and a private assign codes routine that the encode routine calls after saving the pointer to input list. For now, both routines simply return a success or fail status as a boolean value. The assign codes routine loops through the input list and does a switch on the token type.
For a constant token type, and the appropriate code is found for the data type of the constant from the new base code Const (with associated codes ConstInt and ConstStr). An identifier with no parentheses token type is assumed to be a variable (until functions are implemented later). If the reference flag is set, then the appropriate code is found for the data type from the new base code VarRef (with associated codes VarRefInt and VarRefStr). For non-reference tokens, the appropriate code is found for from the new base code Var (with associated codes VarInt and VarStr).
For the command, operator, and internal function (with or without parentheses) token types, no action is preformed since these token types already have a code. For the other token types (identifiers with parentheses, and defined functions with and without parentheses), the token is set as an error in RPN input list with a "not yet implemented" error, the input list is cleared and false is returned.
A pass through find code routine was added to the Table class that takes a single token and a base code. The token is set to the base code and its type is set to the type for the base code. This type replaces the token type present (constant or identifier with no parentheses). The full find code routine is called with the token passed as both the main token and operand token, since the token is the token to be modified (for an associated code as needed) and contains the information (data type) needed to set the code.
Table entries for the new codes were added, each given the internal function with no parentheses token type (same as for the hidden convert codes). Each of these entries were also given the new has operand flag. This flag will be used to determine if the code has a second operand program word. The token text routine was modified for internal function token types, where if the has operand flag is set, then the string of the token is output like a separate token representing a separate program word.
An initial encoder test file was created containing a assignment statement for a variable of each data type assigned to a constant along with a PRINT statement for each variable. This contains each of the nine tokens that need a code assigned (constant, variable, and variable reference for each data type). Encoder command line options were added to the Tester class with an encode input routine that first translates the line and if no error, encodes the line. For now, the tokens of the RPN list are output like with translator testing. The test scripts and batch file were updated to handle the encoder tests.
[commit 94ca2692d0]
The initial Encoder class contains two member variables, a reference to the table instance and a pointer to the RPN input list. The class includes a single public encode routine and a private assign codes routine that the encode routine calls after saving the pointer to input list. For now, both routines simply return a success or fail status as a boolean value. The assign codes routine loops through the input list and does a switch on the token type.
For a constant token type, and the appropriate code is found for the data type of the constant from the new base code Const (with associated codes ConstInt and ConstStr). An identifier with no parentheses token type is assumed to be a variable (until functions are implemented later). If the reference flag is set, then the appropriate code is found for the data type from the new base code VarRef (with associated codes VarRefInt and VarRefStr). For non-reference tokens, the appropriate code is found for from the new base code Var (with associated codes VarInt and VarStr).
For the command, operator, and internal function (with or without parentheses) token types, no action is preformed since these token types already have a code. For the other token types (identifiers with parentheses, and defined functions with and without parentheses), the token is set as an error in RPN input list with a "not yet implemented" error, the input list is cleared and false is returned.
A pass through find code routine was added to the Table class that takes a single token and a base code. The token is set to the base code and its type is set to the type for the base code. This type replaces the token type present (constant or identifier with no parentheses). The full find code routine is called with the token passed as both the main token and operand token, since the token is the token to be modified (for an associated code as needed) and contains the information (data type) needed to set the code.
Table entries for the new codes were added, each given the internal function with no parentheses token type (same as for the hidden convert codes). Each of these entries were also given the new has operand flag. This flag will be used to determine if the code has a second operand program word. The token text routine was modified for internal function token types, where if the has operand flag is set, then the string of the token is output like a separate token representing a separate program word.
An initial encoder test file was created containing a assignment statement for a variable of each data type assigned to a constant along with a PRINT statement for each variable. This contains each of the nine tokens that need a code assigned (constant, variable, and variable reference for each data type). Encoder command line options were added to the Tester class with an encode input routine that first translates the line and if no error, encodes the line. For now, the tokens of the RPN list are output like with translator testing. The test scripts and batch file were updated to handle the encoder tests.
[commit 94ca2692d0]
Pre-Encoder Issues – More Refactoring
As the initial encoder class was being implemented, the need for some more minor refactoring was noticed. The first was the token mode enumeration that was needed by the old translator routines was not removed. The second was the member variable and access function for the double value of a constant token was renamed from valueDbl to just value for consistency. Since double is the default data type, generally the 'Dbl' part is not included (just 'Int' and 'Str') in the name.
[commit 0ec09e02e4] [commit 98e22bda3f]
[commit 0ec09e02e4] [commit 98e22bda3f]
Friday, September 6, 2013
Pre-Encoder Issues – Blank Lines
The final issue is the ability to handle (allow) blanks lines. When using test mode, lines that are blank or begin with a '#' character are ignored and never passed to the translator. However, blank lines are allowed in the GUI and the translator returned an "expected command" error for blank lines.
It was necessary to prevent these errors for the GUI and it is desirable for the encoder test mode to allow blank lines since the end result of the encoder will be to insert actual code into the program and blank lines need to be allowed.
A simple change was made to the get commands routine after getting the first token of a command. If this token is an end-of-line, and only if the RPN output list is empty, the done status is returned. If the output list is not empty, then there was a preceding colon statement separator and an end-of-line is not allowed (a command is expected).
The text output routine for the RPN list class used by the test code already allowed for a blank line (an empty output list), as the for loop that gets the text for each RPN item in the list, simply terminates and an empty string is returned.
This change currently can't be tested using the command line test modes since blank lines are ignored with test input files, and a blank line entered in input entry test mode causes the application to exit. However, the proper handling of blank lines can be seen using the GUI.
[commit 637eb58ea5]
It was necessary to prevent these errors for the GUI and it is desirable for the encoder test mode to allow blank lines since the end result of the encoder will be to insert actual code into the program and blank lines need to be allowed.
A simple change was made to the get commands routine after getting the first token of a command. If this token is an end-of-line, and only if the RPN output list is empty, the done status is returned. If the output list is not empty, then there was a preceding colon statement separator and an end-of-line is not allowed (a command is expected).
The text output routine for the RPN list class used by the test code already allowed for a blank line (an empty output list), as the for loop that gets the text for each RPN item in the list, simply terminates and an empty string is returned.
This change currently can't be tested using the command line test modes since blank lines are ignored with test input files, and a blank line entered in input entry test mode causes the application to exit. However, the proper handling of blank lines can be seen using the GUI.
[commit 637eb58ea5]
Thursday, September 5, 2013
Pre-Encoder Issues – Refactoring
Some minor refactoring (in this case, a fancy term for renaming) was performed. When adding the new index member to the RPN item class, I decided that using the term operands for the items that are attached to an item was not appropriate. Operands really refer to operators. For arrays, attached items are subscripts, and for functions, are arguments. And when implemented, for single line statements like IF, these will simply be attached items, Therefore, the operand term was changed to attached.
Another thing that was bothersome was how the number of variables were named. When programming in C, one convention was to simply prefix the variable with an 'n' character, for example nitems. But when converting to camel casing, this becomes nItems and m_nItems for a class member. This looks kind of ugly (an opinion). Looking at how Qt does naming for inspiration, the convention itemCount is used. Therefore, all uses of 'n' were replaced with the Count convention.
In the RPN item class, there were a number of access functions for the number of operands and the operands themselves (now called attached) that were not actually being used anywhere. So these were removed. (Lesson: Don't added functions until actually needed.)
Finally, table functions and members related to associated codes used the abbreviations assoc and assoc2 (for the second operand associated codes). These were changed to associated and secondAssociated to make the code a little more readable.
[commit e161cf87e0]
Another thing that was bothersome was how the number of variables were named. When programming in C, one convention was to simply prefix the variable with an 'n' character, for example nitems. But when converting to camel casing, this becomes nItems and m_nItems for a class member. This looks kind of ugly (an opinion). Looking at how Qt does naming for inspiration, the convention itemCount is used. Therefore, all uses of 'n' were replaced with the Count convention.
In the RPN item class, there were a number of access functions for the number of operands and the operands themselves (now called attached) that were not actually being used anywhere. So these were removed. (Lesson: Don't added functions until actually needed.)
Finally, table functions and members related to associated codes used the abbreviations assoc and assoc2 (for the second operand associated codes). These were changed to associated and secondAssociated to make the code a little more readable.
[commit e161cf87e0]
Tuesday, September 3, 2013
Pre-Encoder Issues – Output List Indexes
Tokens that are arguments or sub-scripts are attached to identifiers with parentheses and define functions with parentheses tokens. These attached tokens will be used by the Encoder. For tokens that are determined to be an array, the subscript are checked with hidden integer convert codes added for double subscripts and errors reported for string subscripts. For tokens that are functions calls, the arguments are checked with conversion codes added or errors reported as needed.
Later, this mechanism will also be used to attach tokens to certain command tokens. For instance, in a single line IF statement, the ELSE, END IF or last token on the line will be attached to the IF command token. The Encoder will use this attached token to calculate the offset from the IF command to the token, so that during execution, the number of words to skip will be known for when the expression is false.
Currently it is the RPN Items that contain the tokens and it is the RPN Items that are actually attached (the RPN Item contains an array of attached RPN Item pointers). However, there is a problem as there is no way to determine where in the list the attached RPN Item is located. This will be needed to know where to insert conversion code or to calculate an offset.
This was corrected by adding an index member to the RPN Item. When an item is appended to the output list, this index is assigned the value where the item will be located (which is the size of the list just before appending the item). When an item is inserted into the list, the indexes of the items after the insertion point are incremented for each of the items new location.
For testing purposes, the text output for an RPN Item was modified to include the index of attached items. The affected results files were updated. See the commit log for more details of the changes.
[commit 5ef00faa25]
Later, this mechanism will also be used to attach tokens to certain command tokens. For instance, in a single line IF statement, the ELSE, END IF or last token on the line will be attached to the IF command token. The Encoder will use this attached token to calculate the offset from the IF command to the token, so that during execution, the number of words to skip will be known for when the expression is false.
Currently it is the RPN Items that contain the tokens and it is the RPN Items that are actually attached (the RPN Item contains an array of attached RPN Item pointers). However, there is a problem as there is no way to determine where in the list the attached RPN Item is located. This will be needed to know where to insert conversion code or to calculate an offset.
This was corrected by adding an index member to the RPN Item. When an item is appended to the output list, this index is assigned the value where the item will be located (which is the size of the list just before appending the item). When an item is inserted into the list, the indexes of the items after the insertion point are incremented for each of the items new location.
For testing purposes, the text output for an RPN Item was modified to include the index of attached items. The affected results files were updated. See the commit log for more details of the changes.
[commit 5ef00faa25]
Monday, September 2, 2013
Pre-Encoder Issues – Constants
While thinking about the design of the first step of the encoder (assigning codes to all tokens), I realized that there is no reason to add hidden conversion codes after numeric constants - the constants can be converted during translation, not every time the constant is executed at run-time. However, not all constants can be converted. All integer constants can be converted to double but not from double to integer.
The union for the constant in a token for either the double or integer value was removed so that both values are available and the one needed is used. A constant token needs to be able to hold there different data types, one for an integer, one for a double that can be used as a integer, and one for a double that can't be used as integer (outside the range of an integer).
To accomplish this, if a double constant (one that has a decimal point or exponent) that is within the integer range, its data type is set to integer and a new Double sub-code is set. A double constant that can't be converted is set to the double data type. When the constant is parsed and the integer data type is set, both the integer and double values in the token are set (in other words, the parser does the conversion).
For integer constants, the translator changes the data type of a constant token to the data type needed in the expression and the Double sub-code is removed. A hidden conversion code no longer needs to be added after constants. However, for double constants (those outside the range of a integer), if an integer is called for, the new "expected valid integer constant" error is returned.
For debugging, a "%" character is output after a constant if its data type was set to integer by the translator (even though it may be a floating point value). A constant that was set to the double data type lacks this indicator character. The expected results files were updated for the new indicator character and lack of convert codes after constants. New translator test #17 was added to test many aspects of each constant data type (integer, integer with double sub-code, and double). See the commit log for the full details of the changes. A commit was also made renaming the token sub-code access functions for better readability (and functions not being used were removed).
[commit f104bdf37e] [commit 0d3151f5f9]
The union for the constant in a token for either the double or integer value was removed so that both values are available and the one needed is used. A constant token needs to be able to hold there different data types, one for an integer, one for a double that can be used as a integer, and one for a double that can't be used as integer (outside the range of an integer).
To accomplish this, if a double constant (one that has a decimal point or exponent) that is within the integer range, its data type is set to integer and a new Double sub-code is set. A double constant that can't be converted is set to the double data type. When the constant is parsed and the integer data type is set, both the integer and double values in the token are set (in other words, the parser does the conversion).
For integer constants, the translator changes the data type of a constant token to the data type needed in the expression and the Double sub-code is removed. A hidden conversion code no longer needs to be added after constants. However, for double constants (those outside the range of a integer), if an integer is called for, the new "expected valid integer constant" error is returned.
For debugging, a "%" character is output after a constant if its data type was set to integer by the translator (even though it may be a floating point value). A constant that was set to the double data type lacks this indicator character. The expected results files were updated for the new indicator character and lack of convert codes after constants. New translator test #17 was added to test many aspects of each constant data type (integer, integer with double sub-code, and double). See the commit log for the full details of the changes. A commit was also made renaming the token sub-code access functions for better readability (and functions not being used were removed).
[commit f104bdf37e] [commit 0d3151f5f9]
Saturday, August 31, 2013
Pre-Encoder Issues – Array References
Before creating the new encoder class, some issues need to be taken care of in preparation for encoding. The first was the index member variable that was added to the token class for the token caching feature. There also needs to be an index member for use by the encoder. Also, the index token caching index does not have the "m_" prefix that member variables should have. Both of these issues were handled by giving this member variable the "m_id" name.
The second issue was that array references were not being translated correctly. When the get operand routine was called to get a reference, the process parentheses token routine called for the identifier with parentheses token was only asking for numeric expressions and if the expression only contained an identifier with or without parentheses and did not have the parentheses sub-code set, the reference flag of the token was set. Consider this sample statement and its incorrect translation:
At the end of the process parentheses token routine when a close parentheses is obtained, a check was added if the token being processed has its reference flag set, then no tokens are attached since it is an array reference and its sub-scripts have already been checked for integers.
The expected results for several translator tests (#1, #4, #7, #8 and #12) were updated for the proper handling of array reference sub-scripts.
[commit 34a796984a] [commit 5d3b918e67e]
The second issue was that array references were not being translated correctly. When the get operand routine was called to get a reference, the process parentheses token routine called for the identifier with parentheses token was only asking for numeric expressions and if the expression only contained an identifier with or without parentheses and did not have the parentheses sub-code set, the reference flag of the token was set. Consider this sample statement and its incorrect translation:
A(B,C)=DThe reference flags on the sub-scripts should not have been set, there should be integer conversion code inserted after the B and C variables, and there is no reason to attach the sub-scripts to the A( token since this token is known to be an array as functions can not be used in this way. This also applies to INPUT statements with array elements. Here is the correct translation:
B<ref> C<Ref> A(<ref>[B<ref>,C<ref> D Assign
B CvtInt C CvtInt A(<ref> D AssignIn the process parentheses token routine, if the token being processed has its reference flag set, then the expected data type of the expressions is set to integer for array subscripts. After getting each expression, for identifier with parentheses tokens being processed (this routine also handles defined functions with parentheses), if its reference flag is set, the resulting item on top of the done stack is dropped (it has already been checked for an integer). Otherwise, the done stack top items reference flag is set conditionally as before. This is for the possibility that the token being processed is a function call, which the encoder will handle.
At the end of the process parentheses token routine when a close parentheses is obtained, a check was added if the token being processed has its reference flag set, then no tokens are attached since it is an array reference and its sub-scripts have already been checked for integers.
The expected results for several translator tests (#1, #4, #7, #8 and #12) were updated for the proper handling of array reference sub-scripts.
[commit 34a796984a] [commit 5d3b918e67e]
Friday, August 30, 2013
The Encoding Procedure
There are several steps for encoding a translated RPN token list for a program line into an internal program code line.
Step 1: Each token type that does not have a code needs to be assigned a code. These token types include identifiers with and without parentheses, constants, and define functions with and without parentheses. For identifier tokens, what the identifier refers to (variable, array or user function) needs to be determined before a code can be assigned.
Initially only variables will be supported, so identifiers without parentheses will be assumed to be variables, and except for the constants, the other token types will report a "not yet implemented" error. For a variable, there will be a total of six program codes including Double Variable, Integer Variable, String Variable, Double Reference Variable, Integer Reference Variable and String Reference Variable. The specific code is selected based on the data type and reference flag of the token.
Later when support for arrays and functions is implemented, the dictionaries will be used to determine the type of the identifier. For arrays, the attached arguments are integer subscripts, so each needs to be checked for an integer value. For double type subscripts, a hidden integer conversion code will be inserted. An error will be reported for string subscripts. The number of subscripts will also be validated. Similarly for function arguments, the data type of each argument will be checked adding numeric conversion codes or reporting errors as needed.
Step 2: The instruction size of each token will be determined. Instructions are either one or two program words. The token type can be used for this determination. The operator and internal function (with and without parentheses) token types are one word, the command token type can be either and the others are two words. For commands, a new table entry flag is needed for the size of each command.
The translated token list will be scanned while maintaining the total encoded size of the line. For each token, an index (a new member for a token) will be set to the current total size. The total size is then incremented for the encode size of the token. This index will be used later for calculating the offset for single structure statements (like a single line IF statement).
Step 3: The encoded line is generated (its total size is now known). For each token, the first instruction word is created from the code and sub-code of the token. For two word instructions, the second operand word is determined. For index values, the identifier is looked up in a dictionary and the second operand word is set to the index of the dictionary entry. For offset values, the offset is calculated from the attached token. Block numbers will probably work similar to index values with an associated block dictionary (this mechanism is not defined yet).
Once the line has been encoded, it can be inserted into the program. At this point things get complicated. Some dictionary entries will refer to specific locations in the code (consider the IF and END IF example from the previous post). For any line inserted, replaced or deleted, all these references to program locations need to be adjusted if located after the point of change. However, this is not a worry initially since only dictionaries for variables, constants and remarks are needed and these will not contain program locations.
Step 1: Each token type that does not have a code needs to be assigned a code. These token types include identifiers with and without parentheses, constants, and define functions with and without parentheses. For identifier tokens, what the identifier refers to (variable, array or user function) needs to be determined before a code can be assigned.
Initially only variables will be supported, so identifiers without parentheses will be assumed to be variables, and except for the constants, the other token types will report a "not yet implemented" error. For a variable, there will be a total of six program codes including Double Variable, Integer Variable, String Variable, Double Reference Variable, Integer Reference Variable and String Reference Variable. The specific code is selected based on the data type and reference flag of the token.
Later when support for arrays and functions is implemented, the dictionaries will be used to determine the type of the identifier. For arrays, the attached arguments are integer subscripts, so each needs to be checked for an integer value. For double type subscripts, a hidden integer conversion code will be inserted. An error will be reported for string subscripts. The number of subscripts will also be validated. Similarly for function arguments, the data type of each argument will be checked adding numeric conversion codes or reporting errors as needed.
Step 2: The instruction size of each token will be determined. Instructions are either one or two program words. The token type can be used for this determination. The operator and internal function (with and without parentheses) token types are one word, the command token type can be either and the others are two words. For commands, a new table entry flag is needed for the size of each command.
The translated token list will be scanned while maintaining the total encoded size of the line. For each token, an index (a new member for a token) will be set to the current total size. The total size is then incremented for the encode size of the token. This index will be used later for calculating the offset for single structure statements (like a single line IF statement).
Step 3: The encoded line is generated (its total size is now known). For each token, the first instruction word is created from the code and sub-code of the token. For two word instructions, the second operand word is determined. For index values, the identifier is looked up in a dictionary and the second operand word is set to the index of the dictionary entry. For offset values, the offset is calculated from the attached token. Block numbers will probably work similar to index values with an associated block dictionary (this mechanism is not defined yet).
Once the line has been encoded, it can be inserted into the program. At this point things get complicated. Some dictionary entries will refer to specific locations in the code (consider the IF and END IF example from the previous post). For any line inserted, replaced or deleted, all these references to program locations need to be adjusted if located after the point of change. However, this is not a worry initially since only dictionaries for variables, constants and remarks are needed and these will not contain program locations.
Thursday, August 29, 2013
Program Code – Internal Format
The internal program code of a BASIC program will consist of 16-bit instruction words. Each instruction word will consist of two parts, the instruction code (command, operator, internal function, etc.) to perform, and the sub-code information that will only used to recreate the original program text (with the Parentheses, Colon and Let sub-codes). The sub-code information will not be used by the run-time module, but there will be a few exceptions (the Question and Keep on the various INPUT statement codes).
Some instruction words will have a second 16-bit operand word, which could contain one of three types of information depending on the instruction code. For instructions that are variables, arrays, constants, remarks, define functions, user functions, etc., this second word will be an index into one of the dictionaries. For single line structure statements (an IF statement for example), the second word will contain an offset to where to jump to. For example, in an IF statement followed by a set of commands to execute upon a true expression, the offset will tell the IF command how many words to skip when the expression is false.
The final type of information in the operand word is a block number, which will be used on multiple line structure statements. For example, an IF/END IF structure over several lines, both the IF and END IF commands will have the same block number. Structured block will probably also have a dictionary, so technically this operand type is also an index. The dictionary entry for a block will contain the locations of the IF and END IF statements. When running, if the IF expression is false, it will go to the dictionary for the block number to find out where the associated END IF is located and jump the instruction after it.
Some instruction words will have a second 16-bit operand word, which could contain one of three types of information depending on the instruction code. For instructions that are variables, arrays, constants, remarks, define functions, user functions, etc., this second word will be an index into one of the dictionaries. For single line structure statements (an IF statement for example), the second word will contain an offset to where to jump to. For example, in an IF statement followed by a set of commands to execute upon a true expression, the offset will tell the IF command how many words to skip when the expression is false.
The final type of information in the operand word is a block number, which will be used on multiple line structure statements. For example, an IF/END IF structure over several lines, both the IF and END IF commands will have the same block number. Structured block will probably also have a dictionary, so technically this operand type is also an index. The dictionary entry for a block will contain the locations of the IF and END IF statements. When running, if the IF expression is false, it will go to the dictionary for the block number to find out where the associated END IF is located and jump the instruction after it.
Subscribe to:
Posts (Atom)