There are several steps for encoding a translated RPN token list for a program line into an internal program code line.
Step 1: Each token type that does not have a code needs to be assigned a code. These token types include identifiers with and without parentheses, constants, and define functions with and without parentheses. For identifier tokens, what the identifier refers to (variable, array or user function) needs to be determined before a code can be assigned.
Initially only variables will be supported, so identifiers without parentheses will be assumed to be variables, and except for the constants, the other token types will report a "not yet implemented" error. For a variable, there will be a total of six program codes including Double Variable, Integer Variable, String Variable, Double Reference Variable, Integer Reference Variable and String Reference Variable. The specific code is selected based on the data type and reference flag of the token.
Later when support for arrays and functions is implemented, the dictionaries will be used to determine the type of the identifier. For arrays, the attached arguments are integer subscripts, so each needs to be checked for an integer value. For double type subscripts, a hidden integer conversion code will be inserted. An error will be reported for string subscripts. The number of subscripts will also be validated. Similarly for function arguments, the data type of each argument will be checked adding numeric conversion codes or reporting errors as needed.
Step 2: The instruction size of each token will be determined. Instructions are either one or two program words. The token type can be used for this determination. The operator and internal function (with and without parentheses) token types are one word, the command token type can be either and the others are two words. For commands, a new table entry flag is needed for the size of each command.
The translated token list will be scanned while maintaining the total encoded size of the line. For each token, an index (a new member for a token) will be set to the current total size. The total size is then incremented for the encode size of the token. This index will be used later for calculating the offset for single structure statements (like a single line IF statement).
Step 3: The encoded line is generated (its total size is now known). For each token, the first instruction word is created from the code and sub-code of the token. For two word instructions, the second operand word is determined. For index values, the identifier is looked up in a dictionary and the second operand word is set to the index of the dictionary entry. For offset values, the offset is calculated from the attached token. Block numbers will probably work similar to index values with an associated block dictionary (this mechanism is not defined yet).
Once the line has been encoded, it can be inserted into the program. At this point things get complicated. Some dictionary entries will refer to specific locations in the code (consider the IF and END IF example from the previous post). For any line inserted, replaced or deleted, all these references to program locations need to be adjusted if located after the point of change. However, this is not a worry initially since only dictionaries for variables, constants and remarks are needed and these will not contain program locations.
Friday, August 30, 2013
Thursday, August 29, 2013
Program Code – Internal Format
The internal program code of a BASIC program will consist of 16-bit instruction words. Each instruction word will consist of two parts, the instruction code (command, operator, internal function, etc.) to perform, and the sub-code information that will only used to recreate the original program text (with the Parentheses, Colon and Let sub-codes). The sub-code information will not be used by the run-time module, but there will be a few exceptions (the Question and Keep on the various INPUT statement codes).
Some instruction words will have a second 16-bit operand word, which could contain one of three types of information depending on the instruction code. For instructions that are variables, arrays, constants, remarks, define functions, user functions, etc., this second word will be an index into one of the dictionaries. For single line structure statements (an IF statement for example), the second word will contain an offset to where to jump to. For example, in an IF statement followed by a set of commands to execute upon a true expression, the offset will tell the IF command how many words to skip when the expression is false.
The final type of information in the operand word is a block number, which will be used on multiple line structure statements. For example, an IF/END IF structure over several lines, both the IF and END IF commands will have the same block number. Structured block will probably also have a dictionary, so technically this operand type is also an index. The dictionary entry for a block will contain the locations of the IF and END IF statements. When running, if the IF expression is false, it will go to the dictionary for the block number to find out where the associated END IF is located and jump the instruction after it.
Some instruction words will have a second 16-bit operand word, which could contain one of three types of information depending on the instruction code. For instructions that are variables, arrays, constants, remarks, define functions, user functions, etc., this second word will be an index into one of the dictionaries. For single line structure statements (an IF statement for example), the second word will contain an offset to where to jump to. For example, in an IF statement followed by a set of commands to execute upon a true expression, the offset will tell the IF command how many words to skip when the expression is false.
The final type of information in the operand word is a block number, which will be used on multiple line structure statements. For example, an IF/END IF structure over several lines, both the IF and END IF commands will have the same block number. Structured block will probably also have a dictionary, so technically this operand type is also an index. The dictionary entry for a block will contain the locations of the IF and END IF statements. When running, if the IF expression is false, it will go to the dictionary for the block number to find out where the associated END IF is located and jump the instruction after it.
Encoder – Introduction
As mentioned a while ago (March 25, 2011), the translation of more BASIC commands is being postponed so that the other modules can be developed. The translation of enough commands with expressions has been implemented (INPUT, LET, and PRINT) to make a useful, though very limited, BASIC program (limited by the lack of conditionals and loops).
These modules include the encoder to convert a translated program line into the internal program code, the recreator to convert the internal program code back to program text, and the run-time module to execute the internal program code. Once initial versions of these three modules are complete and connected to the GUI, additional commands will be implemented one at a time for each of the four modules.
Only certain elements of the BASIC language will be implemented initially to simplify development of the remaining modules. This includes just simple variables and constants, with arrays, defined functions and user functions implemented later. Variables come from the identifier with no parentheses token type, which could also be user functions (either a call to a function with no arguments, or an assignment of the function return value inside the function). For now this token type will be assumed to be a variable until functions are implemented.
A major part of encoder development is to define the internal program code format, the code that will be stored in the program and executed by the run-time module. The other major part are the dictionaries that will hold the information about variables, arrays, constants, remarks, functions, etc., which will be referenced from the program code. For example, the actual names (variables, functions, etc.) are not be stored in the internal program, but in a dictionary and the program code contains references to these dictionary entries.
A minor part of encoder development (not needed by the final application), is the conversion of individual instructions of the program code into text. This is similar to conversion of the translated tokens into text for output by the command line test mode, or in the GUI program view. The GUI program view translator output will be replaced by the encoder output, which will have the same reverse polish notation layout as the translator output.
These modules include the encoder to convert a translated program line into the internal program code, the recreator to convert the internal program code back to program text, and the run-time module to execute the internal program code. Once initial versions of these three modules are complete and connected to the GUI, additional commands will be implemented one at a time for each of the four modules.
Only certain elements of the BASIC language will be implemented initially to simplify development of the remaining modules. This includes just simple variables and constants, with arrays, defined functions and user functions implemented later. Variables come from the identifier with no parentheses token type, which could also be user functions (either a call to a function with no arguments, or an assignment of the function return value inside the function). For now this token type will be assumed to be a variable until functions are implemented.
A major part of encoder development is to define the internal program code format, the code that will be stored in the program and executed by the run-time module. The other major part are the dictionaries that will hold the information about variables, arrays, constants, remarks, functions, etc., which will be referenced from the program code. For example, the actual names (variables, functions, etc.) are not be stored in the internal program, but in a dictionary and the program code contains references to these dictionary entries.
A minor part of encoder development (not needed by the final application), is the conversion of individual instructions of the program code into text. This is similar to conversion of the translated tokens into text for output by the command line test mode, or in the GUI program view. The GUI program view translator output will be replaced by the encoder output, which will have the same reverse polish notation layout as the translator output.
Tuesday, August 27, 2013
New Translator – Release
The implementation of
new translator routines is now complete, at least with all the
commands that were previously implemented in the old translator
routines with the additional of support for multiple statements per
line (separated by colons). Version 0.4.6 has been released
(branch0.4 was merged to the master branch and tagged v0.4.6).
While preparing the
archives for uploading to SourceForge, I noticed that the test
scripts included in the binary archives were not correct and the scripts may not run properly. The previous uploads have been corrected.
To prevent this problem in the future, the CMake build file was modified to copy
the test files into the build directory and the scripts modified to
use these copies, not the ones from the source directory. Also to
prevent problems, a check was added to make sure the build directory
is not the same as the source directory.
The CMake build was
also modified to build an script that creates the binary release
archive with the needed files for the current platform (Linux or
Windows). This also means a .zip file on Windows and a .tar.gz file
on Linux. The files included on the application executable, read me,
license, release notes, test files and regression test script. On
Windows, the regression test batch file is also included. On Linux,
the memory test script and memory error suppression file are included.
Archive files
containing the source and binary files (with test files) have been
uploaded to SourceForge. For Windows, there is also the
ibcp-libs.zip file that contains all the dynamic linked libraries
required to run the program if the MinGW and QtSDK packages have not
been installed (extract into the ibcp directory). Linux should
already have the required libraries installed. This concludes the
0.4 development series. Implementation of the next step (encoder)
will now begin with the 0.5 development series.
Sunday, August 25, 2013
New Translator – Code Reorganization
Now that the old
translator routines have now been completely removed, it is time to
reorganize the main translator source file to put the functions in a
more logical top to bottom calling order. But first several of the
routines were renamed:
translate2() →
translate()
processOperator2() →
processOperator2()
getInternalFunction() →
processInternalFunction()
getParenToken() →
processParenToken()
The second two
functions were renamed because these functions are support functions
to the getOperand() function like the processCommand()
function is a support function to the getCommands(). Also,
these functions were made private since they will not be called from
outside the translator class (from the command translate functions).
Comments on the some of the functions were also reworded, corrected
and cleaned up. This concludes the implementation of the new
translator scheme.
Looking at some code
statistics, about 1330 lines of code were added during the
implementation of the new translator routines, but about 257 of these
lines were due to the additional token leak and extra delete
detection routines, so the new translator routines account for about
a net of 1073 lines. After the old translator routines were removed,
the code was about 2509 lines less. So the new translator is about
43% the size of the old translator. The simpler design will be
easier to maintain and utilize, so the change was worthwhile.
Saturday, August 24, 2013
Table Initialization – Expected Data Type
While reviewing the various To-Do entries marked in the code - words NOTE, TODO, and FIXME in comments that QtCreator highlights when the Todo plugin is enabled (Help/Plugins under Utilities) - there was a FIXME "remove" on a check in the table setup and check routine called during initialization. This check was for an unset expression information structure for an associated code.
This check was in a loop that scans all the tables entries and for each entry that contains operands (operators or internal functions), sets the expected data type for the last operand of an operator or first operand of non-operator. It does this by scanning the main code and all its associated codes recording the data type expected for each. After recording all the data types, if both double and integer was found, the expected data type is set to number, or if all data types (double, integer and string) were found, the expected data type is set to any.
However, the associated codes for the sub-string functions, which are set to the related assign sub-string function codes should not be searched and therefore have the second associated code index set to -1. The check loop was not checking for a -1 index and proceeded into the loop and should not have, so the check was added. The check with the FIXME for an unset expression information structure was in fact necessary because some associated codes do not have this structure, specifically the input parse type codes.
While studying this code, it was discovered that the AssignList code contained associated codes for AssignListInt and AssignListStr. This was used by the old translator routines, but not for the new translator routines because the AssignListType codes are now associated codes of the AssignType codes. The unnecessary associated codes were removed.
[commit 07ec1e4c4f]
This check was in a loop that scans all the tables entries and for each entry that contains operands (operators or internal functions), sets the expected data type for the last operand of an operator or first operand of non-operator. It does this by scanning the main code and all its associated codes recording the data type expected for each. After recording all the data types, if both double and integer was found, the expected data type is set to number, or if all data types (double, integer and string) were found, the expected data type is set to any.
However, the associated codes for the sub-string functions, which are set to the related assign sub-string function codes should not be searched and therefore have the second associated code index set to -1. The check loop was not checking for a -1 index and proceeded into the loop and should not have, so the check was added. The check with the FIXME for an unset expression information structure was in fact necessary because some associated codes do not have this structure, specifically the input parse type codes.
While studying this code, it was discovered that the AssignList code contained associated codes for AssignListInt and AssignListStr. This was used by the old translator routines, but not for the new translator routines because the AssignListType codes are now associated codes of the AssignType codes. The unnecessary associated codes were removed.
[commit 07ec1e4c4f]
Old Translator Removal – Reference Flag Cleanup
The process done
stack top routine (formally the find code
routine, see post from August 17)
contained a section that for the first operand of a sub-string
function used in an assignment or an assignment internal code token
(determined if the sub-string token had the reference flag set) if
the item on top of the done stack did not have its reference flag
set, an “expected assignment item” error was
reported. If the reference flag of the token was not set, then the
reference flag of the item on top of the done stack was cleared (a
reference is not needed). At the end of the routine. if the data
type of the done stack top item was not correct or could not be
converted, the reference flag state was used to determine which error
to return.
This reference flag
functionality is no longer needed in this routine since the checking
of references is handled else where in the new translator routines
(specifically by the using the reference argument of the get
operand routine and by the get internal function
for sub-string assignments). This code was removed, and since it was
removed, the INPUT and LET translate routines no longer
need to set the reference flag before calling this routine
(indirectly via the process final operand from the
INPUT translate routine) and clearing it afterward.
To simply the code a
bit more with respect to the reference flag, specifically pertaining
to sub-string assignments where the reference flag is set for a
sub-string function token in the get operand routine
(which the get internal function routine uses to
determine if a string reference should be requested for the first
operand), the reference flag is cleared upon return since the
reference status is not needed (a reference was already obtained).
Old Translator Removal – Process Final Operand
For the old translator, the process final operand routine handled the processing of the final operand of operators, internal functions, tokens with parentheses (arrays or functions) and internal codes (for example assign type, print type, and input assign type). The handling of internal functions and tokens with parentheses are now handled elsewhere by their respective get routines. So for the new translator, this routine is only called for operators and internal codes.
The code for handling tokens with parentheses, which included the attaching of the operands from the done stack, was removed. It turned out that is was no longer necessary to check for a reference token, which was used to determine if the item to be added to the RPN output list should also be pushed to the done stack. Since no internal codes need to be pushed to the done stack, it now only pushes operator tokens to the done stack. The PRINT translate routine was modified to only drop the done stack top item for the print only functions (TAB and SPC).
The process final operand routine was simplified after removing the unused code. The process done stack top routine (formally the find code routine, see post from August 17) is stilled called, which returns the first and last operands of the item was on the done stack top (the item is popped before returning). Afterward, the first operand is deleted if it is an open parentheses. For an operator token, the first operand is set to the operator token for a unary operator or the first operand of a binary operator, and the last operand is set from last operand of the item that was on done stack top. For an internal code, the last operand from the done stack top item is deleted if it is a closing parentheses.
[commit ca3b513ac9]
The code for handling tokens with parentheses, which included the attaching of the operands from the done stack, was removed. It turned out that is was no longer necessary to check for a reference token, which was used to determine if the item to be added to the RPN output list should also be pushed to the done stack. Since no internal codes need to be pushed to the done stack, it now only pushes operator tokens to the done stack. The PRINT translate routine was modified to only drop the done stack top item for the print only functions (TAB and SPC).
The process final operand routine was simplified after removing the unused code. The process done stack top routine (formally the find code routine, see post from August 17) is stilled called, which returns the first and last operands of the item was on the done stack top (the item is popped before returning). Afterward, the first operand is deleted if it is an open parentheses. For an operator token, the first operand is set to the operator token for a unary operator or the first operand of a binary operator, and the last operand is set from last operand of the item that was on done stack top. For an internal code, the last operand from the done stack top item is deleted if it is a closing parentheses.
[commit ca3b513ac9]
Old Translator – Removal (Unused Definitions)
There are two sub-code definitions that were only used by the old translator routines. The semicolon sub-code was used for unnecessary semicolons that were entered. Since this is no longer permitted, this sub-code was removed. The end sub-code was used to mark the last input parse code in an INPUT statement. Since the new INPUT translation uses the input begin code to mark the end of the input parse codes, this sub-code is not needed and was removed.
The values of the sub-codes were modified to close the bit gaps left by the semicolon and end sub-codes. Also several sub-codes that are only used by the translator (used, last, and unused) were given higher bit code values. It will be convenient and desirable if the same sub-code definitions can be used by the translator and program code. The program sub-code bit values must fit in a limited space of a 16-bit instruction word that will be shared with the code value. It may also possible to share bit values for sub-codes that will never be used with the same code. For example, the question and keep sub-codes will never be used on the same code (input begin string vs. input and input-prompt) so the same bit values could be used.
There was also the end-expression flag on table entries that could signal the end of an expression (close parentheses, comma, semicolon, rem-operator, and end-of-line). This flag was needed for the token centric old translator, but not used for the new translator, so it was removed.
[commit d356621d95]
The values of the sub-codes were modified to close the bit gaps left by the semicolon and end sub-codes. Also several sub-codes that are only used by the translator (used, last, and unused) were given higher bit code values. It will be convenient and desirable if the same sub-code definitions can be used by the translator and program code. The program sub-code bit values must fit in a limited space of a 16-bit instruction word that will be shared with the code value. It may also possible to share bit values for sub-codes that will never be used with the same code. For example, the question and keep sub-codes will never be used on the same code (input begin string vs. input and input-prompt) so the same bit values could be used.
There was also the end-expression flag on table entries that could signal the end of an expression (close parentheses, comma, semicolon, rem-operator, and end-of-line). This flag was needed for the token centric old translator, but not used for the new translator, so it was removed.
[commit d356621d95]
Old Translator – Removal (Sub-String Data Type)
The sub-string data type was used by the old translator to identify the sub-string functions (LEFT$, MID$, and RIGHT$) that can be used to assign part of a string variable. The idea was more appropriate with the original String class that would make handling during run-time easy. The String class has since been replaced with the Qt QString class, which has different requirements during run-time and this has been accounted for with the new sub-string assignment translation scheme (see posts on new design and with multiple assignments).
The sub-string data type is not needed in the new translator routines and has been removed. The returning data type of the LEFT$, MID$ and RIGHT$ functions is now just a String like all of the other string functions. For assignments with these functions, the new sub-string flag is used. The AssignSubStr code was removed because it was replaced with the AssignLeft, AssignMid2, AssignMid3, AssignRight codes and the AssignListMix code was removed because the various AssignKeep codes replace its functionality (see posts referenced above). See the commit log for other changes made to remove the sub-string data type.
[commit 71e97bffa2]
The sub-string data type is not needed in the new translator routines and has been removed. The returning data type of the LEFT$, MID$ and RIGHT$ functions is now just a String like all of the other string functions. For assignments with these functions, the new sub-string flag is used. The AssignSubStr code was removed because it was replaced with the AssignLeft, AssignMid2, AssignMid3, AssignRight codes and the AssignListMix code was removed because the various AssignKeep codes replace its functionality (see posts referenced above). See the commit log for other changes made to remove the sub-string data type.
[commit 71e97bffa2]
Old Translator – Removal (Step 1)
The old translator routines will be removed in steps, the first step being the largest. All of the functions related to the old translator were removed along with the token handler and command handler functions. Also removed were the translator variables only used by the old translator routines including the state, mode, count stack (used for arrays and functions) and command stack (used for commands). This were are needed for the token centric old translator.
The program model was changed temporarily to call the new main translate routine, however, this will be changed back once the new translator routines are renamed (removing the "2" in their names that were added to avoid a conflict with the old translator routines).
The temporary test command line options to activate the new translator were also removed. The original test command line options now use the new translator. The old translator expected results files that were previously saved were removed. The temporary test scripts for running the new translator were removed and the commands added to look for old expected results files were removed from the original test scripts. All tests pass with the original test scripts using the new translator routines (Windows testing was not performed yet).
[commit f150fbeb3f]
The program model was changed temporarily to call the new main translate routine, however, this will be changed back once the new translator routines are renamed (removing the "2" in their names that were added to avoid a conflict with the old translator routines).
The temporary test command line options to activate the new translator were also removed. The original test command line options now use the new translator. The old translator expected results files that were previously saved were removed. The temporary test scripts for running the new translator were removed and the commands added to look for old expected results files were removed from the original test scripts. All tests pass with the original test scripts using the new translator routines (Windows testing was not performed yet).
[commit f150fbeb3f]
Friday, August 23, 2013
New Translator – Complete (Tagged)
The implementation of the new translator routines is now complete for all the items previously completed in the old translator routines (LET, PRINT, INPUT and REM, with the additional item of multiple statement support). Before removing the old translator routines, this is a good place to set a tag even though it has been only three commits from the last tag. This \will be the last commit that will contain the old translator routines.
Version v0.4.5 has been tagged. All tests, including the new multiple statement test (#16) with the new translator routines. Test #16 does not pass with the old translator routines since multiple statement support was never added (would have required a new colon token handler). Some comments were added indicating items that need to be removed with the old translators, which will now commence.
[commit 51ef63448b]
Version v0.4.5 has been tagged. All tests, including the new multiple statement test (#16) with the new translator routines. Test #16 does not pass with the old translator routines since multiple statement support was never added (would have required a new colon token handler). Some comments were added indicating items that need to be removed with the old translators, which will now commence.
[commit 51ef63448b]
New Translator – Colons
Multiple BASIC statements per line will be supported where statements are separated by a colon. As mentioned in the last post, the newly implemented get commands routine, as indicated by its name, was intended to be able to process multiple statements per line, though the code to do that was not implemented.
For multiple statements, colons will not be stored in the program. Instead, there will be a colon sub-code set on the last token of the statement, which is usually the command token, but not always (for example, an assign of a LET or a semicolon of a PRINT). Multiple consecutive colons or a lone trailing colon will not be permitted as they add nothing.
To add support for multiple statements, the get commands routine was modified by adding a check for a colon token after checking for a RemOp token. For a colon token, the token is not needed and so it is deleted. The colon sub-code is set on the last token added to the RPN output list. The loop then continues with looking for the another command. Other changes requires was to add the end-statement flag to the colon table entry, to add the colon sub-code definition, and to modify the token text routine to detect and output the colon sub-code.
A new translator test (#16) was added for testing multiple statements with various commands before colons, and with several statements with errors. The old translator routines crashes on this test as expected since multiple statements are not supported. This will not be fixed since now that the new translator supports everything the old translator supported, the old translator routines can now be removed.
[commit 734521d17f]
For multiple statements, colons will not be stored in the program. Instead, there will be a colon sub-code set on the last token of the statement, which is usually the command token, but not always (for example, an assign of a LET or a semicolon of a PRINT). Multiple consecutive colons or a lone trailing colon will not be permitted as they add nothing.
To add support for multiple statements, the get commands routine was modified by adding a check for a colon token after checking for a RemOp token. For a colon token, the token is not needed and so it is deleted. The colon sub-code is set on the last token added to the RPN output list. The loop then continues with looking for the another command. Other changes requires was to add the end-statement flag to the colon table entry, to add the colon sub-code definition, and to modify the token text routine to detect and output the colon sub-code.
A new translator test (#16) was added for testing multiple statements with various commands before colons, and with several statements with errors. The old translator routines crashes on this test as expected since multiple statements are not supported. This will not be fixed since now that the new translator supports everything the old translator supported, the old translator routines can now be removed.
[commit 734521d17f]
Thursday, August 22, 2013
New Translator – Remarks
There are two forms of remarks (comments), the REM command and the remark operator (a single quote). The remark operator can be placed anywhere an end-of-statement token can be put including at the beginning of the line where a command is. There are no commands after a remark since all characters are part of the comment up to the end of the line. Remarks were implemented a couple of different ways and while these all worked, these solutions were not very clean (there were multiple checks for the Rem and RemOp tokens in several routines).
The final solution was to implement a new get commands routine, which as the name implies will be able to handle multiple commands on the line separated by colons (which wasn't implemented yet). The new routine contains a loop, which begins by getting a token. If the token is a Rem or RemOp, the loop is exited. Otherwise the token is processed by calling the process command routine (see below). If the terminating token of the command is a RemOp, the loop is exited. Otherwise, for now, the routine returns the terminating token from the command translator routine and the done status.
When the loop is exited in the new get commands routine, which will be due to a Rem or RemOp token, the token is appended to the RPN output list, and the next token is obtained, which should be an end-of-line token unless there is a bug in the parser. The end-of-line token is returned as the terminating token with the done status.
The original get command routine was renamed to the process command routine, which made more sense and this routine won't be public as only the get commands routine will be the only caller. It was modified to receive the first token from the token argument instead of getting the token itself.
The expected results for translator test #15 (Remark tests) needed to be updated due to the change in translation of the PRINT and INPUT commands (the old results were saved). Now all the translator tests pass with the new translator routines.
[commit a3a71526f9]
The final solution was to implement a new get commands routine, which as the name implies will be able to handle multiple commands on the line separated by colons (which wasn't implemented yet). The new routine contains a loop, which begins by getting a token. If the token is a Rem or RemOp, the loop is exited. Otherwise the token is processed by calling the process command routine (see below). If the terminating token of the command is a RemOp, the loop is exited. Otherwise, for now, the routine returns the terminating token from the command translator routine and the done status.
When the loop is exited in the new get commands routine, which will be due to a Rem or RemOp token, the token is appended to the RPN output list, and the next token is obtained, which should be an end-of-line token unless there is a bug in the parser. The end-of-line token is returned as the terminating token with the done status.
The original get command routine was renamed to the process command routine, which made more sense and this routine won't be public as only the get commands routine will be the only caller. It was modified to receive the first token from the token argument instead of getting the token itself.
The expected results for translator test #15 (Remark tests) needed to be updated due to the change in translation of the PRINT and INPUT commands (the old results were saved). Now all the translator tests pass with the new translator routines.
[commit a3a71526f9]
Sunday, August 18, 2013
New Translator – INPUT Statements (Tagged)
The implementation of INPUT statement translation went fairly quickly using the new translation routines. Only a couple of days were required using the new command centric translator routines, compared to well over a month using the old token centric translator routines. This should be an indicator how the implementation should go for the rest of the BASIC commands.
The INPUT statements implementation in the new translator is now complete and version v0.4.4 has been tagged. All tests still pass with the old translator routines. All translator tests pass with the new translator routines with the exception of the REM statements (test #15) as this has not been implemented yet. Some slight cleanup was done with the latest commit along with updating the files for v0.4.4. Implementation of REM statements and operators can now commence in the new translator.
[commit 00815b4bac]
The INPUT statements implementation in the new translator is now complete and version v0.4.4 has been tagged. All tests still pass with the old translator routines. All translator tests pass with the new translator routines with the exception of the REM statements (test #15) as this has not been implemented yet. Some slight cleanup was done with the latest commit along with updating the files for v0.4.4. Implementation of REM statements and operators can now commence in the new translator.
[commit 00815b4bac]
New INPUT Translation
The new INPUT translate routine required several local variables besides a status variable, including an index into the RPN output list of the InputBegin or InputBeginStr code where to insert the input parse codes (see below), a done flag, and an input token. This routine will be used for both the INPUT and INPUT PROMPT commands.
For the INPUT command a new token is created for the InputBegin code. For the INPUT PROMPT command, the get expression routine is called to get a string expression. The done stack top item is dropped and the terminating token is checked. If the terminating token is a comma, the 'Question' sub-code is set in the token, else if the token is not a semicolon, the "expecting operator, semicolon or comma" error is returned. The token is set to the InputBeginStr code.
The index to where the input begin token will be inserted into the RPN output list is obtained by the current count of the number of items in the list. The token is appended to the output and a loop is entered to get each of the input variables.
The get operand routine is called to get any type of variable reference. The next token is obtained. If the token is a comma, the done flag is set to false and the input token is set to the token (to be reused for the input assign token). If the token is a semicolon, the 'Keep' sub-code is set in the command token, the done flag is set to true, the input token is set to the token (for reuse), and the next token is obtained. For all other tokens, the done flag is set to true, and the input token is set to a new token.
The input token is set to the InputAssign code and the process final operand routine is called to process the reference on top of the done stack and set the input token code to the appropriate input assign code for the data type of the reference. A new token is created with the second associated code of the input assign code, which will be the coordinating input parse code. The input parse token is then inserted into the output list at the begin index. The input parse token for each subsequent reference will be inserted in front of the last input parse token, and therefore all the input parse tokens will be in the desired reverse order (making execution easier). The loop continues while the done flag is not set.
Upon exit from the loop, if the status is set to an error, the error is returned. Otherwise, the command token is appended to the output and if terminating token is not an end-of-statement, the "expecting comma, semicolon or end-of-statement" error is returned. (Note the reversal of the "comma" and "semicolon" words in the error message used for the PRINT statement, which makes more sense for the INPUT statement.) Finally the done status is returned. See the commit log for the other minor changes required to implement the INPUT translate routine.
[commit 8b079faf61]
For the INPUT command a new token is created for the InputBegin code. For the INPUT PROMPT command, the get expression routine is called to get a string expression. The done stack top item is dropped and the terminating token is checked. If the terminating token is a comma, the 'Question' sub-code is set in the token, else if the token is not a semicolon, the "expecting operator, semicolon or comma" error is returned. The token is set to the InputBeginStr code.
The index to where the input begin token will be inserted into the RPN output list is obtained by the current count of the number of items in the list. The token is appended to the output and a loop is entered to get each of the input variables.
The get operand routine is called to get any type of variable reference. The next token is obtained. If the token is a comma, the done flag is set to false and the input token is set to the token (to be reused for the input assign token). If the token is a semicolon, the 'Keep' sub-code is set in the command token, the done flag is set to true, the input token is set to the token (for reuse), and the next token is obtained. For all other tokens, the done flag is set to true, and the input token is set to a new token.
The input token is set to the InputAssign code and the process final operand routine is called to process the reference on top of the done stack and set the input token code to the appropriate input assign code for the data type of the reference. A new token is created with the second associated code of the input assign code, which will be the coordinating input parse code. The input parse token is then inserted into the output list at the begin index. The input parse token for each subsequent reference will be inserted in front of the last input parse token, and therefore all the input parse tokens will be in the desired reverse order (making execution easier). The loop continues while the done flag is not set.
Upon exit from the loop, if the status is set to an error, the error is returned. Otherwise, the command token is appended to the output and if terminating token is not an end-of-statement, the "expecting comma, semicolon or end-of-statement" error is returned. (Note the reversal of the "comma" and "semicolon" words in the error message used for the PRINT statement, which makes more sense for the INPUT statement.) Finally the done status is returned. See the commit log for the other minor changes required to implement the INPUT translate routine.
[commit 8b079faf61]
Subscribe to:
Posts (Atom)