From the last change, most uses of the table instance reference member of the parser class was removed; replaced with use of static table functions. The last two uses were for setting of the code (plus type and data type) for constant tokens. For constant string tokens, the setting of the token code was moved to the token constructor for string constants.
The other use of the table instance was for number constants also in the main function operator function of the parser, but with some statements to determine from the desired data type what the data type of the number constant should be and whether to set the Integer Constant sub-code. These statements and the setting of the code of the token were moved to the token constructors for integer and double constants.
For integer constant tokens, the data type is set to integer unless the desired data type is double. There is no need to for the Integer Constant sub-code since the constant is an integer. For double constant tokens, if the value is within the integer range, the Integer Constant is only set if the desired data type is not an integer or double. If the desired data type is indeterminate (Number or Any) then the data type of the token is set to Double. For values outside the integer range, the data type is set to Double.
The table instance reference member was removed since it was no longer being used. The remaining parser constructor was only initializing the input string stream member, so it was moved to and made in-line in the header file.
[branch table commit b89a1b663b]
Showing posts with label Parser. Show all posts
Showing posts with label Parser. Show all posts
Sunday, January 4, 2015
Parser – Table Entry Pointers
Before the code enumeration (and token type enumeration) can be replaced with the new code type enumeration, uses of the code enumeration type need to be replaced with the use of table entry pointers. This will be done on different parts at a time adding table entry class access functions as needed.
This process was started by changing the table find functions from returning a code enumerator to a table entry pointer (returning a null pointer to indicate no table entry was found). Only the parser routines were using the find functions, so the get identifier and get operator functions were updated to use table entry pointers instead of a code enumerator.
A few table entry access functions were added to support the parser changes including the code, is code, name, has flag and alternate functions. For now the code function simply returns an index value of the entry by subtracting the base of the table entries array. This function is temporary. The is code function checks if the table entry is for a particular code. For now it compares to the code function return value, but eventually will compare to the code member that will be added to the table entry. The alternate function is similar to the alternate code function but returns a table entry pointer.
The code argument of the two token constructors for codes were changed to table entry pointers. For now they just access the code function of the table entry. The immediate goal is to change the interfaces and later to change the underlying code when the token code member is replaced with a table entry pointer. The token constructor taking a code was temporarily left (though the unneeded arguments were removed) for use by the translator routines.
The table entry class was made a friend class of the table class (specifically so the alternate member function can access the static alternate member of the table class). Eventually, the table entry and table classes will be combined into a single class.
[branch table commit 78f0b39780]
This process was started by changing the table find functions from returning a code enumerator to a table entry pointer (returning a null pointer to indicate no table entry was found). Only the parser routines were using the find functions, so the get identifier and get operator functions were updated to use table entry pointers instead of a code enumerator.
A few table entry access functions were added to support the parser changes including the code, is code, name, has flag and alternate functions. For now the code function simply returns an index value of the entry by subtracting the base of the table entries array. This function is temporary. The is code function checks if the table entry is for a particular code. For now it compares to the code function return value, but eventually will compare to the code member that will be added to the table entry. The alternate function is similar to the alternate code function but returns a table entry pointer.
The code argument of the two token constructors for codes were changed to table entry pointers. For now they just access the code function of the table entry. The immediate goal is to change the interfaces and later to change the underlying code when the token code member is replaced with a table entry pointer. The token constructor taking a code was temporarily left (though the unneeded arguments were removed) for use by the translator routines.
The table entry class was made a friend class of the table class (specifically so the alternate member function can access the static alternate member of the table class). Eventually, the table entry and table classes will be combined into a single class.
[branch table commit 78f0b39780]
Saturday, January 3, 2015
Parser – Parentheses Token Handling
With the forthcoming change from the token type and code enumerations to the code type enumeration, it will be advantageous if all similar codes have the same code type. Table flags will be used command, operator and functions codes since some will need their own code type (for example the LET command and equal operator codes).
The codes being discussed are the codes with operands: constants, variables, arrays, defined functions and user functions. Each have separate codes for each of the three data types, and each (except constants) have an additional three of each data type for reference codes. Currently only constants and variables are fully implemented. For variables, there will be a single Variable code type for each of its six codes. The translator will only need to check this single code type for a variable code.
There was a distinction between token types with parentheses and without (internal functions, defined functions, and generic tokens). For defined and internal functions, this distinction was removed (the token type enumerators were combined for each). For internal functions, instead of checking the token type to determine if there are parentheses, the number of operands is now checked. Eventually defined functions (and later user functions) will have a similar check as the number of operands will be stored in their associated dictionaries.
The parser get word helper function was modified to only check if an open parentheses is present and add it to the identifier string as before, but not take it from the input stream. The parentheses is still needed for functions when searching the table. The get identifier function already removed the parentheses if a table entry was not found, but was modified to remove the parentheses from the input for internal functions only. A new get parentheses access function was added to check if the next character in the stream is an open parentheses, remove the parentheses, and return whether it was a parentheses.
The translator get operand function was modified accordingly. For the single internal function token type, the process internal function routine is only called if the function has operands. For the single defined function token type, the process parentheses token is only called if the next character in the parser is a parentheses (calls the new get parentheses function to remove the parentheses). For the generic parentheses token type (array or user function), the get parentheses function is called to remove the parentheses.
The two separate table entries for defined functions with and without parentheses remain for now with their associated code enumerators and are used to determine if parentheses are present. Once defined functions are fully implemented, these two codes will be replaced with six codes (as described above) and the defined function dictionary will contain whether there should be parentheses (if there are operands).
The test token stream inserter and print token functions were modified for the change in token type enumerators. The token has parentheses access function was only used by the print token function, which used a static map member. The print token function was modified to not require this access function, so it and the static map member were removed. There was also a static map member for precedence. Since all tokens now have a code assigned, all precedences can be obtained from the table, so this static member and its access function were also removed. The expected parser results for tests #2, #4 and #5 were updated for the change in the token types.
[branch table commit 3dd768f604]
The codes being discussed are the codes with operands: constants, variables, arrays, defined functions and user functions. Each have separate codes for each of the three data types, and each (except constants) have an additional three of each data type for reference codes. Currently only constants and variables are fully implemented. For variables, there will be a single Variable code type for each of its six codes. The translator will only need to check this single code type for a variable code.
There was a distinction between token types with parentheses and without (internal functions, defined functions, and generic tokens). For defined and internal functions, this distinction was removed (the token type enumerators were combined for each). For internal functions, instead of checking the token type to determine if there are parentheses, the number of operands is now checked. Eventually defined functions (and later user functions) will have a similar check as the number of operands will be stored in their associated dictionaries.
The parser get word helper function was modified to only check if an open parentheses is present and add it to the identifier string as before, but not take it from the input stream. The parentheses is still needed for functions when searching the table. The get identifier function already removed the parentheses if a table entry was not found, but was modified to remove the parentheses from the input for internal functions only. A new get parentheses access function was added to check if the next character in the stream is an open parentheses, remove the parentheses, and return whether it was a parentheses.
The translator get operand function was modified accordingly. For the single internal function token type, the process internal function routine is only called if the function has operands. For the single defined function token type, the process parentheses token is only called if the next character in the parser is a parentheses (calls the new get parentheses function to remove the parentheses). For the generic parentheses token type (array or user function), the get parentheses function is called to remove the parentheses.
The two separate table entries for defined functions with and without parentheses remain for now with their associated code enumerators and are used to determine if parentheses are present. Once defined functions are fully implemented, these two codes will be replaced with six codes (as described above) and the defined function dictionary will contain whether there should be parentheses (if there are operands).
The test token stream inserter and print token functions were modified for the change in token type enumerators. The token has parentheses access function was only used by the print token function, which used a static map member. The print token function was modified to not require this access function, so it and the static map member were removed. There was also a static map member for precedence. Since all tokens now have a code assigned, all precedences can be obtained from the table, so this static member and its access function were also removed. The expected parser results for tests #2, #4 and #5 were updated for the change in the token types.
[branch table commit 3dd768f604]
Friday, January 2, 2015
Parser – Token Creation
A side effect of the last change was that tokens for both codes for operators, functions and commands and codes with operands (constants, variables, arrays, defined functions and user functions) were using the same token constructor. This token constructor searched through alternate codes for the code with the appropriate return data type.
This was unnecessary for operator, function and command codes. The table new token function called for these codes passed in the return data type from the table entry of the code. The token constructor then called the new table set token code function. Since the data type matched the return data type (which was just passed in), no alternates were checked and the code, type and data type of the token was set. This was extra unnecessary work.
A new token constructor was added for operator, function and command codes, which only required arguments for the code, column, length and string. The string argument is only used for the REM and REM operator codes. This constructor replaces the table new token function. This constructor calls the table set code function which just sets the code, type and data type of the token from the table entry of the code. For consistency the code argument was put first in the other token constructor for codes with operands.
While looking at the creation of tokens, I decided that using the standard unique pointer within the parser was unnecessary. The parser can just allocate a token and return its pointer. The translator then can put the allocated tokens into a standard shared pointer. The parser was changed to use plain token pointers. The translator routines were changed to use the new token constructor directly via the standard make shared function. The translator get operand was changed to use the reset function to set the token member since shared pointers cannot be assigned directly to a pointer.
[branch table commit 1bfb76ae0a]
This was unnecessary for operator, function and command codes. The table new token function called for these codes passed in the return data type from the table entry of the code. The token constructor then called the new table set token code function. Since the data type matched the return data type (which was just passed in), no alternates were checked and the code, type and data type of the token was set. This was extra unnecessary work.
A new token constructor was added for operator, function and command codes, which only required arguments for the code, column, length and string. The string argument is only used for the REM and REM operator codes. This constructor replaces the table new token function. This constructor calls the table set code function which just sets the code, type and data type of the token from the table entry of the code. For consistency the code argument was put first in the other token constructor for codes with operands.
While looking at the creation of tokens, I decided that using the standard unique pointer within the parser was unnecessary. The parser can just allocate a token and return its pointer. The translator then can put the allocated tokens into a standard shared pointer. The parser was changed to use plain token pointers. The translator routines were changed to use the new token constructor directly via the standard make shared function. The translator get operand was changed to use the reset function to set the token member since shared pointers cannot be assigned directly to a pointer.
[branch table commit 1bfb76ae0a]
Parser – Codes With Operands
The last token type not being set fully in the parser were codes with operands (constants, variables, arrays, defined functions and user functions). Constant tokens were corrected with the last change. Arrays, defined functions and user functions are not fully implemented and so did not need to be changed. Variables however, were only partially set in the parser (only to the base Variable or Variable Reference code) and weren't set for the data type of the variable until the translator.
The parser get identifier function was modified to set the data type to Double if the word obtained from the input does not have a type. This applies to all identifiers not found in the table. The token constructor for codes is used for commands, operators, functions and codes with operands. The type argument was unnecessary since that is set from the table entry. However, an issue was found with how codes were found in the table.
For operators and functions, the [return] data type of the token is set from the table entry. (This issue doesn't affect commands since command don't have a return data type.) For codes with operands, the data type of the identifier is used to find the appropriate table entry (for example, Variable, Variable Integer, or Variable String) by looking at the data types of alternate codes. The current table set token code function did not work correctly because it searches alternate codes by operand data type. For this instance, the alternate codes need to be search by return data type.
A new set token code function was added without an operand index argument to search by return data type. If the data type (of the identifier) does match the code passed, then the alternate codes are searched for a matching return data type. If there are no alternates or none were found, then the code passed is set in the token along with the token type of the code. The data type is set to the data type of the identifier and not from the table entry (which may not match for codes like arrays that are not fully implemented yet).
The type argument was removed from the token constructor for codes. The type from the table entry of the code was passed (and the new set token code now does this). A call to the new set token code was added to the body of the constructor (previously empty).
Since codes for constants, variable, and variable references were found in the table incorrectly by operand data type, these table entries contained operand data types so that it would work. These codes do not have operands (in the sense that operands and functions do within expressions; not to be confused that in the program, these codes do have an operand index). These table entries were corrected with expression info instances containing no operands.
The translator get operand previously set the default data type of the token just obtained (set to Double if None and not a function). This was removed since the parser now does this. The token set default data type function called to do this was removed. The call to set the code for a no parentheses (variable) token was also no longer needed. With the parser now setting the default data type to Double, the expected results to the parser tests (#2, #3 and #5) needed to be updated.
[branch table commit acc37f0650]
The parser get identifier function was modified to set the data type to Double if the word obtained from the input does not have a type. This applies to all identifiers not found in the table. The token constructor for codes is used for commands, operators, functions and codes with operands. The type argument was unnecessary since that is set from the table entry. However, an issue was found with how codes were found in the table.
For operators and functions, the [return] data type of the token is set from the table entry. (This issue doesn't affect commands since command don't have a return data type.) For codes with operands, the data type of the identifier is used to find the appropriate table entry (for example, Variable, Variable Integer, or Variable String) by looking at the data types of alternate codes. The current table set token code function did not work correctly because it searches alternate codes by operand data type. For this instance, the alternate codes need to be search by return data type.
A new set token code function was added without an operand index argument to search by return data type. If the data type (of the identifier) does match the code passed, then the alternate codes are searched for a matching return data type. If there are no alternates or none were found, then the code passed is set in the token along with the token type of the code. The data type is set to the data type of the identifier and not from the table entry (which may not match for codes like arrays that are not fully implemented yet).
The type argument was removed from the token constructor for codes. The type from the table entry of the code was passed (and the new set token code now does this). A call to the new set token code was added to the body of the constructor (previously empty).
Since codes for constants, variable, and variable references were found in the table incorrectly by operand data type, these table entries contained operand data types so that it would work. These codes do not have operands (in the sense that operands and functions do within expressions; not to be confused that in the program, these codes do have an operand index). These table entries were corrected with expression info instances containing no operands.
The translator get operand previously set the default data type of the token just obtained (set to Double if None and not a function). This was removed since the parser now does this. The token set default data type function called to do this was removed. The call to set the code for a no parentheses (variable) token was also no longer needed. With the parser now setting the default data type to Double, the expected results to the parser tests (#2, #3 and #5) needed to be updated.
[branch table commit acc37f0650]
Wednesday, December 31, 2014
Constant Token Codes
The code and token type enumerations will be combined into a single code type enumeration. Before proceeding, the parser needs to return all tokens assigned to a code. This has been mostly accomplished, though one exception is constant tokens, which were still being assigned codes in the translator. This is complicated because the type of numerical constants may not be known when the constant is parsed. Consider these statements:
Instead of passing whether a number is allowed flag to the parser, the requested data type is now passed. If the data type is integer or double, the code of a constant token is fixed (the Integer Constant sub-code is not needed and is cleared if set). For other requested data types, either the Constant or Constant Integer code is set is described above with the Integer Constant sub-code set for small doubles. The parser also now sets the Constant String code for string constants. The parser makes no attempt to report any errors for data type mismatches.
The decimal flag argument of the token constructor for double constants was removed as the data type is set to Double and the Integer Constant sub-code is set if the value is within the integer range. The convert code was cleaned up by making the desired data type the primary switch and there was no need for secondary switches on the token data type since only one of two data types need to be checked for each desired data type. A convert constant helper function was added to handle changing constant token codes.
The table find code function was simplified due to the change on how constants are represented. The first argument of the set token and set token code functions were changed from a standard shared token pointer reference to a straight token pointer so that they can be called from the parser (with a standard unique pointer), translator (with a standard shared pointer) or token member function (with just a pointer). This simply required calling the get access function of the unique or shared pointers.
The translator get operand function no longer sets the code for constants. The get expression and process internal function functions no longer need to look for and set the codes for constants (the later needs to clear the Integer Constant sub-code for functions taking both number argument types, specifically ABS, SGN and STR$). And the get token function now only needs to pass the data type to the parser. The token convert and table find code functions are used by the translator and will finalize constants not set by the parser once the final data type is known.
[branch table commit 3099f8850f]
A = B + 5For numerical constants, both the integer and double representations of the constant is stored in the constant dictionary except for the case where a double constant does not fit into a 32-bit signed integer. Optimally the representation required is used without a hidden conversion code to unnecessarily convert the constant. For the first statement above, the double value of the constant is used. With the other two statements, the integer value of the constant is used. Number constant tokens have three states:
A% = B% + 5
A% = B% + 5.4
- An integer (no decimal point or exponent; fits into 32 bits)
- A small double (has a decimal point or an exponent; fits into 32 bits when converted)
- A large double (does not fit into 32 bits; cannot be converted)
Instead of passing whether a number is allowed flag to the parser, the requested data type is now passed. If the data type is integer or double, the code of a constant token is fixed (the Integer Constant sub-code is not needed and is cleared if set). For other requested data types, either the Constant or Constant Integer code is set is described above with the Integer Constant sub-code set for small doubles. The parser also now sets the Constant String code for string constants. The parser makes no attempt to report any errors for data type mismatches.
The decimal flag argument of the token constructor for double constants was removed as the data type is set to Double and the Integer Constant sub-code is set if the value is within the integer range. The convert code was cleaned up by making the desired data type the primary switch and there was no need for secondary switches on the token data type since only one of two data types need to be checked for each desired data type. A convert constant helper function was added to handle changing constant token codes.
The table find code function was simplified due to the change on how constants are represented. The first argument of the set token and set token code functions were changed from a standard shared token pointer reference to a straight token pointer so that they can be called from the parser (with a standard unique pointer), translator (with a standard shared pointer) or token member function (with just a pointer). This simply required calling the get access function of the unique or shared pointers.
The translator get operand function no longer sets the code for constants. The get expression and process internal function functions no longer need to look for and set the codes for constants (the later needs to clear the Integer Constant sub-code for functions taking both number argument types, specifically ABS, SGN and STR$). And the get token function now only needs to pass the data type to the parser. The token convert and table find code functions are used by the translator and will finalize constants not set by the parser once the final data type is known.
[branch table commit 3099f8850f]
Thursday, December 25, 2014
Double Identifier Problem
An existing problem was discovered when the parser was modified to not store the data type character of identifiers. The issue was with double identifiers when using the optional # data type character. The identifiers Variable and Variable# were incorrectly added to the dictionary as separate entries when they should have been the same entry.
The parser get identifier function was modified to not store the data type character in the token. This caused a problem when recreating double identifiers where the # character entered would disappear. Recreating all double identifiers with a # character was also not desirable. This was corrected by adding the Double sub-code to the token. A sub-code argument was added to the token constructor for identifiers. This sub-code is encoded into the program code so that the # character is recreated when it is entered.
The Double sub-code was only being used for constants. When the value of a constant is within the integer range, its data type is set to integer, and if a decimal point is present, the Double sub-code is set. The translator uses this sub-code to determine if a constant can be used as a double even though the data type is integer (see post from October 28 for details). This sub-code does not survive past the translator (not put into the program code).
A new string with data type access function was added to the token to add the data type character (#, % for integers, and $ for strings) to the token string returned. A # character is only added if the Double sub-code is set. This function replaced the string access function in the test token stream insert operator, tester print token, and several recreate functions.
A sub-code argument was added to the table entry operand text functions. The variable operand text functions were modified to add the data type character to the variable name. For double variables, the character is only added if the Double sub-code is set. An Ignore sub-code enumerator was added, and when passed to the operand text function, no data type character is added to the variable name. This option was needed for the program model decode function that uses the operand text function to set the string of the token (since tokens no longer store the data type character).
The value of the Double sub-code was changed so that its bit value was within the range of the sub-code bits (not necessary before since this sub-code was not used in the program code). The return type of program code instruction sub-code access function was changed to the Sub-Code enumeration type (from an integer). The expected encoder test results were updated, specifically the dictionaries output since the data type characters are no longer present in the entries.
[branch table commit e97057efca]
The parser get identifier function was modified to not store the data type character in the token. This caused a problem when recreating double identifiers where the # character entered would disappear. Recreating all double identifiers with a # character was also not desirable. This was corrected by adding the Double sub-code to the token. A sub-code argument was added to the token constructor for identifiers. This sub-code is encoded into the program code so that the # character is recreated when it is entered.
The Double sub-code was only being used for constants. When the value of a constant is within the integer range, its data type is set to integer, and if a decimal point is present, the Double sub-code is set. The translator uses this sub-code to determine if a constant can be used as a double even though the data type is integer (see post from October 28 for details). This sub-code does not survive past the translator (not put into the program code).
A new string with data type access function was added to the token to add the data type character (#, % for integers, and $ for strings) to the token string returned. A # character is only added if the Double sub-code is set. This function replaced the string access function in the test token stream insert operator, tester print token, and several recreate functions.
A sub-code argument was added to the table entry operand text functions. The variable operand text functions were modified to add the data type character to the variable name. For double variables, the character is only added if the Double sub-code is set. An Ignore sub-code enumerator was added, and when passed to the operand text function, no data type character is added to the variable name. This option was needed for the program model decode function that uses the operand text function to set the string of the token (since tokens no longer store the data type character).
The value of the Double sub-code was changed so that its bit value was within the range of the sub-code bits (not necessary before since this sub-code was not used in the program code). The return type of program code instruction sub-code access function was changed to the Sub-Code enumeration type (from an integer). The expected encoder test results were updated, specifically the dictionaries output since the data type characters are no longer present in the entries.
[branch table commit e97057efca]
Parser – Identifier Codes
The parser previously set the code for an identifier token only when the word was found in the table (command, operator or function). The codes for other identifiers were set in the translator: defined functions with no parentheses and variables (get operand); arrays, functions, and defined functions with parentheses (process parentheses tokens). This was changed to set all codes in the parser.
To do this in the parser, the parser needed to know if a reference operand was being requested. For now identifiers with no parentheses are set to variables, and with parentheses are set to arrays unless they start with an F (temporary check for testing). Defined functions are identifiers that start with an FN. Eventually the parser will need access to the program dictionaries to fully determine which code to assign to an identifier token.
The get identifier function was modified to set the code as described above for identifiers not found in the table. A reference argument was added, which was also added to the parser function operator. (The Reference enumeration was moved from the translator class to the main header file so that its enumerators are accessible.) The token constructor for codes and identifiers were combined to a single constructor with default arguments for the string and reference members.
For variables, the reference argument is used to determine if the code is a variable or a variable reference. Only the base code is set as the translator changed the code for the data type. In the case of a variable reference, the reference member of the token is not set (the translator did not previously set it either).
Several token type cases in the translator get operand function was modified. For defined functions with no parentheses, the token reference and code members no longer need to be set. For no parentheses tokens (variables), the code is still updated for the data type. The parser will do this once the new table model is implemented. For parentheses tokens (arrays), the token reference member no longer needs to be set.
The translator process parentheses token function no longer does the check for functions (temporarily identifiers starting with F), or set the code of the token. For determining an array (to set the expected expression types to integer for the subscripts), the Array code is checked for. This check will need to be modified when arrays are implemented since there will be different array codes for each data type, which will be set by the parser.
[branch table commit 69dff18e26]
To do this in the parser, the parser needed to know if a reference operand was being requested. For now identifiers with no parentheses are set to variables, and with parentheses are set to arrays unless they start with an F (temporary check for testing). Defined functions are identifiers that start with an FN. Eventually the parser will need access to the program dictionaries to fully determine which code to assign to an identifier token.
The get identifier function was modified to set the code as described above for identifiers not found in the table. A reference argument was added, which was also added to the parser function operator. (The Reference enumeration was moved from the translator class to the main header file so that its enumerators are accessible.) The token constructor for codes and identifiers were combined to a single constructor with default arguments for the string and reference members.
For variables, the reference argument is used to determine if the code is a variable or a variable reference. Only the base code is set as the translator changed the code for the data type. In the case of a variable reference, the reference member of the token is not set (the translator did not previously set it either).
Several token type cases in the translator get operand function was modified. For defined functions with no parentheses, the token reference and code members no longer need to be set. For no parentheses tokens (variables), the code is still updated for the data type. The parser will do this once the new table model is implemented. For parentheses tokens (arrays), the token reference member no longer needs to be set.
The translator process parentheses token function no longer does the check for functions (temporarily identifiers starting with F), or set the code of the token. For determining an array (to set the expected expression types to integer for the subscripts), the Array code is checked for. This check will need to be modified when arrays are implemented since there will be different array codes for each data type, which will be set by the parser.
[branch table commit 69dff18e26]
Saturday, November 8, 2014
Parser – Standard Input Stream
The parser functions have been modified to use a standard input string stream, so the input member could now be changed to the std::istringstream class, and the input position member removed. The current input position can be obtained directly from the input stream using the tellg member function. The skip white space function was removed since this can be done directly on the input stream (in other words, extract white space):
When using an input stream, care must be taken when using the tellg function to obtain the current input position. This function returns an EOF value (-1) once the input stream has been read past the end. So this function can't be used is a previous operation could have possibly read past the end.
There times when the input position must be reset (like when the second word of a possibly two-word command is not valid). The seekg function is used to the input position. However, this function does not work once the input stream has been read past the end. This is because the EOF flag is set. To clear this condition, the clear function needs to be called, so this call precedes all seekg function calls except one where an EOF cannot have occurred.
In the get string function, the characters read are counted so that the length of the string in the input is known when the token is constructed and returned (pairs of double quotes count as one character in the string, but take two characters in the input string, so must be counted as two). The ending input position cannot be used to determine the length in the input string because the position is not valid if the string constant is at the end of the line (see issue with tellg function above).
The constructor of the parser was changed to take a standard string input. Both callers were modified accordingly - the tester class already had a standard string, but the translator needs to convert from its QString to a standard string (until the translator is modified). Dependency on Qt has almost been removed from the parser except for one call to obtain the name for the REM command (which will be handled when the table is modified).
[branch parser commit 8e71a71fd5]
One outstanding item remains - the token string member is still a QString though its constructors have been modified to take standard string arguments. This will not a trivial change since many users of the token string still expect a QString. Therefore, this work will take place in a new development branch. This concludes work on the parser, so the parser branch was merged to the develop branch and deleted.
[branch develop merge commit 2cafb22a8e]
m_input >> std::ws;In several places, a temporary position or length integer variable is used to get the current position (tellg) or length (length) because these functions return a pos_type and size_type values, which are 64-bit integers. The token constructors and error structure only accept integers (32-bit). There is no reason to change the member variable types to 64-bit integers as there will never be input lines or strings that are long enough to require 64-bit integers.
When using an input stream, care must be taken when using the tellg function to obtain the current input position. This function returns an EOF value (-1) once the input stream has been read past the end. So this function can't be used is a previous operation could have possibly read past the end.
There times when the input position must be reset (like when the second word of a possibly two-word command is not valid). The seekg function is used to the input position. However, this function does not work once the input stream has been read past the end. This is because the EOF flag is set. To clear this condition, the clear function needs to be called, so this call precedes all seekg function calls except one where an EOF cannot have occurred.
In the get string function, the characters read are counted so that the length of the string in the input is known when the token is constructed and returned (pairs of double quotes count as one character in the string, but take two characters in the input string, so must be counted as two). The ending input position cannot be used to determine the length in the input string because the position is not valid if the string constant is at the end of the line (see issue with tellg function above).
The constructor of the parser was changed to take a standard string input. Both callers were modified accordingly - the tester class already had a standard string, but the translator needs to convert from its QString to a standard string (until the translator is modified). Dependency on Qt has almost been removed from the parser except for one call to obtain the name for the REM command (which will be handled when the table is modified).
[branch parser commit 8e71a71fd5]
One outstanding item remains - the token string member is still a QString though its constructors have been modified to take standard string arguments. This will not a trivial change since many users of the token string still expect a QString. Therefore, this work will take place in a new development branch. This concludes work on the parser, so the parser branch was merged to the develop branch and deleted.
[branch develop merge commit 2cafb22a8e]
Thursday, November 6, 2014
Parsing Identifiers – Standard Library
The get identifier function was changed to use a standard input stream (again using a temporary input string stream like the previous functions). This function used scan word support function to look for a word and was renamed more appropriately to get word. Instead of checking for a REM command first (because unlike other commands, a space is not required after the command), the get word function is called first to get a word. If no valid word is found, an empty token pointer is returned.
The first check on a valid word is if the word starts with letters in the REM command name using the std::equal function with the no case compare lambda function. Since the REM command name is still in the table as a QString, it is temporarily converted to standard string. For a remark, the input position is set to beginning of the string of the remark, and the word string is then replaced with the rest of the characters on the line, from which the token is created and returned.
An issue was discovered with the parsing of define function tokens (identifiers that start with "FN"). A valid defined function name should start with a letter, but there was no check for a letter or even a check if there were any characters after the "FN" so identifiers like FN and FN1 were incorrectly accepted as defined function tokens. Instead of rejecting these names as invalid defined function names, the decision was made to allow these names, and treat them as regular names (variables and arrays).
The get word support function was modified in the same way (by using a temporary input stream). It also returned three values, the position after the word found, the data type of the word and whether the word has a parentheses. Two of these were returned by passing references. The position is not needed since it will be obtained from the input stream, however, a string for the word is needed because it is read from the stream. A new Word structure was added to hold the word string, data type and parentheses flag, which is now returned. An empty word string indicates no valid word found.
To simplify the handling of two word commands in the get identifier function, the check of the second to make sure that it does not have a data type or parentheses was moved to the get word function. If the second word does, an empty word string is returned and the input stream is repositioned back to the beginning of the word. A word type argument was added with values first and second to enable this second word checking.
The get identifier function uses the two-word table search function, which was modified to take two standard strings. The token constructor for identifiers was modified to take a standard string argument, which is temporarily converted to a c-style string to initialize the QString token member. Several invalid defined function names were added to parser test #2 (identifiers) to verify these names are treated as plain identifiers (with and without parentheses).
[branch parser commit dbbd9fe054]
The first check on a valid word is if the word starts with letters in the REM command name using the std::equal function with the no case compare lambda function. Since the REM command name is still in the table as a QString, it is temporarily converted to standard string. For a remark, the input position is set to beginning of the string of the remark, and the word string is then replaced with the rest of the characters on the line, from which the token is created and returned.
An issue was discovered with the parsing of define function tokens (identifiers that start with "FN"). A valid defined function name should start with a letter, but there was no check for a letter or even a check if there were any characters after the "FN" so identifiers like FN and FN1 were incorrectly accepted as defined function tokens. Instead of rejecting these names as invalid defined function names, the decision was made to allow these names, and treat them as regular names (variables and arrays).
The get word support function was modified in the same way (by using a temporary input stream). It also returned three values, the position after the word found, the data type of the word and whether the word has a parentheses. Two of these were returned by passing references. The position is not needed since it will be obtained from the input stream, however, a string for the word is needed because it is read from the stream. A new Word structure was added to hold the word string, data type and parentheses flag, which is now returned. An empty word string indicates no valid word found.
To simplify the handling of two word commands in the get identifier function, the check of the second to make sure that it does not have a data type or parentheses was moved to the get word function. If the second word does, an empty word string is returned and the input stream is repositioned back to the beginning of the word. A word type argument was added with values first and second to enable this second word checking.
The get identifier function uses the two-word table search function, which was modified to take two standard strings. The token constructor for identifiers was modified to take a standard string argument, which is temporarily converted to a c-style string to initialize the QString token member. Several invalid defined function names were added to parser test #2 (identifiers) to verify these names are treated as plain identifiers (with and without parentheses).
[branch parser commit dbbd9fe054]
Sunday, November 2, 2014
Parsing Operators – Standard Library
The get operator function was changed to use a standard input stream (again using a temporary input string stream like the two previous functions). This function uses one of the table search functions, which was modified to take a standard string.
The table search function used the compare function from the QString class with the case insensitive option. There is no equivalent function in the standard string class. The std::equal function is used instead by passing a no case comparison lambda function. This is the same lambda function used in the Tester class, so this definition was moved to the main header file. Since the name in the table is still a QString, it is temporarily converted to a standard string. The std::equal function assumes the arguments are the same size, so the size of the strings are checked first.
The token constructor for codes was changed to take a standard string, which defaults to an empty string. For now, these are converted for the QString member variable by obtaining a c-style string from the standard string, which is implicitly converted. The only caller of this constructor using this argument is the new token table function, which was also modified to take a standard string. Callers of this function using the string argument were modified to pass a standard string.
[branch parser commit 27f06e8714]
The table search function used the compare function from the QString class with the case insensitive option. There is no equivalent function in the standard string class. The std::equal function is used instead by passing a no case comparison lambda function. This is the same lambda function used in the Tester class, so this definition was moved to the main header file. Since the name in the table is still a QString, it is temporarily converted to a standard string. The std::equal function assumes the arguments are the same size, so the size of the strings are checked first.
The token constructor for codes was changed to take a standard string, which defaults to an empty string. For now, these are converted for the QString member variable by obtaining a c-style string from the standard string, which is implicitly converted. The only caller of this constructor using this argument is the new token table function, which was also modified to take a standard string. Callers of this function using the string argument were modified to pass a standard string.
[branch parser commit 27f06e8714]
Parsing Strings – Standard Library
The get string function was changed to use a standard input stream (temporarily putting the input string from the current position into a temporary local input string stream of the same name as the member variable to simulate the final parser code). The looking at and the obtaining of current character was changed as previously described.
Instead of incrementing the local position variable for each character in the string constant, this variable is just set to the current input position. This will be changed to get the position within the input stream stream once the member variable is changed. The current input position is incremented for each character. After the change, the current input position member variable will not be needed.
[branch parser commit ab3b1c08f18]
Instead of incrementing the local position variable for each character in the string constant, this variable is just set to the current input position. This will be changed to get the position within the input stream stream once the member variable is changed. The current input position is incremented for each character. After the change, the current input position member variable will not be needed.
[branch parser commit ab3b1c08f18]
Parsing Numbers – Standard Library
When the parser is changed to use the standard library, instead of placing the input string into a string member variable, it will put into a standard input string stream from which the characters will be pulled from. A position into the input string will not need to be maintained during the processing of the line.
The get number function was the first to be changed to this model. Temporarily, the input string from the current position (a substring) is transferred into a temporary local input string stream of the same name as the member variable to simulate the final parser code. Two failed attempts were made to use standard library functions to parse and read numbers.
The first attempt used the stoi function to convert the number directly. The problem was that it doesn't report the specifics of the error when the conversion fails, throwing only a invalid argument or out-of-range exception. The type of error could be determined by a series of complex checks of the string. A working solution was mostly achieved with one remaining issue. When an out-of-range exception was thrown, there is no clue as to the length of the string that was processed (which is needed to properly highlight the error).
The second attempt used the extraction operator (>>) to get the number directly into a double or integer variable. Again detecting an error and determining the type of error was difficult (and was not actually achieved when this attempt was abandoned). The code again was also complicated.
These attempts were made to try to eliminate the involved (but working) algorithm already in place to parse numbers. The decision was made to use the current algorithm, and was modified to read from a standard input stream. The reading of the current character was changed to:
The integer and double token constructors were modified to take standard string arguments. For now, these are converted for the QString member variable by obtaining a c-style string from the standard string, which is implicitly converted. This is temporary until the token string member is changed to a standard string.
[branch parser commit 8513875e33]
The get number function was the first to be changed to this model. Temporarily, the input string from the current position (a substring) is transferred into a temporary local input string stream of the same name as the member variable to simulate the final parser code. Two failed attempts were made to use standard library functions to parse and read numbers.
The first attempt used the stoi function to convert the number directly. The problem was that it doesn't report the specifics of the error when the conversion fails, throwing only a invalid argument or out-of-range exception. The type of error could be determined by a series of complex checks of the string. A working solution was mostly achieved with one remaining issue. When an out-of-range exception was thrown, there is no clue as to the length of the string that was processed (which is needed to properly highlight the error).
The second attempt used the extraction operator (>>) to get the number directly into a double or integer variable. Again detecting an error and determining the type of error was difficult (and was not actually achieved when this attempt was abandoned). The code again was also complicated.
These attempts were made to try to eliminate the involved (but working) algorithm already in place to parse numbers. The decision was made to use the current algorithm, and was modified to read from a standard input stream. The reading of the current character was changed to:
m_input[pos] → m_input.peek()The original code incremented a local position when a character had been processed (will become part of the number string). This position increment was replaced with pulling a character from the input stream and appending it to the local number string:
pos++; → number.push_back(m_input.get());The various character type tests (to upper, is digit) were changed from the QChar functions to the standard ctype tests. Once a possible valid number was parsed into the local string, if itt didn't contain a decimal point or exponent, an attempt is made to convert it to an integer using the stoi function. If successful, an integer number token is created from the local string and returned. If an out-of-range exception is thrown or had a decimal point or exponent, an attempt is made to convert it to a double using the stod function. If successful, a double number token is created from the local string and returned. Another out-of-range exception results in a floating point out of range exception being thrown.
The integer and double token constructors were modified to take standard string arguments. For now, these are converted for the QString member variable by obtaining a c-style string from the standard string, which is implicitly converted. This is temporary until the token string member is changed to a standard string.
[branch parser commit 8513875e33]
Saturday, November 1, 2014
Parser – Number Error Corrections
Qt functions are currently being used to convert strings of numbers to a double or an integer in the get number routine. This routine will be changed to use STL functions. While investigating this, a few problems were discovered with how some of the number errors were being reported.
The "expected sign or digits for exponent in floating point constant" error was being reported even when the exponent sign was present. A new "expected digits for exponent in floating point constant" error for this situation. When an incorrectly formed number contained a single decimal point followed by the start of an exponent ('E'), the "expected digits in mantissa of floating point constant" error only pointed to the decimal point. The error was changed to point to both the decimal point and the 'E' character.
The translator was not reporting the "expected command" error correctly when there was a number error - the error was pointing to the number error which was either not at the beginning of the command or its length was not one. This occurred because the number error was not correctly reported as an unexpected token error when a reference was request (at the beginning of a statement).
This was corrected by adding a reference argument to the get token translator routine with a default of None. Only when this argument is None are number tokens allowed. When an unknown token error is returned from the parser, the reference argument is used to generate the appropriate expected error status. For the first token obtained from the get commands translator routine, this argument was set to All, which prevents number tokens (an unexpected token error is return for all number including number errors).
The get operands translator routine was modified to pass its reference argument directly to the get token call. Since get token now generates the appropriate error for references, this routine no longer needs to intercept the error to return the appropriate error or set the error length to one for references. The status is simply returned when the status is not Good. The LET translate routine handles reporting errors when neither a command nor a reference starts a line. The section handling errors was structured poorly and was rewritten.
Certain types of errors were reported differently as a result of these changes. Previously, the error for an incorrect statement like 34=A was reported as an "expected item for assignment" error pointing to the 34. Now the "expected command" error is reported pointing to only the first character of the number. Both errors are technically correct, and it would be difficult to report the previous error. The error was changed to "expected command or item for assignment" since both are applicable at the beginning of a statement.
The expected results for parser test #3 (numbers), translator tests #1 (assignments), #3 (more assignments), and #14 (parser errors) were updated for these changes. Some addition tests were added to translator test #14 for the new expecting digits for exponent error. Many of the translator tests results were also updated for the expected command message change.
[branch parser commit 20e46cc617]
The "expected sign or digits for exponent in floating point constant" error was being reported even when the exponent sign was present. A new "expected digits for exponent in floating point constant" error for this situation. When an incorrectly formed number contained a single decimal point followed by the start of an exponent ('E'), the "expected digits in mantissa of floating point constant" error only pointed to the decimal point. The error was changed to point to both the decimal point and the 'E' character.
The translator was not reporting the "expected command" error correctly when there was a number error - the error was pointing to the number error which was either not at the beginning of the command or its length was not one. This occurred because the number error was not correctly reported as an unexpected token error when a reference was request (at the beginning of a statement).
This was corrected by adding a reference argument to the get token translator routine with a default of None. Only when this argument is None are number tokens allowed. When an unknown token error is returned from the parser, the reference argument is used to generate the appropriate expected error status. For the first token obtained from the get commands translator routine, this argument was set to All, which prevents number tokens (an unexpected token error is return for all number including number errors).
The get operands translator routine was modified to pass its reference argument directly to the get token call. Since get token now generates the appropriate error for references, this routine no longer needs to intercept the error to return the appropriate error or set the error length to one for references. The status is simply returned when the status is not Good. The LET translate routine handles reporting errors when neither a command nor a reference starts a line. The section handling errors was structured poorly and was rewritten.
Certain types of errors were reported differently as a result of these changes. Previously, the error for an incorrect statement like 34=A was reported as an "expected item for assignment" error pointing to the 34. Now the "expected command" error is reported pointing to only the first character of the number. Both errors are technically correct, and it would be difficult to report the previous error. The error was changed to "expected command or item for assignment" since both are applicable at the beginning of a statement.
The expected results for parser test #3 (numbers), translator tests #1 (assignments), #3 (more assignments), and #14 (parser errors) were updated for these changes. Some addition tests were added to translator test #14 for the new expecting digits for exponent error. Many of the translator tests results were also updated for the expected command message change.
[branch parser commit 20e46cc617]
Thursday, October 30, 2014
Parser – Unique Pointers
The parser routines create a token held in a shared pointer upon return. The main function operator routine returns this shared pointer. The token is not actually being shared, just moved until it reaches the caller. There is no reason to use a shared pointer in the parser as a standard unique pointer is sufficient. The parser routines were changed to return a unique pointer. Another alias was added for a unique token pointer:
One other small unrelated change was made to the get identifier routine with the creation of the REM command token. This code was simplified as it was not necessary to copy the comment string from the input into a temporary string before creating the token. The string can be passed directly when the token is created. The new position can simply be set to the length of the input string. This was already done for the remark operator in the get operator routine.
[branch parser commit 336ad07bf8]
using TokenUniquePtr = std::unique_ptr<Token>;Unfortunately, there is no equivalent function for std::unique_ptr like the std::make_shared() function for std::shared_ptr (though one has been added for C++14) so unique pointers must be initialized using the new operator with the unique token pointer alias constructor:
return TokenUniquePtr{new Token {pos, len, type, dataType, m_input}};The callers of the parser operator function did not needed to be modified since there is a shared pointer constructor that takes unique pointer as an argument (the shared pointer takes ownership of unique pointer). The table new token function was also modified to return a unique token pointer.
One other small unrelated change was made to the get identifier routine with the creation of the REM command token. This code was simplified as it was not necessary to copy the comment string from the input into a temporary string before creating the token. The string can be passed directly when the token is created. The new position can simply be set to the length of the input string. This was already done for the remark operator in the get operator routine.
[branch parser commit 336ad07bf8]
Wednesday, October 29, 2014
Parser – Operator Tokens
The get operator routine was modified to create a new token upon returning when a valid token is found. If the first character is not the start of an operator, a default token pointer is returned. The existing table new token function is used to create the new token upon return. The flow of the function was cleaned up by checking for an invalid operator first, a remark operator next and finally for a two-character operator.
In the main function operator routine, the call to get string was changed like the other get function calls with the member token initialization was finally removed along with the token member.
[branch parser commit 632ce89f80]
In the main function operator routine, the call to get string was changed like the other get function calls with the member token initialization was finally removed along with the token member.
[branch parser commit 632ce89f80]
Parser – Constant String Tokens
The get string routine was modified to create a new token upon returning when a valid token is found. If the first character is not the start of a string constant (a double quote), a default token pointer is returned. A token constructor was added to support creating a string constant token, which in addition to the column and length takes the string constant without the surrounding double quotes.
This routine was changed from setting characters into the token string (by a length index counter that was not otherwise used) to simply appending the characters to a local string (since the token is not created until the return statement). The former will not work with standard strings. This local string is moved to the constructor, though this has no effect with a QString (copies if class doesn't support move), but will with standard strings.
The set string character token access function was removed since this routine was the only caller. In the main function operator routine, the call to get string was changed like the other get function calls with the member token initialization moved below this.
[branch parser commit 87e4ed4fb8]
This routine was changed from setting characters into the token string (by a length index counter that was not otherwise used) to simply appending the characters to a local string (since the token is not created until the return statement). The former will not work with standard strings. This local string is moved to the constructor, though this has no effect with a QString (copies if class doesn't support move), but will with standard strings.
The set string character token access function was removed since this routine was the only caller. In the main function operator routine, the call to get string was changed like the other get function calls with the member token initialization moved below this.
[branch parser commit 87e4ed4fb8]
Tuesday, October 28, 2014
Parser – Constant Number Tokens
The get number routine was next to be modified to create a new token upon returning when a valid token is found. The character parsing part of the routine was left intact except the two instances where no number is found were changed to return a default token pointer. The token creation lines at the end were replaced with return statements creating a token in a shared pointer. Two more constructors were added to the token class to these return statements.
The first, in addition to the column and length takes the string of the number and the integer value of the number, and automatically sets the type to constant and the data type to integer. The integer value member is initialized to the integer value, however, the double value member is set to the integer value in the body to do the conversion from integer to double (which can't be done with an initializer because the types are different).
The other constructor also takes the string of the number, the double value and a flag for whether a decimal point was present, sets the type to constant. The body checks if the double value is within the range of an integer, and if it is, the sets the data type to integer, and sets the double sub-code only if there was a decimal point. The translator uses this sub-code to determine if a constant can be used as a double even though the data type is integer (a hidden conversion from integer to double code is not needed). For values outside the integer range, the data type is set to double (indicating conversion to an integer is not possible).
The body of the second constructor was taken from the get number routine because this code primary sets token members (via access functions), and it seemed appropriate to do this within the token class. Since the body was not trivial, the constructor was put into the token source and not the header file. Another reason was that the C-style integer minimum and maximum constants were replaced with C++ standard numerical constants from the limits STL header file (no reason to burden source files including the token header file with another header file).
In the main function operator routine, the call to get number was changed like the call to get identifier with the member token initialization moved below this. It appears redundant to declare a if-scoped token pointer at each if statement, but if there was a single token pointer for the entire routine, it would first be initialized to a default value, then reinitialized at each if statement. The if-scoped variable is initialized directly with the return value of the get routine.
[branch parser commit d328c0a720]
The first, in addition to the column and length takes the string of the number and the integer value of the number, and automatically sets the type to constant and the data type to integer. The integer value member is initialized to the integer value, however, the double value member is set to the integer value in the body to do the conversion from integer to double (which can't be done with an initializer because the types are different).
The other constructor also takes the string of the number, the double value and a flag for whether a decimal point was present, sets the type to constant. The body checks if the double value is within the range of an integer, and if it is, the sets the data type to integer, and sets the double sub-code only if there was a decimal point. The translator uses this sub-code to determine if a constant can be used as a double even though the data type is integer (a hidden conversion from integer to double code is not needed). For values outside the integer range, the data type is set to double (indicating conversion to an integer is not possible).
The body of the second constructor was taken from the get number routine because this code primary sets token members (via access functions), and it seemed appropriate to do this within the token class. Since the body was not trivial, the constructor was put into the token source and not the header file. Another reason was that the C-style integer minimum and maximum constants were replaced with C++ standard numerical constants from the limits STL header file (no reason to burden source files including the token header file with another header file).
In the main function operator routine, the call to get number was changed like the call to get identifier with the member token initialization moved below this. It appears redundant to declare a if-scoped token pointer at each if statement, but if there was a single token pointer for the entire routine, it would first be initialized to a default value, then reinitialized at each if statement. The if-scoped variable is initialized directly with the return value of the get routine.
[branch parser commit d328c0a720]
Monday, October 27, 2014
Parser – Create Token As Needed
The parser will be modified to create a token only when a valid token is found in the input string and is returned directly. This means that the token will be created on a return statement, which is automatically moved to the caller since the created token (in a shared pointer) is temporary and going out of scope.
Once all the get routines in the parser are changed, it will no longer be necessary to have a member variable to hold the token and there will be no worry of a token being left allocated for an error. Right now the returned token will be an shared pointer, though is not necessary. The return pointer will be changed to a unique pointer, which can be assigned to a shared pointer.
The get identifier routine was the first to modified. Most of the token creation lines were replaced with returns statements creating a token in a shared pointer:
Once all the locations where in the get identifier routine were replaced, it could be seen that the code was repetitive, so the whole function was reorganized and reduced. If no valid identifier token is found, a default token pointer is returned, which the caller can check as a boolean.
The main function operator routine was modified to support this partial transition. When the end-of-line is reached, a new token is created and returned (using the new token table function). The get identifier routine is called in an if statement by itself receiving the return value in a if-scoped variable, which is returned if set:
[branch parser commit e34fbccacc]
Once all the get routines in the parser are changed, it will no longer be necessary to have a member variable to hold the token and there will be no worry of a token being left allocated for an error. Right now the returned token will be an shared pointer, though is not necessary. The return pointer will be changed to a unique pointer, which can be assigned to a shared pointer.
The get identifier routine was the first to modified. Most of the token creation lines were replaced with returns statements creating a token in a shared pointer:
return std::make_shared<Token>(pos, len, type, dataType, m_input);To support this, two new constructors were added to the token class. One that in addition to the column, length, type and data types values takes the input string (from which a string is created using the column and length values) as shown above. The other constructor taking a code and optional string, which is used by a new token function added to the table class that uses the table to set the type and data type values of the new token.
Once all the locations where in the get identifier routine were replaced, it could be seen that the code was repetitive, so the whole function was reorganized and reduced. If no valid identifier token is found, a default token pointer is returned, which the caller can check as a boolean.
The main function operator routine was modified to support this partial transition. When the end-of-line is reached, a new token is created and returned (using the new token table function). The get identifier routine is called in an if statement by itself receiving the return value in a if-scoped variable, which is returned if set:
if (TokenPtr token = getIdentifier()) {For now, the current creation of a new token was moved to after the statements above. It will continue being moved as each get routine is changed until all have been changed at which time it will be removed along with token member pointer.
return token;
}
[branch parser commit e34fbccacc]
Sunday, October 26, 2014
Parser – STL Preparations
The changes required to make the parser routines use the STL are going to be extensive, but an attempt will be made to break the changes into smaller incremental changes. Since the parser routines make use of various table functions, it will be necessary to modify the table entries and its functions to use STL. Some preparatory changes were made.
The table entries are divided into several groups for searching, which includes plain word, parentheses words, data type words and symbols. The parser utilizes these groups when searching if strings have a code. The data type words section was empty and upon consideration it was concluded that this group is not needed. This group may have been originally conceived for internal functions that don't have arguments (for example, a DATE$ function). However, these internal functions can go into the plain word group (the RND no argument function is already in this group). This group type along with its bracketing entries were removed.
[branch parser commit 0043105154]
The issue of the token being left allocated when the parser throws an exception could be resolved by not creating a token until a valid token is found. Used of the token member before an exception is thrown were examined and the only use was in the creation of the error exception. The only token members used were its column and length. The column was always the same as the current input position (except one instance) and the length was always 1.
All of the throw statements were modified to not use the token instead using the current input position with a length of 1. One case used a length of 2 (2 was previously used). For the "floating point constant is out of range" error, the statement to set the new input position was moved to after an error is thrown.
For the "expected sign or digits for exponent in floating point constant" error, two columns were reported, the column at the beginning of the number (for operator state) and the alternate column at the beginning of the error (for operand state). A number token is no longer accepted when invalid, so only the alternate column was being used. This mechanism was not needed, and for this error, the position of the error only is reported. This mechanism was also removed from the tester print error function.
[branch parser commit e92da57ef1]
The table entries are divided into several groups for searching, which includes plain word, parentheses words, data type words and symbols. The parser utilizes these groups when searching if strings have a code. The data type words section was empty and upon consideration it was concluded that this group is not needed. This group may have been originally conceived for internal functions that don't have arguments (for example, a DATE$ function). However, these internal functions can go into the plain word group (the RND no argument function is already in this group). This group type along with its bracketing entries were removed.
[branch parser commit 0043105154]
The issue of the token being left allocated when the parser throws an exception could be resolved by not creating a token until a valid token is found. Used of the token member before an exception is thrown were examined and the only use was in the creation of the error exception. The only token members used were its column and length. The column was always the same as the current input position (except one instance) and the length was always 1.
All of the throw statements were modified to not use the token instead using the current input position with a length of 1. One case used a length of 2 (2 was previously used). For the "floating point constant is out of range" error, the statement to set the new input position was moved to after an error is thrown.
For the "expected sign or digits for exponent in floating point constant" error, two columns were reported, the column at the beginning of the number (for operator state) and the alternate column at the beginning of the error (for operand state). A number token is no longer accepted when invalid, so only the alternate column was being used. This mechanism was not needed, and for this error, the position of the error only is reported. This mechanism was also removed from the tester print error function.
[branch parser commit e92da57ef1]
Subscribe to:
Posts (Atom)