The token equal operator function uses case insensitive string comparisons that need to be replaced with standard string equivalents. There is no direct equivalent, so the solution used so far was to use the std::equal function passing a case insensitive character comparison lambda function. Since the std::equal function assumes equal length containers, the size of the strings need to be compared first to make sure the strings are the same length (or at least the primary string is not less than the secondary string when doing a string begins with comparison).
This pattern of comparing the string lengths before comparing strings is repeating, therefore two in line functions were added to do this. The first, named no case string equal, checks if the string are equal and the second, named no case string begins with, checks if the length of the primary string is greater than or equal to the string being compared. The lambda function was renamed to no case character equal for consistency. The lambda function was moved from the main header file to the utility header file along with the two new in line functions.
[branch token commit 1dde64fb7b]
Saturday, November 8, 2014
Parser – Standard Input Stream
The parser functions have been modified to use a standard input string stream, so the input member could now be changed to the std::istringstream class, and the input position member removed. The current input position can be obtained directly from the input stream using the tellg member function. The skip white space function was removed since this can be done directly on the input stream (in other words, extract white space):
When using an input stream, care must be taken when using the tellg function to obtain the current input position. This function returns an EOF value (-1) once the input stream has been read past the end. So this function can't be used is a previous operation could have possibly read past the end.
There times when the input position must be reset (like when the second word of a possibly two-word command is not valid). The seekg function is used to the input position. However, this function does not work once the input stream has been read past the end. This is because the EOF flag is set. To clear this condition, the clear function needs to be called, so this call precedes all seekg function calls except one where an EOF cannot have occurred.
In the get string function, the characters read are counted so that the length of the string in the input is known when the token is constructed and returned (pairs of double quotes count as one character in the string, but take two characters in the input string, so must be counted as two). The ending input position cannot be used to determine the length in the input string because the position is not valid if the string constant is at the end of the line (see issue with tellg function above).
The constructor of the parser was changed to take a standard string input. Both callers were modified accordingly - the tester class already had a standard string, but the translator needs to convert from its QString to a standard string (until the translator is modified). Dependency on Qt has almost been removed from the parser except for one call to obtain the name for the REM command (which will be handled when the table is modified).
[branch parser commit 8e71a71fd5]
One outstanding item remains - the token string member is still a QString though its constructors have been modified to take standard string arguments. This will not a trivial change since many users of the token string still expect a QString. Therefore, this work will take place in a new development branch. This concludes work on the parser, so the parser branch was merged to the develop branch and deleted.
[branch develop merge commit 2cafb22a8e]
m_input >> std::ws;In several places, a temporary position or length integer variable is used to get the current position (tellg) or length (length) because these functions return a pos_type and size_type values, which are 64-bit integers. The token constructors and error structure only accept integers (32-bit). There is no reason to change the member variable types to 64-bit integers as there will never be input lines or strings that are long enough to require 64-bit integers.
When using an input stream, care must be taken when using the tellg function to obtain the current input position. This function returns an EOF value (-1) once the input stream has been read past the end. So this function can't be used is a previous operation could have possibly read past the end.
There times when the input position must be reset (like when the second word of a possibly two-word command is not valid). The seekg function is used to the input position. However, this function does not work once the input stream has been read past the end. This is because the EOF flag is set. To clear this condition, the clear function needs to be called, so this call precedes all seekg function calls except one where an EOF cannot have occurred.
In the get string function, the characters read are counted so that the length of the string in the input is known when the token is constructed and returned (pairs of double quotes count as one character in the string, but take two characters in the input string, so must be counted as two). The ending input position cannot be used to determine the length in the input string because the position is not valid if the string constant is at the end of the line (see issue with tellg function above).
The constructor of the parser was changed to take a standard string input. Both callers were modified accordingly - the tester class already had a standard string, but the translator needs to convert from its QString to a standard string (until the translator is modified). Dependency on Qt has almost been removed from the parser except for one call to obtain the name for the REM command (which will be handled when the table is modified).
[branch parser commit 8e71a71fd5]
One outstanding item remains - the token string member is still a QString though its constructors have been modified to take standard string arguments. This will not a trivial change since many users of the token string still expect a QString. Therefore, this work will take place in a new development branch. This concludes work on the parser, so the parser branch was merged to the develop branch and deleted.
[branch develop merge commit 2cafb22a8e]
Thursday, November 6, 2014
Parsing Identifiers – Standard Library
The get identifier function was changed to use a standard input stream (again using a temporary input string stream like the previous functions). This function used scan word support function to look for a word and was renamed more appropriately to get word. Instead of checking for a REM command first (because unlike other commands, a space is not required after the command), the get word function is called first to get a word. If no valid word is found, an empty token pointer is returned.
The first check on a valid word is if the word starts with letters in the REM command name using the std::equal function with the no case compare lambda function. Since the REM command name is still in the table as a QString, it is temporarily converted to standard string. For a remark, the input position is set to beginning of the string of the remark, and the word string is then replaced with the rest of the characters on the line, from which the token is created and returned.
An issue was discovered with the parsing of define function tokens (identifiers that start with "FN"). A valid defined function name should start with a letter, but there was no check for a letter or even a check if there were any characters after the "FN" so identifiers like FN and FN1 were incorrectly accepted as defined function tokens. Instead of rejecting these names as invalid defined function names, the decision was made to allow these names, and treat them as regular names (variables and arrays).
The get word support function was modified in the same way (by using a temporary input stream). It also returned three values, the position after the word found, the data type of the word and whether the word has a parentheses. Two of these were returned by passing references. The position is not needed since it will be obtained from the input stream, however, a string for the word is needed because it is read from the stream. A new Word structure was added to hold the word string, data type and parentheses flag, which is now returned. An empty word string indicates no valid word found.
To simplify the handling of two word commands in the get identifier function, the check of the second to make sure that it does not have a data type or parentheses was moved to the get word function. If the second word does, an empty word string is returned and the input stream is repositioned back to the beginning of the word. A word type argument was added with values first and second to enable this second word checking.
The get identifier function uses the two-word table search function, which was modified to take two standard strings. The token constructor for identifiers was modified to take a standard string argument, which is temporarily converted to a c-style string to initialize the QString token member. Several invalid defined function names were added to parser test #2 (identifiers) to verify these names are treated as plain identifiers (with and without parentheses).
[branch parser commit dbbd9fe054]
The first check on a valid word is if the word starts with letters in the REM command name using the std::equal function with the no case compare lambda function. Since the REM command name is still in the table as a QString, it is temporarily converted to standard string. For a remark, the input position is set to beginning of the string of the remark, and the word string is then replaced with the rest of the characters on the line, from which the token is created and returned.
An issue was discovered with the parsing of define function tokens (identifiers that start with "FN"). A valid defined function name should start with a letter, but there was no check for a letter or even a check if there were any characters after the "FN" so identifiers like FN and FN1 were incorrectly accepted as defined function tokens. Instead of rejecting these names as invalid defined function names, the decision was made to allow these names, and treat them as regular names (variables and arrays).
The get word support function was modified in the same way (by using a temporary input stream). It also returned three values, the position after the word found, the data type of the word and whether the word has a parentheses. Two of these were returned by passing references. The position is not needed since it will be obtained from the input stream, however, a string for the word is needed because it is read from the stream. A new Word structure was added to hold the word string, data type and parentheses flag, which is now returned. An empty word string indicates no valid word found.
To simplify the handling of two word commands in the get identifier function, the check of the second to make sure that it does not have a data type or parentheses was moved to the get word function. If the second word does, an empty word string is returned and the input stream is repositioned back to the beginning of the word. A word type argument was added with values first and second to enable this second word checking.
The get identifier function uses the two-word table search function, which was modified to take two standard strings. The token constructor for identifiers was modified to take a standard string argument, which is temporarily converted to a c-style string to initialize the QString token member. Several invalid defined function names were added to parser test #2 (identifiers) to verify these names are treated as plain identifiers (with and without parentheses).
[branch parser commit dbbd9fe054]
Sunday, November 2, 2014
Parsing Operators – Standard Library
The get operator function was changed to use a standard input stream (again using a temporary input string stream like the two previous functions). This function uses one of the table search functions, which was modified to take a standard string.
The table search function used the compare function from the QString class with the case insensitive option. There is no equivalent function in the standard string class. The std::equal function is used instead by passing a no case comparison lambda function. This is the same lambda function used in the Tester class, so this definition was moved to the main header file. Since the name in the table is still a QString, it is temporarily converted to a standard string. The std::equal function assumes the arguments are the same size, so the size of the strings are checked first.
The token constructor for codes was changed to take a standard string, which defaults to an empty string. For now, these are converted for the QString member variable by obtaining a c-style string from the standard string, which is implicitly converted. The only caller of this constructor using this argument is the new token table function, which was also modified to take a standard string. Callers of this function using the string argument were modified to pass a standard string.
[branch parser commit 27f06e8714]
The table search function used the compare function from the QString class with the case insensitive option. There is no equivalent function in the standard string class. The std::equal function is used instead by passing a no case comparison lambda function. This is the same lambda function used in the Tester class, so this definition was moved to the main header file. Since the name in the table is still a QString, it is temporarily converted to a standard string. The std::equal function assumes the arguments are the same size, so the size of the strings are checked first.
The token constructor for codes was changed to take a standard string, which defaults to an empty string. For now, these are converted for the QString member variable by obtaining a c-style string from the standard string, which is implicitly converted. The only caller of this constructor using this argument is the new token table function, which was also modified to take a standard string. Callers of this function using the string argument were modified to pass a standard string.
[branch parser commit 27f06e8714]
Parsing Strings – Standard Library
The get string function was changed to use a standard input stream (temporarily putting the input string from the current position into a temporary local input string stream of the same name as the member variable to simulate the final parser code). The looking at and the obtaining of current character was changed as previously described.
Instead of incrementing the local position variable for each character in the string constant, this variable is just set to the current input position. This will be changed to get the position within the input stream stream once the member variable is changed. The current input position is incremented for each character. After the change, the current input position member variable will not be needed.
[branch parser commit ab3b1c08f18]
Instead of incrementing the local position variable for each character in the string constant, this variable is just set to the current input position. This will be changed to get the position within the input stream stream once the member variable is changed. The current input position is incremented for each character. After the change, the current input position member variable will not be needed.
[branch parser commit ab3b1c08f18]
Parsing Numbers – Standard Library
When the parser is changed to use the standard library, instead of placing the input string into a string member variable, it will put into a standard input string stream from which the characters will be pulled from. A position into the input string will not need to be maintained during the processing of the line.
The get number function was the first to be changed to this model. Temporarily, the input string from the current position (a substring) is transferred into a temporary local input string stream of the same name as the member variable to simulate the final parser code. Two failed attempts were made to use standard library functions to parse and read numbers.
The first attempt used the stoi function to convert the number directly. The problem was that it doesn't report the specifics of the error when the conversion fails, throwing only a invalid argument or out-of-range exception. The type of error could be determined by a series of complex checks of the string. A working solution was mostly achieved with one remaining issue. When an out-of-range exception was thrown, there is no clue as to the length of the string that was processed (which is needed to properly highlight the error).
The second attempt used the extraction operator (>>) to get the number directly into a double or integer variable. Again detecting an error and determining the type of error was difficult (and was not actually achieved when this attempt was abandoned). The code again was also complicated.
These attempts were made to try to eliminate the involved (but working) algorithm already in place to parse numbers. The decision was made to use the current algorithm, and was modified to read from a standard input stream. The reading of the current character was changed to:
The integer and double token constructors were modified to take standard string arguments. For now, these are converted for the QString member variable by obtaining a c-style string from the standard string, which is implicitly converted. This is temporary until the token string member is changed to a standard string.
[branch parser commit 8513875e33]
The get number function was the first to be changed to this model. Temporarily, the input string from the current position (a substring) is transferred into a temporary local input string stream of the same name as the member variable to simulate the final parser code. Two failed attempts were made to use standard library functions to parse and read numbers.
The first attempt used the stoi function to convert the number directly. The problem was that it doesn't report the specifics of the error when the conversion fails, throwing only a invalid argument or out-of-range exception. The type of error could be determined by a series of complex checks of the string. A working solution was mostly achieved with one remaining issue. When an out-of-range exception was thrown, there is no clue as to the length of the string that was processed (which is needed to properly highlight the error).
The second attempt used the extraction operator (>>) to get the number directly into a double or integer variable. Again detecting an error and determining the type of error was difficult (and was not actually achieved when this attempt was abandoned). The code again was also complicated.
These attempts were made to try to eliminate the involved (but working) algorithm already in place to parse numbers. The decision was made to use the current algorithm, and was modified to read from a standard input stream. The reading of the current character was changed to:
m_input[pos] → m_input.peek()The original code incremented a local position when a character had been processed (will become part of the number string). This position increment was replaced with pulling a character from the input stream and appending it to the local number string:
pos++; → number.push_back(m_input.get());The various character type tests (to upper, is digit) were changed from the QChar functions to the standard ctype tests. Once a possible valid number was parsed into the local string, if itt didn't contain a decimal point or exponent, an attempt is made to convert it to an integer using the stoi function. If successful, an integer number token is created from the local string and returned. If an out-of-range exception is thrown or had a decimal point or exponent, an attempt is made to convert it to a double using the stod function. If successful, a double number token is created from the local string and returned. Another out-of-range exception results in a floating point out of range exception being thrown.
The integer and double token constructors were modified to take standard string arguments. For now, these are converted for the QString member variable by obtaining a c-style string from the standard string, which is implicitly converted. This is temporary until the token string member is changed to a standard string.
[branch parser commit 8513875e33]
Saturday, November 1, 2014
Parser – Number Error Corrections
Qt functions are currently being used to convert strings of numbers to a double or an integer in the get number routine. This routine will be changed to use STL functions. While investigating this, a few problems were discovered with how some of the number errors were being reported.
The "expected sign or digits for exponent in floating point constant" error was being reported even when the exponent sign was present. A new "expected digits for exponent in floating point constant" error for this situation. When an incorrectly formed number contained a single decimal point followed by the start of an exponent ('E'), the "expected digits in mantissa of floating point constant" error only pointed to the decimal point. The error was changed to point to both the decimal point and the 'E' character.
The translator was not reporting the "expected command" error correctly when there was a number error - the error was pointing to the number error which was either not at the beginning of the command or its length was not one. This occurred because the number error was not correctly reported as an unexpected token error when a reference was request (at the beginning of a statement).
This was corrected by adding a reference argument to the get token translator routine with a default of None. Only when this argument is None are number tokens allowed. When an unknown token error is returned from the parser, the reference argument is used to generate the appropriate expected error status. For the first token obtained from the get commands translator routine, this argument was set to All, which prevents number tokens (an unexpected token error is return for all number including number errors).
The get operands translator routine was modified to pass its reference argument directly to the get token call. Since get token now generates the appropriate error for references, this routine no longer needs to intercept the error to return the appropriate error or set the error length to one for references. The status is simply returned when the status is not Good. The LET translate routine handles reporting errors when neither a command nor a reference starts a line. The section handling errors was structured poorly and was rewritten.
Certain types of errors were reported differently as a result of these changes. Previously, the error for an incorrect statement like 34=A was reported as an "expected item for assignment" error pointing to the 34. Now the "expected command" error is reported pointing to only the first character of the number. Both errors are technically correct, and it would be difficult to report the previous error. The error was changed to "expected command or item for assignment" since both are applicable at the beginning of a statement.
The expected results for parser test #3 (numbers), translator tests #1 (assignments), #3 (more assignments), and #14 (parser errors) were updated for these changes. Some addition tests were added to translator test #14 for the new expecting digits for exponent error. Many of the translator tests results were also updated for the expected command message change.
[branch parser commit 20e46cc617]
The "expected sign or digits for exponent in floating point constant" error was being reported even when the exponent sign was present. A new "expected digits for exponent in floating point constant" error for this situation. When an incorrectly formed number contained a single decimal point followed by the start of an exponent ('E'), the "expected digits in mantissa of floating point constant" error only pointed to the decimal point. The error was changed to point to both the decimal point and the 'E' character.
The translator was not reporting the "expected command" error correctly when there was a number error - the error was pointing to the number error which was either not at the beginning of the command or its length was not one. This occurred because the number error was not correctly reported as an unexpected token error when a reference was request (at the beginning of a statement).
This was corrected by adding a reference argument to the get token translator routine with a default of None. Only when this argument is None are number tokens allowed. When an unknown token error is returned from the parser, the reference argument is used to generate the appropriate expected error status. For the first token obtained from the get commands translator routine, this argument was set to All, which prevents number tokens (an unexpected token error is return for all number including number errors).
The get operands translator routine was modified to pass its reference argument directly to the get token call. Since get token now generates the appropriate error for references, this routine no longer needs to intercept the error to return the appropriate error or set the error length to one for references. The status is simply returned when the status is not Good. The LET translate routine handles reporting errors when neither a command nor a reference starts a line. The section handling errors was structured poorly and was rewritten.
Certain types of errors were reported differently as a result of these changes. Previously, the error for an incorrect statement like 34=A was reported as an "expected item for assignment" error pointing to the 34. Now the "expected command" error is reported pointing to only the first character of the number. Both errors are technically correct, and it would be difficult to report the previous error. The error was changed to "expected command or item for assignment" since both are applicable at the beginning of a statement.
The expected results for parser test #3 (numbers), translator tests #1 (assignments), #3 (more assignments), and #14 (parser errors) were updated for these changes. Some addition tests were added to translator test #14 for the new expecting digits for exponent error. Many of the translator tests results were also updated for the expected command message change.
[branch parser commit 20e46cc617]
Thursday, October 30, 2014
Parser – Unique Pointers
The parser routines create a token held in a shared pointer upon return. The main function operator routine returns this shared pointer. The token is not actually being shared, just moved until it reaches the caller. There is no reason to use a shared pointer in the parser as a standard unique pointer is sufficient. The parser routines were changed to return a unique pointer. Another alias was added for a unique token pointer:
One other small unrelated change was made to the get identifier routine with the creation of the REM command token. This code was simplified as it was not necessary to copy the comment string from the input into a temporary string before creating the token. The string can be passed directly when the token is created. The new position can simply be set to the length of the input string. This was already done for the remark operator in the get operator routine.
[branch parser commit 336ad07bf8]
using TokenUniquePtr = std::unique_ptr<Token>;Unfortunately, there is no equivalent function for std::unique_ptr like the std::make_shared() function for std::shared_ptr (though one has been added for C++14) so unique pointers must be initialized using the new operator with the unique token pointer alias constructor:
return TokenUniquePtr{new Token {pos, len, type, dataType, m_input}};The callers of the parser operator function did not needed to be modified since there is a shared pointer constructor that takes unique pointer as an argument (the shared pointer takes ownership of unique pointer). The table new token function was also modified to return a unique token pointer.
One other small unrelated change was made to the get identifier routine with the creation of the REM command token. This code was simplified as it was not necessary to copy the comment string from the input into a temporary string before creating the token. The string can be passed directly when the token is created. The new position can simply be set to the length of the input string. This was already done for the remark operator in the get operator routine.
[branch parser commit 336ad07bf8]
Wednesday, October 29, 2014
Parser – Operator Tokens
The get operator routine was modified to create a new token upon returning when a valid token is found. If the first character is not the start of an operator, a default token pointer is returned. The existing table new token function is used to create the new token upon return. The flow of the function was cleaned up by checking for an invalid operator first, a remark operator next and finally for a two-character operator.
In the main function operator routine, the call to get string was changed like the other get function calls with the member token initialization was finally removed along with the token member.
[branch parser commit 632ce89f80]
In the main function operator routine, the call to get string was changed like the other get function calls with the member token initialization was finally removed along with the token member.
[branch parser commit 632ce89f80]
Parser – Constant String Tokens
The get string routine was modified to create a new token upon returning when a valid token is found. If the first character is not the start of a string constant (a double quote), a default token pointer is returned. A token constructor was added to support creating a string constant token, which in addition to the column and length takes the string constant without the surrounding double quotes.
This routine was changed from setting characters into the token string (by a length index counter that was not otherwise used) to simply appending the characters to a local string (since the token is not created until the return statement). The former will not work with standard strings. This local string is moved to the constructor, though this has no effect with a QString (copies if class doesn't support move), but will with standard strings.
The set string character token access function was removed since this routine was the only caller. In the main function operator routine, the call to get string was changed like the other get function calls with the member token initialization moved below this.
[branch parser commit 87e4ed4fb8]
This routine was changed from setting characters into the token string (by a length index counter that was not otherwise used) to simply appending the characters to a local string (since the token is not created until the return statement). The former will not work with standard strings. This local string is moved to the constructor, though this has no effect with a QString (copies if class doesn't support move), but will with standard strings.
The set string character token access function was removed since this routine was the only caller. In the main function operator routine, the call to get string was changed like the other get function calls with the member token initialization moved below this.
[branch parser commit 87e4ed4fb8]
Tuesday, October 28, 2014
Parser – Constant Number Tokens
The get number routine was next to be modified to create a new token upon returning when a valid token is found. The character parsing part of the routine was left intact except the two instances where no number is found were changed to return a default token pointer. The token creation lines at the end were replaced with return statements creating a token in a shared pointer. Two more constructors were added to the token class to these return statements.
The first, in addition to the column and length takes the string of the number and the integer value of the number, and automatically sets the type to constant and the data type to integer. The integer value member is initialized to the integer value, however, the double value member is set to the integer value in the body to do the conversion from integer to double (which can't be done with an initializer because the types are different).
The other constructor also takes the string of the number, the double value and a flag for whether a decimal point was present, sets the type to constant. The body checks if the double value is within the range of an integer, and if it is, the sets the data type to integer, and sets the double sub-code only if there was a decimal point. The translator uses this sub-code to determine if a constant can be used as a double even though the data type is integer (a hidden conversion from integer to double code is not needed). For values outside the integer range, the data type is set to double (indicating conversion to an integer is not possible).
The body of the second constructor was taken from the get number routine because this code primary sets token members (via access functions), and it seemed appropriate to do this within the token class. Since the body was not trivial, the constructor was put into the token source and not the header file. Another reason was that the C-style integer minimum and maximum constants were replaced with C++ standard numerical constants from the limits STL header file (no reason to burden source files including the token header file with another header file).
In the main function operator routine, the call to get number was changed like the call to get identifier with the member token initialization moved below this. It appears redundant to declare a if-scoped token pointer at each if statement, but if there was a single token pointer for the entire routine, it would first be initialized to a default value, then reinitialized at each if statement. The if-scoped variable is initialized directly with the return value of the get routine.
[branch parser commit d328c0a720]
The first, in addition to the column and length takes the string of the number and the integer value of the number, and automatically sets the type to constant and the data type to integer. The integer value member is initialized to the integer value, however, the double value member is set to the integer value in the body to do the conversion from integer to double (which can't be done with an initializer because the types are different).
The other constructor also takes the string of the number, the double value and a flag for whether a decimal point was present, sets the type to constant. The body checks if the double value is within the range of an integer, and if it is, the sets the data type to integer, and sets the double sub-code only if there was a decimal point. The translator uses this sub-code to determine if a constant can be used as a double even though the data type is integer (a hidden conversion from integer to double code is not needed). For values outside the integer range, the data type is set to double (indicating conversion to an integer is not possible).
The body of the second constructor was taken from the get number routine because this code primary sets token members (via access functions), and it seemed appropriate to do this within the token class. Since the body was not trivial, the constructor was put into the token source and not the header file. Another reason was that the C-style integer minimum and maximum constants were replaced with C++ standard numerical constants from the limits STL header file (no reason to burden source files including the token header file with another header file).
In the main function operator routine, the call to get number was changed like the call to get identifier with the member token initialization moved below this. It appears redundant to declare a if-scoped token pointer at each if statement, but if there was a single token pointer for the entire routine, it would first be initialized to a default value, then reinitialized at each if statement. The if-scoped variable is initialized directly with the return value of the get routine.
[branch parser commit d328c0a720]
Monday, October 27, 2014
Parser – Create Token As Needed
The parser will be modified to create a token only when a valid token is found in the input string and is returned directly. This means that the token will be created on a return statement, which is automatically moved to the caller since the created token (in a shared pointer) is temporary and going out of scope.
Once all the get routines in the parser are changed, it will no longer be necessary to have a member variable to hold the token and there will be no worry of a token being left allocated for an error. Right now the returned token will be an shared pointer, though is not necessary. The return pointer will be changed to a unique pointer, which can be assigned to a shared pointer.
The get identifier routine was the first to modified. Most of the token creation lines were replaced with returns statements creating a token in a shared pointer:
Once all the locations where in the get identifier routine were replaced, it could be seen that the code was repetitive, so the whole function was reorganized and reduced. If no valid identifier token is found, a default token pointer is returned, which the caller can check as a boolean.
The main function operator routine was modified to support this partial transition. When the end-of-line is reached, a new token is created and returned (using the new token table function). The get identifier routine is called in an if statement by itself receiving the return value in a if-scoped variable, which is returned if set:
[branch parser commit e34fbccacc]
Once all the get routines in the parser are changed, it will no longer be necessary to have a member variable to hold the token and there will be no worry of a token being left allocated for an error. Right now the returned token will be an shared pointer, though is not necessary. The return pointer will be changed to a unique pointer, which can be assigned to a shared pointer.
The get identifier routine was the first to modified. Most of the token creation lines were replaced with returns statements creating a token in a shared pointer:
return std::make_shared<Token>(pos, len, type, dataType, m_input);To support this, two new constructors were added to the token class. One that in addition to the column, length, type and data types values takes the input string (from which a string is created using the column and length values) as shown above. The other constructor taking a code and optional string, which is used by a new token function added to the table class that uses the table to set the type and data type values of the new token.
Once all the locations where in the get identifier routine were replaced, it could be seen that the code was repetitive, so the whole function was reorganized and reduced. If no valid identifier token is found, a default token pointer is returned, which the caller can check as a boolean.
The main function operator routine was modified to support this partial transition. When the end-of-line is reached, a new token is created and returned (using the new token table function). The get identifier routine is called in an if statement by itself receiving the return value in a if-scoped variable, which is returned if set:
if (TokenPtr token = getIdentifier()) {For now, the current creation of a new token was moved to after the statements above. It will continue being moved as each get routine is changed until all have been changed at which time it will be removed along with token member pointer.
return token;
}
[branch parser commit e34fbccacc]
Sunday, October 26, 2014
Parser – STL Preparations
The changes required to make the parser routines use the STL are going to be extensive, but an attempt will be made to break the changes into smaller incremental changes. Since the parser routines make use of various table functions, it will be necessary to modify the table entries and its functions to use STL. Some preparatory changes were made.
The table entries are divided into several groups for searching, which includes plain word, parentheses words, data type words and symbols. The parser utilizes these groups when searching if strings have a code. The data type words section was empty and upon consideration it was concluded that this group is not needed. This group may have been originally conceived for internal functions that don't have arguments (for example, a DATE$ function). However, these internal functions can go into the plain word group (the RND no argument function is already in this group). This group type along with its bracketing entries were removed.
[branch parser commit 0043105154]
The issue of the token being left allocated when the parser throws an exception could be resolved by not creating a token until a valid token is found. Used of the token member before an exception is thrown were examined and the only use was in the creation of the error exception. The only token members used were its column and length. The column was always the same as the current input position (except one instance) and the length was always 1.
All of the throw statements were modified to not use the token instead using the current input position with a length of 1. One case used a length of 2 (2 was previously used). For the "floating point constant is out of range" error, the statement to set the new input position was moved to after an error is thrown.
For the "expected sign or digits for exponent in floating point constant" error, two columns were reported, the column at the beginning of the number (for operator state) and the alternate column at the beginning of the error (for operand state). A number token is no longer accepted when invalid, so only the alternate column was being used. This mechanism was not needed, and for this error, the position of the error only is reported. This mechanism was also removed from the tester print error function.
[branch parser commit e92da57ef1]
The table entries are divided into several groups for searching, which includes plain word, parentheses words, data type words and symbols. The parser utilizes these groups when searching if strings have a code. The data type words section was empty and upon consideration it was concluded that this group is not needed. This group may have been originally conceived for internal functions that don't have arguments (for example, a DATE$ function). However, these internal functions can go into the plain word group (the RND no argument function is already in this group). This group type along with its bracketing entries were removed.
[branch parser commit 0043105154]
The issue of the token being left allocated when the parser throws an exception could be resolved by not creating a token until a valid token is found. Used of the token member before an exception is thrown were examined and the only use was in the creation of the error exception. The only token members used were its column and length. The column was always the same as the current input position (except one instance) and the length was always 1.
All of the throw statements were modified to not use the token instead using the current input position with a length of 1. One case used a length of 2 (2 was previously used). For the "floating point constant is out of range" error, the statement to set the new input position was moved to after an error is thrown.
For the "expected sign or digits for exponent in floating point constant" error, two columns were reported, the column at the beginning of the number (for operator state) and the alternate column at the beginning of the error (for operand state). A number token is no longer accepted when invalid, so only the alternate column was being used. This mechanism was not needed, and for this error, the position of the error only is reported. This mechanism was also removed from the tester print error function.
[branch parser commit e92da57ef1]
Saturday, October 25, 2014
Parser – Exceptions
When an error was detected, the parser set its internal error status to an enumerator for the error (either unknown token or a number error) and returned a an error token with its column and length set to the error. So to return an exception, the status, column and length values need to be included in the exception thrown. A simple Error structure was added to hold this information. The parser routines were modified to throw an error exception when detecting an error. The necessary values are included for this structure:
The translator get token routine was modified to catch parser exceptions (using C++ try and catch blocks). For no exception, the Good status enumerator is returned. For an exception, an error token is created from the column and length in the error structure. (The token constructor was modified with an additional length argument and to the C++ initializer syntax.) The rest of catch section remains the same except the status in the error structure is returned. The creation of an error token may be removed later if the translator is modified to the exception model for handling errors.
The tester parse input routine was also modified to catch parser exceptions. The while loop was replaced with a forever loop and the more flag removed. For no exception, the print token routine is called. The routine continues with the next token unless an End-of-Line token was returned. For an exception, the print error routine is called directly, and the routine returns immediately. The use of exceptions in the parse input routine allowed some simplifications in these print routines.
The print token previously had an error status argument. If the token type was an error, then the error status was passed to the print error routine with the token column and length. This check and call was removed, and so was the error status argument. The tab argument was always true, so it was also removed. The column, length and status arguments of the print error function were replaced with an error structure reference argument. This required creating a temporary error structure in the translate input and encode input routines (perhaps later the translator will be modified to throw an error and the program module modified to return an error structure).
It should be noted that the token created at the beginning of the parser function operator routine if left allocated when an exception is thrown. Previously the token is moved to the caller when the token contained an error the same as with a good token. This is not a problem because the parser will go out of scope once the translator handles the error and returns. The parser routines may be able to be changed to only create a token when a good token is found. This will be considered as the parser is changed to use the STL.
[branch parser commit 14265956f7]
throw Error {Status::Error, m_token->column(), m_token->length()};The set error functions were removed along with the error status member and its access function. Since the Error token type enumerator doesn't indicate an error token anymore, this enumerator was removed. The table entries that used with enumerator were changed to the default token enumerator (required the first enumerator to be set to 1). A leftover check in the get identifier routine was removed that set the token string to "invalid two work command" for an error token type, but this won't occur.
The translator get token routine was modified to catch parser exceptions (using C++ try and catch blocks). For no exception, the Good status enumerator is returned. For an exception, an error token is created from the column and length in the error structure. (The token constructor was modified with an additional length argument and to the C++ initializer syntax.) The rest of catch section remains the same except the status in the error structure is returned. The creation of an error token may be removed later if the translator is modified to the exception model for handling errors.
The tester parse input routine was also modified to catch parser exceptions. The while loop was replaced with a forever loop and the more flag removed. For no exception, the print token routine is called. The routine continues with the next token unless an End-of-Line token was returned. For an exception, the print error routine is called directly, and the routine returns immediately. The use of exceptions in the parse input routine allowed some simplifications in these print routines.
The print token previously had an error status argument. If the token type was an error, then the error status was passed to the print error routine with the token column and length. This check and call was removed, and so was the error status argument. The tab argument was always true, so it was also removed. The column, length and status arguments of the print error function were replaced with an error structure reference argument. This required creating a temporary error structure in the translate input and encode input routines (perhaps later the translator will be modified to throw an error and the program module modified to return an error structure).
It should be noted that the token created at the beginning of the parser function operator routine if left allocated when an exception is thrown. Previously the token is moved to the caller when the token contained an error the same as with a good token. This is not a problem because the parser will go out of scope once the translator handles the error and returns. The parser routines may be able to be changed to only create a token when a good token is found. This will be considered as the parser is changed to use the STL.
[branch parser commit 14265956f7]
Parser Errors – Removed Date Type
When the parser returned an error, it set the data type of the token to Double to indicate a number error or None for an unknown token. This was necessary since a number error could be returned when the parser was in operator state. This no longer occurs after the last change as the unknown token error is returned if the parser finds a character of a number when numbers are not allowed, so the error status alone can be used to determine the error type. Setting of the token data type for errors was removed. This reduces the amount of data to send back with an exception.
When the parser returned an error, the get token function of the translator returned the special Parser status enumerator. The translator routines used this enumerator along with the token data type to determine if the error was an unknown token or a number error. The parser error type can now determined directly with the error status from the parser, so the get token function was changed to return this status instead of the Parser enumerator.
The checks in the rest of the translator routines for the Parser enumerator and None data type were changed to just check for unknown token. The check in the get operand routine to set the token length to 1 for non-references when there was a parser error was removed since this is the only possible error. When getting the token after an argument in the process internal function routine, the check for a unary operator (an error) was moved to before the check for an error, which was changed to return all errors except unknown token. When this token was not a comma or closing parentheses the appropriate error is determined for the error or bad token.
The special Parser status enumerator was no longer used, so it was removed. The concludes all the prep work for changing parser errors into exceptions.
[branch parser commit ae9e97696e]
When the parser returned an error, the get token function of the translator returned the special Parser status enumerator. The translator routines used this enumerator along with the token data type to determine if the error was an unknown token or a number error. The parser error type can now determined directly with the error status from the parser, so the get token function was changed to return this status instead of the Parser enumerator.
The checks in the rest of the translator routines for the Parser enumerator and None data type were changed to just check for unknown token. The check in the get operand routine to set the token length to 1 for non-references when there was a parser error was removed since this is the only possible error. When getting the token after an argument in the process internal function routine, the check for a unary operator (an error) was moved to before the check for an error, which was changed to return all errors except unknown token. When this token was not a comma or closing parentheses the appropriate error is determined for the error or bad token.
The special Parser status enumerator was no longer used, so it was removed. The concludes all the prep work for changing parser errors into exceptions.
[branch parser commit ae9e97696e]
Thursday, October 23, 2014
Parser – Numbers (Operand State)
Upon making the next set of changes, I realized that State with Operator and Operand enumerators were not accurate terms for what the parser did with this option. All Operator state did was prevent numeric constants, but still allowed other operand type tokens (like string constants, identifiers, and functions). This was renamed Number with Yes and No enumerators, which more clearly expresses what the code does.
This change made it obvious where some simplifications could be made to the get token function of the translator. The value of the number (previously state) argument was selecting number (Yes) if the desired data type was not the default data type value (indicating the caller wants an operator token). When the desired data type is string, the number token would be invalid, so the condition of the desired data type is not string was added to the setting of the number argument.
Number tokens no longer are returned when looking for a string (an unknown token error will be returned). For error tokens, there was a check for operator state and the data type was double (indicating a number error), or the desired data type is string, which set the token length to 1 (so the error only points to the first character) and the token data type set to None (to indicate to callers that there is no number error). These situations now return an unknown token error for numbers with the length set to one and data type set to None, so this check was not needed.
The other check was much more involved (and confusing) but basically said if looking for an operand and there was not a number error, return an expression expected error. Otherwise return a parser error. This check was changed to if the desired data type was not empty (expecting operator) and not None (indicating a PRINT function is allowed) and there was an unknown token error, then return an expression expected error.
[branch parser commit 5690b1b4e9]
This change made it obvious where some simplifications could be made to the get token function of the translator. The value of the number (previously state) argument was selecting number (Yes) if the desired data type was not the default data type value (indicating the caller wants an operator token). When the desired data type is string, the number token would be invalid, so the condition of the desired data type is not string was added to the setting of the number argument.
Number tokens no longer are returned when looking for a string (an unknown token error will be returned). For error tokens, there was a check for operator state and the data type was double (indicating a number error), or the desired data type is string, which set the token length to 1 (so the error only points to the first character) and the token data type set to None (to indicate to callers that there is no number error). These situations now return an unknown token error for numbers with the length set to one and data type set to None, so this check was not needed.
The other check was much more involved (and confusing) but basically said if looking for an operand and there was not a number error, return an expression expected error. Otherwise return a parser error. This check was changed to if the desired data type was not empty (expecting operator) and not None (indicating a PRINT function is allowed) and there was an unknown token error, then return an expression expected error.
[branch parser commit 5690b1b4e9]
Subscribe to:
Posts (Atom)