The Tester class is another one-use class that fits the pattern of the function operator class. The main run function was changed to function operator function. The caller in the command line constructor was modified accordingly with the instance renamed from tester to test as was done with the parser instances.
The Tester class also had an error mechanism where its error message member was set if an error occurred. Both the constructor and the run function can generate an error. The has error access function returned if an error occurred, and the error message access function returned the error message. These functions were modified to throw an exception containing the error message (a standard string). The function operator function (formerly run) no longer needs to return success status as a boolean. This simplified the command line constructor since errors from both functions are caught with the same section of code. The error message member and its access functions were removed.
The redundant void was also removed from tester function definitions that don't have arguments. This was a practice I used when working with C code where the void in the arguments of a function definition indicates no arguments, as opposed to an empty parentheses, which could also indicate the old Kernighan and Ritchie (K&R) style function definition, which preceded the typed function definitions introduced with the first ANSI C standard. This void usage is used throughout and will slowly be removed as there is no reason to use it anymore.
[branch misc-cpp-stl commit 738eba02e1]
Thursday, November 13, 2014
Wednesday, November 12, 2014
Dictionary Information – Standard Classes
When the dictionary classes were changed to use the standard classes, the derived constant number and string information classes were missed because they were in different source files. For adding elements, these classes also used the method of increasing the vector size by one and then setting the new element. This causes the default constructor to be called when the size is increased, and then a copy when the element is set. It is more efficient to use the C++11 emplace functions to construct directly to the new element.
The info dictionary add function previously did the add element (increase vector size) and set element as two separate calls. The add function was changed to pass the token pointer to the add element function and not call the set element when adding an element. The add element functions get a token pointer argument to be used with the emplace back function to add the new element to the end of the value vectors (which were changed to standard vectors). The value vector of the constant string info class was changed to a vector of standard string pointers. The constant string info destructor was changed to use a C++11 range-for loop to delete the strings in this vector.
When dependency on Qt was removed from the dictionary classes, the quint16 type was changed to the standard uint16_t for holding indexes into the dictionaries (in the program code). This change was also missed in the constant info classes. The compiler didn't complain since these two types are the same under the hood. The constant info classes were changed to use uint16_t.
[branch misc-cpp-stl commit 14d3b329b7]
The info dictionary add function previously did the add element (increase vector size) and set element as two separate calls. The add function was changed to pass the token pointer to the add element function and not call the set element when adding an element. The add element functions get a token pointer argument to be used with the emplace back function to add the new element to the end of the value vectors (which were changed to standard vectors). The value vector of the constant string info class was changed to a vector of standard string pointers. The constant string info destructor was changed to use a C++11 range-for loop to delete the strings in this vector.
When dependency on Qt was removed from the dictionary classes, the quint16 type was changed to the standard uint16_t for holding indexes into the dictionaries (in the program code). This change was also missed in the constant info classes. The compiler didn't complain since these two types are the same under the hood. The constant info classes were changed to use uint16_t.
[branch misc-cpp-stl commit 14d3b329b7]
More Miscellaneous C++ and STL Changes
The Table class is going to be completely redesigned to better utilize C++. Before starting this major development, there are several more miscellaneous C++ and STL changes to be made to some of the other classes. The changes are expected to be minor, so development will be done on a single topic branch (to avoid short lived branches). These changes include:
- Modifying the dictionary information classes to use STL.
- Modifying the Tester class to be a function operator class, and use exceptions for error reporting.
- Modifying the Program Code class to use STL, be in its own header file, and not use a container class as its base class.
- Modifying the Program Model class to use STL.
- Modifying the Translator class to fully use STL, be a function operator class, and use exceptions for error reporting.
- Making other minor miscellaneous changes that come up along the way.
Tuesday, November 11, 2014
Recreator – Function Operator
The Recreator class has a single purpose, to take an RPN list and recreate the original input string of the BASIC code. This is similar to the Parser class, which was already changed to be a function operator class (see October 18). The recreator was also changed to be a function operator class. All that was required was to rename the recreate function to operator().
Unlike the parser that is instanced with the input string and tokens are repeatedly obtained using the function operator function until either an error (exception) occurs or an end-of-line token is returned, the recreator is single use and its function operator function will only be called once for a given RPN list. Therefore, only a temporary instance is needed - no member or local instance is required.
The users of the recreator, the tester and program model classes, previously contained a pointer to a recreator instance, which was created by their constructors. The recreator instance was deleted automatically when these class instances were deleted because a standard unique pointer was used. Since the recreator is single use, only a local instance is required and these members were removed. Instead of creating a local variable instance, a temporary instance is used (the instance is created and deleted during the statement):
Since this is the last change to be made to the recreator class, the redundant 'void' keywords were removed from several of the recreator function definitions, and the use of Q_UNUSED macro was replaced with the '(void)' syntax on unused variables in the various recreate functions. This concludes work on the recreator class, so the recreator branch was merged to the develop branch and deleted.
[branch recreator commit a0e99f6dcf]
[branch develop merge commit dc4dd26876]
Unlike the parser that is instanced with the input string and tokens are repeatedly obtained using the function operator function until either an error (exception) occurs or an end-of-line token is returned, the recreator is single use and its function operator function will only be called once for a given RPN list. Therefore, only a temporary instance is needed - no member or local instance is required.
The users of the recreator, the tester and program model classes, previously contained a pointer to a recreator instance, which was created by their constructors. The recreator instance was deleted automatically when these class instances were deleted because a standard unique pointer was used. Since the recreator is single use, only a local instance is required and these members were removed. Instead of creating a local variable instance, a temporary instance is used (the instance is created and deleted during the statement):
Recreator recreator; → string = Recreator{}(rpnList);
string = recreator(rpnList);
Since this is the last change to be made to the recreator class, the redundant 'void' keywords were removed from several of the recreator function definitions, and the use of Q_UNUSED macro was replaced with the '(void)' syntax on unused variables in the various recreate functions. This concludes work on the recreator class, so the recreator branch was merged to the develop branch and deleted.
[branch recreator commit a0e99f6dcf]
[branch develop merge commit dc4dd26876]
Monday, November 10, 2014
Recreator – Standard Strings
The strings in the recreator stack item and several local stacks were changed to standard strings along with the recreator output string. The QString::append function was previously used to append to these strings. The std::string class also has an append function but it does not support a single character argument (though does support a count with the single character). Use of append function was replaced with the addition assignment operator (+=).
The append and top append functions of the recreator are still used by the various external recreate functions. Unlike with QString where a plain character is implicitly converted to a QChar, which then implicitly converted to a QString, the std::string class does not have similar functionality. For the same functionality, additional append and top append functions were added that take a plain character argument. All of the append functions use the addition assignment to append to the strings.
The append and top append functions, along with the pop with operands function, were changed to take a string rvalue reference argument (std::string &&). In most cases, the argument given to these functions is a temporary value, which using this type moves the temporary value to the function instead of copying the value. In other cases, the variable passed was no longer needed and was going out of scope, so the variable is passed via the std::move function. The advantage with using an rvalue reference argument is that it requires a temporary, so an error is given when the argument not a temporary value or the variable is not moved.
There are several cases in the recreator functions where values are obtained from the table. Since the table functions are still returning QString values, a call to the toStdString function was added.
The constant string recreate function used the QString::replace function that substitutes all instances of a particular character with a string. This function was used to convert all of the double quote characters in the constant to two double quote characters. There is no equivalent easy to use standard function to do the same thing. A range-for statement was added to iterate though the string making a copy into a local string. For each double quote character added, a second double quote is added.
The recreator contained an output is empty access function, for determining if the output string was empty, and a output last character function, for returning the last character added to the output string. These functions were only used by the remark recreate function to check if the last character added to a non-empty output was not a space (to see is a space should be added before the REM operator). These functions were replaced with the single back is not space function.
[branch recreator commit 56b40e3411]
The append and top append functions of the recreator are still used by the various external recreate functions. Unlike with QString where a plain character is implicitly converted to a QChar, which then implicitly converted to a QString, the std::string class does not have similar functionality. For the same functionality, additional append and top append functions were added that take a plain character argument. All of the append functions use the addition assignment to append to the strings.
The append and top append functions, along with the pop with operands function, were changed to take a string rvalue reference argument (std::string &&). In most cases, the argument given to these functions is a temporary value, which using this type moves the temporary value to the function instead of copying the value. In other cases, the variable passed was no longer needed and was going out of scope, so the variable is passed via the std::move function. The advantage with using an rvalue reference argument is that it requires a temporary, so an error is given when the argument not a temporary value or the variable is not moved.
There are several cases in the recreator functions where values are obtained from the table. Since the table functions are still returning QString values, a call to the toStdString function was added.
The constant string recreate function used the QString::replace function that substitutes all instances of a particular character with a string. This function was used to convert all of the double quote characters in the constant to two double quote characters. There is no equivalent easy to use standard function to do the same thing. A range-for statement was added to iterate though the string making a copy into a local string. For each double quote character added, a second double quote is added.
The recreator contained an output is empty access function, for determining if the output string was empty, and a output last character function, for returning the last character added to the output string. These functions were only used by the remark recreate function to check if the last character added to a non-empty output was not a space (to see is a space should be added before the REM operator). These functions were replaced with the single back is not space function.
[branch recreator commit 56b40e3411]
Sunday, November 9, 2014
Recreator – Standard Stacks
The stack member was changed to a standard stack. Two local stacks were also changed to standard stacks (in the assign recreate and push with operands functions). The stack access functions were modified accordingly:
To use the emplace stack function, a constructor was added to the stack item structure. The constructor has arguments for each of the three member variables. The precedence and unary operator arguments were made optional with defaults (from the replaced push function). The member variables of the stack item were also renamed with the "m_" prefix.
[branch recreator commit 5451552470]
- The push function was replaced with the emplace function defined as a template member function to forward the arguments to the stack emplace function (see details below).
- The pop function was replaced with the pop string function with basically does the same thing except is not longer checks if the stack is empty (which was only needed during initial development of the recreator).
- The top function was replaced with individual functions for returning the precedence and unary operator flag members of the top stack item. No similar function was added for the string member as there are other functions for accessing the string member: pop string, top append, and new top add parentheses.
- A new top add parentheses function was added to add parentheses around the string of the top stack item. This function is only called by the parentheses recreate function.
- The precedence and unary operator pointer arguments were removed from the pop with parentheses function since they were only used by the parentheses recreate function (which now has its own access function).
- The stack is empty function was renamed to just empty to be consistent with the other stack access functions (which don't have the stack name included) and with the STL naming convention.
template <typename... Args>This utilizes the new C++11 perfect forwarding feature, allowing a function template to pass its arguments through to another function and avoids unnecessary copying. What the above template does is pass the stack item constructor arguments to the stack emplace function (using the std::forward template function), which are then passed to the stack item constructor when the new item is created on the stack, without copying the arguments.
void emplace(Args&&... args) {
m_stack.emplace(std::forward<Args>(args)...);
}
To use the emplace stack function, a constructor was added to the stack item structure. The constructor has arguments for each of the three member variables. The precedence and unary operator arguments were made optional with defaults (from the replaced push function). The member variables of the stack item were also renamed with the "m_" prefix.
[branch recreator commit 5451552470]
Saturday, November 8, 2014
Recreator – Separator Member
Since there were a number standard string to c-style string conversions to convert to a QString in the recreator functions, the Recreator class is next to be transitioned to the STL. This was started by changing the separator member from QChar to char along with its access functions. An initializer for the separator member was added since the plain char type does not have a default constructor like the QChar class.
Two uses of the separator access function that required special handling. These were in the input assign recreate and assign string recreate functions. Previously the separator was added to other literal character constants. Now that the separator is the plain char type, the compiler will treat the plus operator as addition instead of concatenation. The separator is temporarily converted to a QChar so that the plus gets compiled as string concatenation instead of character addition.
[branch recreator commit 3b5b789198]
Two uses of the separator access function that required special handling. These were in the input assign recreate and assign string recreate functions. Previously the separator was added to other literal character constants. Now that the separator is the plain char type, the compiler will treat the plus operator as addition instead of concatenation. The separator is temporarily converted to a QChar so that the plus gets compiled as string concatenation instead of character addition.
[branch recreator commit 3b5b789198]
Token – Standard String Member
It turned out that changing the token string member to a standard string did not require a lot of other changes. The string access functions were modified and the c_str function call was removed from the constructor initializers. The equality operator function was modified to use straight equality operator for case sensitivity comparison (REM, REM operator and string constants) and the new no case string equal function for case insensitive comparison.
In the rest of the code, the c_str function call was removed where a standard string was present, and the toStdString function call was removed from the token string access function where the result is put into a standard string or output in a standard output stream. In several places in the recreator functions, a c_str function call was added where a QString is needed (the c-style string returned is implicitly converted). Finally, one line in the translator using the QString::startsWith function with the case insensitive option was changed to convert the first character to upper case before comparing it (only the first character was being compared).
[branch token commit ee7b926daa]
Since there is no more work for the token class, the short-lived token branch was merged to the develop branch.
[branch develop merge commit ee12854e36]
In the rest of the code, the c_str function call was removed where a standard string was present, and the toStdString function call was removed from the token string access function where the result is put into a standard string or output in a standard output stream. In several places in the recreator functions, a c_str function call was added where a QString is needed (the c-style string returned is implicitly converted). Finally, one line in the translator using the QString::startsWith function with the case insensitive option was changed to convert the first character to upper case before comparing it (only the first character was being compared).
[branch token commit ee7b926daa]
Since there is no more work for the token class, the short-lived token branch was merged to the develop branch.
[branch develop merge commit ee12854e36]
Utility – Case Insensitive Comparisons
The token equal operator function uses case insensitive string comparisons that need to be replaced with standard string equivalents. There is no direct equivalent, so the solution used so far was to use the std::equal function passing a case insensitive character comparison lambda function. Since the std::equal function assumes equal length containers, the size of the strings need to be compared first to make sure the strings are the same length (or at least the primary string is not less than the secondary string when doing a string begins with comparison).
This pattern of comparing the string lengths before comparing strings is repeating, therefore two in line functions were added to do this. The first, named no case string equal, checks if the string are equal and the second, named no case string begins with, checks if the length of the primary string is greater than or equal to the string being compared. The lambda function was renamed to no case character equal for consistency. The lambda function was moved from the main header file to the utility header file along with the two new in line functions.
[branch token commit 1dde64fb7b]
This pattern of comparing the string lengths before comparing strings is repeating, therefore two in line functions were added to do this. The first, named no case string equal, checks if the string are equal and the second, named no case string begins with, checks if the length of the primary string is greater than or equal to the string being compared. The lambda function was renamed to no case character equal for consistency. The lambda function was moved from the main header file to the utility header file along with the two new in line functions.
[branch token commit 1dde64fb7b]
Parser – Standard Input Stream
The parser functions have been modified to use a standard input string stream, so the input member could now be changed to the std::istringstream class, and the input position member removed. The current input position can be obtained directly from the input stream using the tellg member function. The skip white space function was removed since this can be done directly on the input stream (in other words, extract white space):
When using an input stream, care must be taken when using the tellg function to obtain the current input position. This function returns an EOF value (-1) once the input stream has been read past the end. So this function can't be used is a previous operation could have possibly read past the end.
There times when the input position must be reset (like when the second word of a possibly two-word command is not valid). The seekg function is used to the input position. However, this function does not work once the input stream has been read past the end. This is because the EOF flag is set. To clear this condition, the clear function needs to be called, so this call precedes all seekg function calls except one where an EOF cannot have occurred.
In the get string function, the characters read are counted so that the length of the string in the input is known when the token is constructed and returned (pairs of double quotes count as one character in the string, but take two characters in the input string, so must be counted as two). The ending input position cannot be used to determine the length in the input string because the position is not valid if the string constant is at the end of the line (see issue with tellg function above).
The constructor of the parser was changed to take a standard string input. Both callers were modified accordingly - the tester class already had a standard string, but the translator needs to convert from its QString to a standard string (until the translator is modified). Dependency on Qt has almost been removed from the parser except for one call to obtain the name for the REM command (which will be handled when the table is modified).
[branch parser commit 8e71a71fd5]
One outstanding item remains - the token string member is still a QString though its constructors have been modified to take standard string arguments. This will not a trivial change since many users of the token string still expect a QString. Therefore, this work will take place in a new development branch. This concludes work on the parser, so the parser branch was merged to the develop branch and deleted.
[branch develop merge commit 2cafb22a8e]
m_input >> std::ws;In several places, a temporary position or length integer variable is used to get the current position (tellg) or length (length) because these functions return a pos_type and size_type values, which are 64-bit integers. The token constructors and error structure only accept integers (32-bit). There is no reason to change the member variable types to 64-bit integers as there will never be input lines or strings that are long enough to require 64-bit integers.
When using an input stream, care must be taken when using the tellg function to obtain the current input position. This function returns an EOF value (-1) once the input stream has been read past the end. So this function can't be used is a previous operation could have possibly read past the end.
There times when the input position must be reset (like when the second word of a possibly two-word command is not valid). The seekg function is used to the input position. However, this function does not work once the input stream has been read past the end. This is because the EOF flag is set. To clear this condition, the clear function needs to be called, so this call precedes all seekg function calls except one where an EOF cannot have occurred.
In the get string function, the characters read are counted so that the length of the string in the input is known when the token is constructed and returned (pairs of double quotes count as one character in the string, but take two characters in the input string, so must be counted as two). The ending input position cannot be used to determine the length in the input string because the position is not valid if the string constant is at the end of the line (see issue with tellg function above).
The constructor of the parser was changed to take a standard string input. Both callers were modified accordingly - the tester class already had a standard string, but the translator needs to convert from its QString to a standard string (until the translator is modified). Dependency on Qt has almost been removed from the parser except for one call to obtain the name for the REM command (which will be handled when the table is modified).
[branch parser commit 8e71a71fd5]
One outstanding item remains - the token string member is still a QString though its constructors have been modified to take standard string arguments. This will not a trivial change since many users of the token string still expect a QString. Therefore, this work will take place in a new development branch. This concludes work on the parser, so the parser branch was merged to the develop branch and deleted.
[branch develop merge commit 2cafb22a8e]
Thursday, November 6, 2014
Parsing Identifiers – Standard Library
The get identifier function was changed to use a standard input stream (again using a temporary input string stream like the previous functions). This function used scan word support function to look for a word and was renamed more appropriately to get word. Instead of checking for a REM command first (because unlike other commands, a space is not required after the command), the get word function is called first to get a word. If no valid word is found, an empty token pointer is returned.
The first check on a valid word is if the word starts with letters in the REM command name using the std::equal function with the no case compare lambda function. Since the REM command name is still in the table as a QString, it is temporarily converted to standard string. For a remark, the input position is set to beginning of the string of the remark, and the word string is then replaced with the rest of the characters on the line, from which the token is created and returned.
An issue was discovered with the parsing of define function tokens (identifiers that start with "FN"). A valid defined function name should start with a letter, but there was no check for a letter or even a check if there were any characters after the "FN" so identifiers like FN and FN1 were incorrectly accepted as defined function tokens. Instead of rejecting these names as invalid defined function names, the decision was made to allow these names, and treat them as regular names (variables and arrays).
The get word support function was modified in the same way (by using a temporary input stream). It also returned three values, the position after the word found, the data type of the word and whether the word has a parentheses. Two of these were returned by passing references. The position is not needed since it will be obtained from the input stream, however, a string for the word is needed because it is read from the stream. A new Word structure was added to hold the word string, data type and parentheses flag, which is now returned. An empty word string indicates no valid word found.
To simplify the handling of two word commands in the get identifier function, the check of the second to make sure that it does not have a data type or parentheses was moved to the get word function. If the second word does, an empty word string is returned and the input stream is repositioned back to the beginning of the word. A word type argument was added with values first and second to enable this second word checking.
The get identifier function uses the two-word table search function, which was modified to take two standard strings. The token constructor for identifiers was modified to take a standard string argument, which is temporarily converted to a c-style string to initialize the QString token member. Several invalid defined function names were added to parser test #2 (identifiers) to verify these names are treated as plain identifiers (with and without parentheses).
[branch parser commit dbbd9fe054]
The first check on a valid word is if the word starts with letters in the REM command name using the std::equal function with the no case compare lambda function. Since the REM command name is still in the table as a QString, it is temporarily converted to standard string. For a remark, the input position is set to beginning of the string of the remark, and the word string is then replaced with the rest of the characters on the line, from which the token is created and returned.
An issue was discovered with the parsing of define function tokens (identifiers that start with "FN"). A valid defined function name should start with a letter, but there was no check for a letter or even a check if there were any characters after the "FN" so identifiers like FN and FN1 were incorrectly accepted as defined function tokens. Instead of rejecting these names as invalid defined function names, the decision was made to allow these names, and treat them as regular names (variables and arrays).
The get word support function was modified in the same way (by using a temporary input stream). It also returned three values, the position after the word found, the data type of the word and whether the word has a parentheses. Two of these were returned by passing references. The position is not needed since it will be obtained from the input stream, however, a string for the word is needed because it is read from the stream. A new Word structure was added to hold the word string, data type and parentheses flag, which is now returned. An empty word string indicates no valid word found.
To simplify the handling of two word commands in the get identifier function, the check of the second to make sure that it does not have a data type or parentheses was moved to the get word function. If the second word does, an empty word string is returned and the input stream is repositioned back to the beginning of the word. A word type argument was added with values first and second to enable this second word checking.
The get identifier function uses the two-word table search function, which was modified to take two standard strings. The token constructor for identifiers was modified to take a standard string argument, which is temporarily converted to a c-style string to initialize the QString token member. Several invalid defined function names were added to parser test #2 (identifiers) to verify these names are treated as plain identifiers (with and without parentheses).
[branch parser commit dbbd9fe054]
Sunday, November 2, 2014
Parsing Operators – Standard Library
The get operator function was changed to use a standard input stream (again using a temporary input string stream like the two previous functions). This function uses one of the table search functions, which was modified to take a standard string.
The table search function used the compare function from the QString class with the case insensitive option. There is no equivalent function in the standard string class. The std::equal function is used instead by passing a no case comparison lambda function. This is the same lambda function used in the Tester class, so this definition was moved to the main header file. Since the name in the table is still a QString, it is temporarily converted to a standard string. The std::equal function assumes the arguments are the same size, so the size of the strings are checked first.
The token constructor for codes was changed to take a standard string, which defaults to an empty string. For now, these are converted for the QString member variable by obtaining a c-style string from the standard string, which is implicitly converted. The only caller of this constructor using this argument is the new token table function, which was also modified to take a standard string. Callers of this function using the string argument were modified to pass a standard string.
[branch parser commit 27f06e8714]
The table search function used the compare function from the QString class with the case insensitive option. There is no equivalent function in the standard string class. The std::equal function is used instead by passing a no case comparison lambda function. This is the same lambda function used in the Tester class, so this definition was moved to the main header file. Since the name in the table is still a QString, it is temporarily converted to a standard string. The std::equal function assumes the arguments are the same size, so the size of the strings are checked first.
The token constructor for codes was changed to take a standard string, which defaults to an empty string. For now, these are converted for the QString member variable by obtaining a c-style string from the standard string, which is implicitly converted. The only caller of this constructor using this argument is the new token table function, which was also modified to take a standard string. Callers of this function using the string argument were modified to pass a standard string.
[branch parser commit 27f06e8714]
Parsing Strings – Standard Library
The get string function was changed to use a standard input stream (temporarily putting the input string from the current position into a temporary local input string stream of the same name as the member variable to simulate the final parser code). The looking at and the obtaining of current character was changed as previously described.
Instead of incrementing the local position variable for each character in the string constant, this variable is just set to the current input position. This will be changed to get the position within the input stream stream once the member variable is changed. The current input position is incremented for each character. After the change, the current input position member variable will not be needed.
[branch parser commit ab3b1c08f18]
Instead of incrementing the local position variable for each character in the string constant, this variable is just set to the current input position. This will be changed to get the position within the input stream stream once the member variable is changed. The current input position is incremented for each character. After the change, the current input position member variable will not be needed.
[branch parser commit ab3b1c08f18]
Parsing Numbers – Standard Library
When the parser is changed to use the standard library, instead of placing the input string into a string member variable, it will put into a standard input string stream from which the characters will be pulled from. A position into the input string will not need to be maintained during the processing of the line.
The get number function was the first to be changed to this model. Temporarily, the input string from the current position (a substring) is transferred into a temporary local input string stream of the same name as the member variable to simulate the final parser code. Two failed attempts were made to use standard library functions to parse and read numbers.
The first attempt used the stoi function to convert the number directly. The problem was that it doesn't report the specifics of the error when the conversion fails, throwing only a invalid argument or out-of-range exception. The type of error could be determined by a series of complex checks of the string. A working solution was mostly achieved with one remaining issue. When an out-of-range exception was thrown, there is no clue as to the length of the string that was processed (which is needed to properly highlight the error).
The second attempt used the extraction operator (>>) to get the number directly into a double or integer variable. Again detecting an error and determining the type of error was difficult (and was not actually achieved when this attempt was abandoned). The code again was also complicated.
These attempts were made to try to eliminate the involved (but working) algorithm already in place to parse numbers. The decision was made to use the current algorithm, and was modified to read from a standard input stream. The reading of the current character was changed to:
The integer and double token constructors were modified to take standard string arguments. For now, these are converted for the QString member variable by obtaining a c-style string from the standard string, which is implicitly converted. This is temporary until the token string member is changed to a standard string.
[branch parser commit 8513875e33]
The get number function was the first to be changed to this model. Temporarily, the input string from the current position (a substring) is transferred into a temporary local input string stream of the same name as the member variable to simulate the final parser code. Two failed attempts were made to use standard library functions to parse and read numbers.
The first attempt used the stoi function to convert the number directly. The problem was that it doesn't report the specifics of the error when the conversion fails, throwing only a invalid argument or out-of-range exception. The type of error could be determined by a series of complex checks of the string. A working solution was mostly achieved with one remaining issue. When an out-of-range exception was thrown, there is no clue as to the length of the string that was processed (which is needed to properly highlight the error).
The second attempt used the extraction operator (>>) to get the number directly into a double or integer variable. Again detecting an error and determining the type of error was difficult (and was not actually achieved when this attempt was abandoned). The code again was also complicated.
These attempts were made to try to eliminate the involved (but working) algorithm already in place to parse numbers. The decision was made to use the current algorithm, and was modified to read from a standard input stream. The reading of the current character was changed to:
m_input[pos] → m_input.peek()The original code incremented a local position when a character had been processed (will become part of the number string). This position increment was replaced with pulling a character from the input stream and appending it to the local number string:
pos++; → number.push_back(m_input.get());The various character type tests (to upper, is digit) were changed from the QChar functions to the standard ctype tests. Once a possible valid number was parsed into the local string, if itt didn't contain a decimal point or exponent, an attempt is made to convert it to an integer using the stoi function. If successful, an integer number token is created from the local string and returned. If an out-of-range exception is thrown or had a decimal point or exponent, an attempt is made to convert it to a double using the stod function. If successful, a double number token is created from the local string and returned. Another out-of-range exception results in a floating point out of range exception being thrown.
The integer and double token constructors were modified to take standard string arguments. For now, these are converted for the QString member variable by obtaining a c-style string from the standard string, which is implicitly converted. This is temporary until the token string member is changed to a standard string.
[branch parser commit 8513875e33]
Saturday, November 1, 2014
Parser – Number Error Corrections
Qt functions are currently being used to convert strings of numbers to a double or an integer in the get number routine. This routine will be changed to use STL functions. While investigating this, a few problems were discovered with how some of the number errors were being reported.
The "expected sign or digits for exponent in floating point constant" error was being reported even when the exponent sign was present. A new "expected digits for exponent in floating point constant" error for this situation. When an incorrectly formed number contained a single decimal point followed by the start of an exponent ('E'), the "expected digits in mantissa of floating point constant" error only pointed to the decimal point. The error was changed to point to both the decimal point and the 'E' character.
The translator was not reporting the "expected command" error correctly when there was a number error - the error was pointing to the number error which was either not at the beginning of the command or its length was not one. This occurred because the number error was not correctly reported as an unexpected token error when a reference was request (at the beginning of a statement).
This was corrected by adding a reference argument to the get token translator routine with a default of None. Only when this argument is None are number tokens allowed. When an unknown token error is returned from the parser, the reference argument is used to generate the appropriate expected error status. For the first token obtained from the get commands translator routine, this argument was set to All, which prevents number tokens (an unexpected token error is return for all number including number errors).
The get operands translator routine was modified to pass its reference argument directly to the get token call. Since get token now generates the appropriate error for references, this routine no longer needs to intercept the error to return the appropriate error or set the error length to one for references. The status is simply returned when the status is not Good. The LET translate routine handles reporting errors when neither a command nor a reference starts a line. The section handling errors was structured poorly and was rewritten.
Certain types of errors were reported differently as a result of these changes. Previously, the error for an incorrect statement like 34=A was reported as an "expected item for assignment" error pointing to the 34. Now the "expected command" error is reported pointing to only the first character of the number. Both errors are technically correct, and it would be difficult to report the previous error. The error was changed to "expected command or item for assignment" since both are applicable at the beginning of a statement.
The expected results for parser test #3 (numbers), translator tests #1 (assignments), #3 (more assignments), and #14 (parser errors) were updated for these changes. Some addition tests were added to translator test #14 for the new expecting digits for exponent error. Many of the translator tests results were also updated for the expected command message change.
[branch parser commit 20e46cc617]
The "expected sign or digits for exponent in floating point constant" error was being reported even when the exponent sign was present. A new "expected digits for exponent in floating point constant" error for this situation. When an incorrectly formed number contained a single decimal point followed by the start of an exponent ('E'), the "expected digits in mantissa of floating point constant" error only pointed to the decimal point. The error was changed to point to both the decimal point and the 'E' character.
The translator was not reporting the "expected command" error correctly when there was a number error - the error was pointing to the number error which was either not at the beginning of the command or its length was not one. This occurred because the number error was not correctly reported as an unexpected token error when a reference was request (at the beginning of a statement).
This was corrected by adding a reference argument to the get token translator routine with a default of None. Only when this argument is None are number tokens allowed. When an unknown token error is returned from the parser, the reference argument is used to generate the appropriate expected error status. For the first token obtained from the get commands translator routine, this argument was set to All, which prevents number tokens (an unexpected token error is return for all number including number errors).
The get operands translator routine was modified to pass its reference argument directly to the get token call. Since get token now generates the appropriate error for references, this routine no longer needs to intercept the error to return the appropriate error or set the error length to one for references. The status is simply returned when the status is not Good. The LET translate routine handles reporting errors when neither a command nor a reference starts a line. The section handling errors was structured poorly and was rewritten.
Certain types of errors were reported differently as a result of these changes. Previously, the error for an incorrect statement like 34=A was reported as an "expected item for assignment" error pointing to the 34. Now the "expected command" error is reported pointing to only the first character of the number. Both errors are technically correct, and it would be difficult to report the previous error. The error was changed to "expected command or item for assignment" since both are applicable at the beginning of a statement.
The expected results for parser test #3 (numbers), translator tests #1 (assignments), #3 (more assignments), and #14 (parser errors) were updated for these changes. Some addition tests were added to translator test #14 for the new expecting digits for exponent error. Many of the translator tests results were also updated for the expected command message change.
[branch parser commit 20e46cc617]
Thursday, October 30, 2014
Parser – Unique Pointers
The parser routines create a token held in a shared pointer upon return. The main function operator routine returns this shared pointer. The token is not actually being shared, just moved until it reaches the caller. There is no reason to use a shared pointer in the parser as a standard unique pointer is sufficient. The parser routines were changed to return a unique pointer. Another alias was added for a unique token pointer:
One other small unrelated change was made to the get identifier routine with the creation of the REM command token. This code was simplified as it was not necessary to copy the comment string from the input into a temporary string before creating the token. The string can be passed directly when the token is created. The new position can simply be set to the length of the input string. This was already done for the remark operator in the get operator routine.
[branch parser commit 336ad07bf8]
using TokenUniquePtr = std::unique_ptr<Token>;Unfortunately, there is no equivalent function for std::unique_ptr like the std::make_shared() function for std::shared_ptr (though one has been added for C++14) so unique pointers must be initialized using the new operator with the unique token pointer alias constructor:
return TokenUniquePtr{new Token {pos, len, type, dataType, m_input}};The callers of the parser operator function did not needed to be modified since there is a shared pointer constructor that takes unique pointer as an argument (the shared pointer takes ownership of unique pointer). The table new token function was also modified to return a unique token pointer.
One other small unrelated change was made to the get identifier routine with the creation of the REM command token. This code was simplified as it was not necessary to copy the comment string from the input into a temporary string before creating the token. The string can be passed directly when the token is created. The new position can simply be set to the length of the input string. This was already done for the remark operator in the get operator routine.
[branch parser commit 336ad07bf8]
Subscribe to:
Posts (Atom)