Thursday, December 4, 2014

Pre-Table – Removed Test Names Header

The test names header file contained an array of C-style strings indexed by a code enumerator each with the name of that code.  This file was automatically generated by an awk script that extracted the information from the table class source file.  This array was only used by the tester print token function (itself only called from the tester parse input function).

There was really no reason that the code name itself couldn't be used for this test output, specifically the debug name name for the code, which included the secondary name if set or the primary name.  The print token function was modified to use the debug name instead.  The auto-generating test names awk script was removed and the CMake build file was updated accordingly.  The expected parser test output files were updated.

A problem was found in the table debug name access function.  For a two-word command, this function was only returning the secondary name.  This only affected the INPUT PROMPT command, which was being output as just PROMPT.  This function was modified to also get the primary name if the multiple table entry member is set to Two Word (with no space between to the two words).  This affected two of the expected encoder test results, which were updated.

[branch table commit 34c45230f6]

Wednesday, December 3, 2014

New Table Implementation

The Table class is only remaining class to be modified to utilize C++ and the STL.  The table was poorly designed and was implemented in a way that was more appropriate for C than is was for C++.  Goals of the new implementation include:
  • Eliminating the auto-generated header files, one containing code enumeration and one containing a code names array of C-style strings.
  • Eliminating some unnecessary table entry variables.
  • Eliminating the use of the untyped C-style macros.
  • Reduce the number of values needed to define each code table entry.  Currently many entries have default values.  This was somewhat alleviated by the Expression Information structure where some entries (like commands) don't have an instance of one these.  This may possibly be accomplish using inheritance and templates.
  • Making setup of the table entries easier and less error prone especially when it comes to the associated codes (which will be renamed alternate codes).
I have been investigating how to redesign the table.  Along the way I discovered some things about the table that could be simplified.  So before beginning the new design, these things will be done that may make the transition to the new design a little easier.  This work will begin in a new table topic branch.

Tuesday, December 2, 2014

Miscellaneous Minor Changes

A bunch of miscellaneous minor changes have been accumulating during this topic branch that were finally taken care of, which included:
  • Adding the C++11 override keyword to functions in derived dictionary information classes that implement virtual function from the base abstract dictionary class.
  • Changing the dictionary information array functions to return a reference to the vector instead of a pointer to the array contained within the vector.  The pointer to the array was originally returned as it was thought would allow more efficient access to the array elements during runtime, but the vector bracket operator should essentially be the same.
  • Adding the noexcept keyword to all parser functions that don't throw exceptions.
  • Replacing several Qt forever macros with an empty for (to remove dependency on the Qt header files).
  • Removing two c_str function calls from recreator functions that were previously missed (these were originally added to work with QString, but were later replaced with standard strings).
  • Correcting some minor code formatting issues.
  • Removing two unused table search functions.
  • Changing two newline '\n' character outputs in tester class functions to std::endl, which outputs the newline character plus flushes the output stream.  This was helpful during debugging to know which input line was being processed when some sort of crash occurred (otherwise the output was still in a buffer in memory making it difficult to know which line it was on).
  • Changing the header argument of the tester translate input function from a constant C-style character string to an rvalue reference to a standard string.  An rvalue reference requires a temporary value, which is what is provided by calls to this function that don't use the blank string default.
  • Assigning the underlying to the sub-code enumeration to an unsigned 16-bit integer (uint16_t), something possible by C++11.  The enumerator values were changed to unsigned 16-bit values (where the four leading zero digits were removed).
[branch misc-cpp-stl commit 69ae2f0b15]

This concludes this development topic and the misc-cpp-stl branch was merged to the develop branch and deleted.  The Table class is the only remaining class that has not been transitioned from Qt to the STL and will also be reimplemented for better C++ utilization.

[branch develop merge commit 7f303f1bb1]

Sunday, November 30, 2014

Translator – Better Token Handling

A common pattern in the translator functions was a reference to a token pointer argument that a token was pass into and out of the function.  The token returned was the next token that the function could not process (a terminating token).  Several of the functions allowed an unset token pointer, in which case, they would get a token, otherwise they would use the token passed in.  This was a strange pattern and somewhat difficult to work with.

The translator functions were modified to a better pattern where a member token pointer was added to the translator class to hold the current token.  The translator functions were modified to use this current token member.  The token is moved out of this member variable when successfully processed (consumed).  The token pointer reference argument was removed from the translator functions.

The get token function was modified to put the token obtained from the parser into this new member, and the token argument was removed.  If the current token member already has a token, then no action is taken.

The process command function when modified was reduced to only a few lines and since it was only called once, its code was moved to the get commands function.  The command token and token arguments were removed from the LET, PRINT and INPUT translate functions, which were modified to get the current token from the translator.  Several access functions for the current token were added to the translator including getting a constant reference to the token pointer member, reseting the token pointer member, and moving the token pointer out of the member.  The latter two force a new token to be obtained upon the next get token function call.

The get operand function was modified to leave the current token pointer member empty if a valid operand was processed (added to the output list and pushed to the done stack).  This forced the next call to the get token function to obtain a new token from the parser.

For sub-string assignments, the get operand function set the reference flag of the token if the token was a sub-string function (LEFT$, MID$, or RIGHT$) and a reference was requested.  The process internal function function to identify a sub-string assignment and to request a string variable for the first argument of the function.  Upon return, this reference flag was cleared.  This reference flag toggling was replaced by passing the reference enumerator as an argument.

There was some code in the get operand function that set the code of the token to the Define Function with Parentheses code enumerator.  These statements were moved to the process parentheses token where the similar statement reside for setting the Array and Function codes.

Some other minor changes were made including changing the RPN list token access function to return a reference to the token pointer instead of the token pointer itself (to prevent the copying of the token pointer and updating the shared pointer use counts), reorganizing some code in the various routines, renaming some local token variables, and updating comments for changes made.

[branch misc-cpp-stl commit 828f210f18]

Friday, November 28, 2014

Translator – Some Minor Improvements

The done stack pop error token function was used create an error token from the item on top of the done stack.  An item on the done stack could consist of an entire expression so in addition to a token, it contained the first and last tokens of the expression, which would be null if the item only contained a single token.

There is no need to create an error token since exceptions are now thrown for errors, which do not contain tokens.  All callers to this function only used the error token to create a token error (status with the column and length from the error token).  This function was modified to return a token error and was more appropriately renamed to the done stack top token error function.  There is also no need to pop the done item from the stack since when an error is thrown, the translator instance goes out of scope and all members are deleted including the done stack.

Several other minor changes were also made including removing the token set through function since it was no longer used, replacing the uses of the Qt forever macros with a plain empty for (;;), renaming the expected error status function (had "Error" abbreviated as "Err"), adding the C++ noexcept keyword to translator functions that do not through an exception, and updating a few comments missed for changes made earlier.

[branch misc-cpp-stl commit 625195f18d]

Thursday, November 27, 2014

Translator Improvements – Getting Tokens

While modifying the translator routines to throw exceptions, it was noticed that some minor  improvements could be made.  The first of these is with calls to the get token function.  There was a similar pattern to most of the calls to this function, where the call was in a try block and for caught errors, the error was set to an appropriate error.

An error status argument was added to the get token function.  The caller puts its desired error status to be returned when the parser returns an Unknown Token.  With this change, the caller no longer needs to catch errors from the get token function call.  Several of the callers need the parser error to be thrown as is, so if the error status argument is a null status, the parser error is thrown as is.  The first status enumerator was assigned to a value of one so that a null status is not one of the existing enumerators.

[branch misc-cpp-stl commit 950d3cf9ed]

Translator Exceptions – Expressions

The get expression function was modified to throw errors.  The return type was removed since a no exception return now indicates success.  The temporary error tokens for errors no longer need to be created.  The Done status that was previously returned for success was no longer used, so this status enumerator was removed.

All callers of the get expression function were modified for an error being thrown, which for the most part meant catching errors instead of looking for an error return status.  However, there was an issue with how errors were processed when an unexpected unary operator token appeared.

When an unexpected unary operator appeared, get expression returned an "expected binary operator or end-of-statement" error along with the unary operator token.  When another error was appropriate, the caller essentially ignored this error when it a unary operator, and threw the appropriate error.  Now that get expression throws an error, there is no token returned, so callers cannot look for a unary operator token.  The callers were were modified to look for this error and then throw the appropriate error.

[branch misc-cpp-stl commit 697a0d25d8]

Wednesday, November 26, 2014

Translator Exceptions – Tokens

The get token function used to get the next token from the parser was modified to throw errors.  The return type was removed since a no exception return now indicates success.  The temporary error token for an error no longer needs to be created.  The Good status that was previously returned for success was no longer used, so this status enumerator was removed.

All the callers of the get token function were modified for errors being thrown, which mostly included catching the error and throwing the appropriate error for the Unknown Token error. The get operand function just passes the error on to its caller (In this case, in addition to an Unknown Token error, a number constant error could be returned for non-reference token requests).

The get expression function was modified to catch the error from the get token function, obtain the error status and create an error token.  Th error token won't be necessary once this function is modified to throw errors (which is now the only function remaining now modified yet to throw errors).

In the LET translate function where the token after a reference token is not a comma or equal character, the token on top of the done stack was checked to see if it has the sub-string flag set, and if is does this item (a sub-string assignment function) is popped.  This is not necessary since once an error is thrown out of the translator, the translator instance is deleted along with the done stack.

[branch misc-cpp-stl commit 217de6d02c]

Tuesday, November 25, 2014

Translator Exceptions – Operators

The process operator function called by the get expression function was modified to throw errors.  This function returned Done status for a token that is not an operator, Good status for a successfully processed unary or binary operator, or an error status.  Since errors on now thrown, the return type was changed to a boolean, true for success and false for not an operator.

The process done stack top function is used to pop an item from the done stack and add a conversion code to the output if needed or report an error if the item can't be converted.  This function used by the process final operand, INPUT translate, and LET translate functions was  modified to throw the error and its return type was removed.  These callers were modified to just call the function and any error thrown will the passed to their caller.

Similarly, the process operator function was changed to just call the process final operand function, and its error is not caught passing any error thrown up to its caller, which is the get expression function.

The process first operand function was called to process the first operand of a binary operator, and push the unary or binary operator to the hold stack.  At first it was also modified to throw errors (again from the process done stack top function), but this function ended up being only a few lines, and since is only called by the process operator function, its code was just moved into the process operator function eliminating the separate function.

The get expression was temporarily modified to catch errors from the process operator function where the status is obtained from the thrown error, and an error token is created from the error column and length.  The error status and the token is returned as before.  This won't be necessary once get expression is modified to throw errors.

[branch misc-cpp-stl commit cf45a8e393]

Sunday, November 23, 2014

Translator Exceptions – Parentheses Tokens

The process parentheses token function used to translate tokens with parentheses token (arrays and user functions) for the get operand function was modified to throw errors.  This function only returned a Good status or an error status, so a return value was no longer needed.  Like the process internal function function, the get expression call was changed to throw an error if an error status is return, which is immediately caught.  This simulates the get expression function throwing an error, which has not been modified yet to throw errors.

Unrelated to this function, the comments for the other functions modified were updated to reflect the changes made so far, which was missed when these functions were changed.  The new expression error status function added for the process internal function function was moved to the support functions section of the source file.

[branch misc-cpp-stl commit 4f03c8bd77]

Saturday, November 22, 2014

Translator Exceptions – Internal Functions

The process internal function function used to translate internal function tokens for the get operand function was modified to throw errors.  Some of the changes made in the failed attempt to add exceptions throughout the translator routines around the get operand call used for sub-string assignments ended up in the code during the last commit.  This did not affect the functionality (since the regression tests had passed, it wasn't noticed).

The expression error status private helper function was added to determine the error status depending on whether at the last operand of the internal function, if the internal function has multiple arguments and if the bad token is a unary operator.  This function is called in three locations.

Since all errors are thrown, there was no longer a need for a return value.  The unary operator local variable was no longer needed and was removed.  The status and expected data type local variable declarations were moved closer to where they are used.  The status variable can be removed once the get expression function is modified to throw errors.

[branch misc-cpp-stl commit 640255514d]

Translator Exceptions – Operands

The get operand function used to get an operand was modified to throw errors.  The command routines either returned an error or a Done status, so once they were modified to throw errors, there was no need to return anything.  In addition to errors, the get operand function returned a Good status representing successfully getting an operand or a Done status representing  no operand token (an operator or command).

The return value of the get operand function was changed to a boolean where true represents successfully getting an operand and false representing no operand.  When a reference operand is requested this function will not return a false status, it will just throw an error.  So for the process internal function, LET translate, and INPUT translate routines, it is not necessary to test the return value since these only request reference operands.

There were three locations in the get operand function where errors were returned that popped the token on top of the hold stack before returning.  This was intended to prevent memory leaks.  This wasn't necessary since the translator cleans up the stack when the translator goes out of scope, so these pop calls were removed.

One of the errors for define function identifiers with parentheses tokens added the length of the token to the column to point the error to the parentheses of the token.  The token class add length to column access function was used for this.  Instead of using this access function (the only caller), the token length is added to the column when the token error instance is created and thrown.  This access function was removed.

[branch misc-cpp-stl commit 640255514d]

Translator Exceptions – Commands

The routines that process commands were the next modified to throw exceptions.  This included the get commands routine used to get command statements separated by colons.  For now, if the two calls to the get token function return an error (not good status), the appropriate error is thrown.  The get token function will also be modified to throw errors soon.  Since get commands now throws errors, it no longer needs to return a status so the return type was removed (along with the local status variable).

The process command routine handles the processing of one command and was modified to throw exceptions.  It calls a specific translate function for a command or the LET translate function if the first token is not a command.  The LET, PRINT and INPUT translate functions were also modified to throw exceptions.  The status return type was also removed and if statement surrounding the call to process command in get commands was removed (which will cause thrown exceptions to be thrown up its caller - the translator function operator function).

In the translator function operator function, a status is no longer returned from the get commands function, so assignment of the local status variable was removed.  And thrown exceptions are passed up to the caller of the function operator function (which will catch the errors).  For a successful return, the local status variable is set to Done.  This is temporary until the get expression function is also modified to throw exceptions.

The LET, PRINT and INPUT translate functions were restructured a bit.  Local variable declarations were moved to where the variables are first used.  Some of the error checking if statements were rearranged to ease the handling of errors (when the lower functions are modified to throw exceptions).  These changes came from the failed total translator exception changes.  The checks for the end-of-statement were also moved to a more logical place in each of these routines.

[branch misc-cpp-stl commit 7217ce58c6]

Friday, November 21, 2014

Translator Exceptions – Top-Level

Several convenience functions were added to the Token Error structure to make it syntactically easier to use.  These include a function operator function with no arguments for returning the status of the error, a function operator function with a status argument for checking if the error status is the passed status, and an assignment operator function taking a status value to assign to the error status.

The translator function operator function was modified to throw errors instead of setting a local status variable, which is then used to throw the error at the end of the function.  Eventually the get expressions and get commands functions will throw exceptions for errors and the local status variable won't be necessary.

The try block in the tester parse input function was reformatted where the try-catch blocks were moved outside of the forever loop to the function block.  The return statement in the catch block could then be removed.

[branch misc-cpp-stl commit c639347b07]

Translator Exceptions – Development Strategy

The next incremental change was initially difficult to see for adding exception throws throughout the translator routines.  Starting at the bottom by changing the get token function to throw exceptions was problematic because all callers would have to catch the errors and create an error token to hold the column and length of the error.  Starting at the top by changing both the get expressions (used for testing) and get commands functions to throw exceptions was problematic because all functions in between would also need to be modified.  An attempt to change the entire translator was made.

After the code was modified and corrected for compile errors, the tests were run, but there were many problems, which was not unexpected considering the large number of changes made.  Instead of trying to debug, the changes were committed to a temporary work branch (though not pushed to the official GitHub repository).  This is a scheme to use git to temporary save work:
git checkout -b work           (create a temporary work branch)
git commit -a                  (stages all changed files and commits them)
git checkout -b misc-cpp-stl   (restore all files before changes)
This saved the changes of the failed attempt and restored the original files.  Changes from the work branch could now be transferred to the working directory piecemeal.  For example, several changes were made to the Token Error structure in the token header file (details in the next post).  For more details on using this scheme, click Continue...

Sunday, November 16, 2014

Token Errors (Minor Refactoring)

The plan is to modify the translator routines to throw error exceptions when an error is detected.  Errors consist of a status, column and length.  For translator errors, the column and length will always be obtained from a token, so it made sense to add a constructor that takes status and token pointer arguments.

The Error structure was a plain structure with no constructors, so a default constructor is generated by the compiler taking arguments for the three member variables.  Once a constructor is added, this default constructor is no longer generated, so one was added.  The constructor taking status and token pointer arguments was also added.  Since the structure now has constructors, the member variables were renamed with the member "m_" prefix.

The Error structure was defined in the main header file.  This header file does not have access to the token header so that the new constructor can retrieve the column and length from the token, and an include couldn't be added for the token header because the token header already includes this main header file.  The Error structure was therefore moved to the token header file.  The name Error was a little generic so this structure was renamed to the more appropriate Token Error.

[branch misc-cpp-stl commit 97ce3592c3]