Table – Code Indexes

Indexes of table entries are used in the program code to represent a table entry of a code.  When the program code is read, then these indexes should be converted back to a table entry point, which is then used to access information about the code.  These code indexes should only be used in the program code.  However, the code indexes have been used throughout mainly within tokens.

As of the last change, they are no longer used within tokens.  The only remaining uses are a few code enumerators to get a table entry and some uses within the tester class.  The few code enumerators still used but will become code type enumerators when these code enumerators are combined with the type enumerators.

The tester class was using code indexes to determine if define functions contain parentheses or not.  When define functions are implemented, their dictionary entries will contain the number of arguments.  No arguments will mean that there are no parentheses.  Since this is not yet implemented, the code indexes were used temporarily to determine if parentheses were present.  The tester class should not be using code indexes.

A new Define Function No Arguments type was added for the three defined function codes without parentheses.  The tester routines were modified to use this new type instead of checking for code indexes.  There will be an equivalent temporary code type enumerator for this same purpose once the type enumeration and code index enumeration is removed.  A case for this new type was added to the translator get operand function at the current Define Function case.  With a new type, the expected results for parser test #2 (identifiers) needed to be updated.

[branch table commit c9c4ddeb44]

Tester – Function Operator/Exceptions

The Tester class is another one-use class that fits the pattern of the function operator class.  The main run function was changed to function operator function.  The caller in the command line constructor was modified accordingly with the instance renamed from tester to test as was done with the parser instances.

The Tester class also had an error mechanism where its error message member was set if an error occurred.  Both the constructor and the run function can generate an error.  The has error access function returned if an error occurred, and the error message access function returned the error message.  These functions were modified to throw an exception containing the error message (a standard string).  The function operator function (formerly run) no longer needs to return success status as a boolean.  This simplified the command line constructor since errors from both functions are caught with the same section of code.  The error message member and its access functions were removed.

The redundant void was also removed from tester function definitions that don't have arguments.  This was a practice I used when working with C code where the void in the arguments of a function definition indicates no arguments, as opposed to an empty parentheses, which could also indicate the old Kernighan and Ritchie (K&R) style function definition, which preceded the typed function definitions introduced with the first ANSI C standard.  This void usage is used throughout and will slowly be removed as there is no reason to use it anymore.

[branch misc-cpp-stl commit 738eba02e1]

Memory Testing Issues – Resolved

After discovering a default Mint 13 system (kubuntu backports not used) containing Qt 4.8.1 did not exhibit the sporadic memory errors, some further investigation was done.  The errors also did not occur when Qt 4.8.2 was built from source.  Before blaming the Qt 4.8.2 from the kubuntu backports, the build directory was wiped and the application was rebuilt from scratch.  The sporadic memory errors were no longer occurring, so there must have been a corrupted file in the build directory causing the errors.

Some memory testing investigation was also done on Mint 17 (based on Ubuntu 14.04).  The conclusion previously was that valgrind 3.10.0.SVN reported errors differently than 3.7.0 (Mint 13) or 3.9.0 (built from the latest source available).  The source for 3.10.0 is now available and 3.10.0 built from source on Mint 13 did not report any additional memory errors.  The issue was found to be with the ld-2.19.so library on Mint 17 (Mint 13 has ld-2.15.so).  This library appears to contain low-level memory allocation functions.

A different error suppression file was needed for the newer version of this library.  The CMake build file was modified to detect the presence of ld-2.19.so (either 32-bit or 64-bit).  If present, then a different error suppression file is copied to the build directory.  This error suppression file generated on Mint 17 is independent of the version of Qt (the Qt libraries are not referenced), so no configuration of the file is needed.

The error suppression files generated for Qt 4.8.2 and 4.8.6 are also independent of the version of Qt, however, the one generated for Qt 4.8.4 is not.  To create a suppression file that works with all versions (at least the ones tested), the suppression file needs to be configured for the specific version of Qt.  The file generated from Qt 4.8.4 was used, and this file works with Qt 4.8.1, 4.8.2, and 4.8.6 once all references of "4.8.4" along with the installation directory of Qt are changed.

There are now two error suppression files, one for Mint 13 (ld-2.19.so not present) and one for Mint 17 (ld-2.19.so present).  The one for Mint 13 is configured for the version of Qt detected, but the one for Mint 17 is not.  Mint 17 has Qt 4.8.6, so no other version of Qt should be present.  This commit was put in the develop branch since it is not related to the STL changes.

[branch develop commit 2fb73b6892]

Tester – Standard Input Streams

The run function of the Tester class either read input from the console (standard input) or from the specified test file.  A QTextStream was used to read the input and was either opened to the standard input device or the test file.  The standard input stream does not work the same way in that an input stream is not opened with a device.  Fortunately the sections of code for reading from the console or a file was separate, so input can be read directly from standard input (std::cin) or from an input file stream (std::fstream).

Since reading from standard input or input file stream returns a standard string, the rest of the code in the tester class (the various functions for processing input) was modified to use standard strings.  However, where these functions interfaced to the various classes not yet converted to use standard strings, the string is converted to a C-style string that is acceptable for a QString argument (via implicit conversion).  This is temporary until those classes are modified.

[branch stl commit 2e57346eab]  (The memory bug has been resolved, see next post.)

Tester – Options List

The Tester class contains a static options function that returned a string list of supported testing options.  The constructor of the Command Line class obtained the testing options and added them to the options it supports (version and help) to generate the usage string with the program name.    The testing options in the list were joined with a vertical bar ('OR') separator character using the join function of QStringList.

There was no reason to join the options in this way to generate the usage string.  The options function was changed to simply return its part of the usage string (with the vertical bar characters between the testing options) as a std::string.  The Command Line constructor was modified accordingly.

[branch stl commit fc61d53041]  (The Qt 4.8.2 memory bug persists.)

Command Line – Standard Argument List

The Tester class still contains functions that have an argument or return value that is a QString or QStringList that need to be changed.  The first of these functions modified was the constructor that had a string list argument for the command line arguments.  These arguments come from the Command Line constructor, which in turn is passed from the Main Window constructor that obtains them from the  Qt application instance.

The Main Window constructor was modified to convert the QStringList arguments to a standard list of standard strings.  There are Qt functions for converting lists and strings to the standard equivalents, but there is no function for converting a string list.  A simple for each loop was added to iterate through the argument list and add (emplace back) to a standard list converting each element to a standard string.  This list is then passed (moved) to the command line constructor.

The Command Line constructor was modified to convert the first argument (the program path name) to a base file name and store it into the program name member (which was changed to a std::string).  This first element is removed from the list.  The rest of the constructor was modified to treat this list and a std::list with one less element.  The is version output and is help option functions were modified similarly.  The program name access function was removed since there were no callers.

The Tester constructor was modified to take the program name and argument list (less program path) as separate arguments.  The program name is used as the initializer of the program name member and the rest of the constructor was modified to treat this list and a std::list with one less element.

A minor bug was also discovered and corrected in the translate input function of the Tester class where the header argument was not being output when it was present.  The bug was recently added when the RPN list text function was changed to a standard put stream operator function.

[branch stl commit 3d3f0d3e5c]  (The Qt 4.8.2 memory bug persists.)

Tester – Removed Translation Calls

The rest of the translation tr() calls were removed from the Tester class, which allowed the Qt translation functions declaration macro to be removed.  A number of changes were made to the Command Line class to support these changes.

The copyright statement was put into a constant C-string with a Qt translate macro, which allowed for a delayed translate call.  This string contained QString style place holders for the program name (or version string for the GUI about box) and the copyright year.  There was a static access function to obtain this string, which the callers filled in as desired.

These were replaced with a static copyright statement function, which internally writes to the std::ostringstream and then returns the result as a std::string.  The program name or version string is no included as the caller is now responsible for this.  The copyright year function used to access the year was removed and the year is used directly since this was the only user.  The main window about box function was updated for this new function.

The warranty statements were put into an array of constant C-strings with Qt translate macros, which allowed for delayed translate calls.  There was a static access function to obtain this array.  This function was only called from the Tester class (the GUI has a different statement).  These were removed and the Tester class now outputs these strings directly without translation.

While making these changes, it was noticed that the copyright year set in the CMake build file had not been updated for 2014.  The application version numbers (major, minor and patch), copyright year and release string are transferred to the source code via a template input file from which CMake creates a header file.  This file contained C-style preprocessor defines, which are not type-safe, and were changed to type-safe C++11 constexpr statements.

[branch stl commit d399ff535f]  (The Qt 4.8.2 memory bug persists.)

Tester – String Members

The next items in the Tester class changed were the string members, the program name, test file name, and error message members, which were changed to std::stringg.  The access function for the error message member was also changed to return a std::string.  There were two considerations with these changes.

The first consideration was the issue of [language] translations, which were used with various strings for generating error messages.  I decided for the testing part of the code, that translations are not necessary.  The testing code is for testing the internals of the application and any output does not need to be internationalized.  Therefore, translations (calls to the tr() function returning QStrings) will be completely removed from the Tester class, which started with these error messages.

The second consideration was the QFileInfo class, which is used to extract the file name from a file name path.  There is no equivalent functionality in the standard library.  It is not appropriate to create a similar function when it already exists.  The Boost C++ library does have equivalent functionality, but there is no reason to add another dependent library when Qt is already present and provides the functionality.

Since the goal is to remove all Qt dependencies from the Tester class, a base file name static function was added to the Command Line class (which functions of the Tester class already call; and is the owner of the Tester class instance).  This function takes a std::string file path input and returns a std::string base file name using QFileInfo.  It is expected that the Command Line class will continue to use Qt since it interfaces with the main application.

[branch stl commit 986268caf7]  (The Qt 4.8.2 memory bug persists.)

Tester – Option Enumeration Class

The Tester class is next to transition from Qt to the STL.  When enumerations were changed to C++11 enumeration classes, the Option enumeration in the Tester class was excluded because it was used for a loop iterator and contained a number of enumerators used for various purposes.  These needed to be removed before changing this enumeration to an enumeration class.

The none enumerator used to indicate no option was removed.  The default Option enumerator, Option{}, will be used to indicate no option.  The first enumerator (parser) needed to be set to 1 for this to work (like done with other enumeration classes).

The size of enumerator was used to dimension an array of strings for the names of each of the options, which is used to compare the command line file name to select the appropriate test option.  The option enumerators were used as indexes to set the elements of this array.  This method is error-prone (elements could be missed), and enumeration class enumerators cannot be used as indexes.  This array was changed to a std::unordered_map with an initializer list for each of the options except for the recreator.  The size of enumerator was removed since it is not needed to initialized the map.

The first and number of enumerators were used to bound the loop through the names of the options to compare to the file name from the command line to set the appropriate option.  The number of enumerator did not include the recreator name (there are no recreator test files).  This loop was changed to range-for iterating over the items in the new name map (why the recreator name was not put into the name map).  The starts with Qt comparison call was changed to the standard equal function using a C++11 lambda to do a case insensitive comparison.

The option member was set to the error enumerator when an error was detected to indicate an error.  For each error, the error message string was also set.  The error enumerator was not needed since a non-empty error message string can be used to indicate an error, and was removed.  The has error access function was changed to check for a non-empty error message string.

The option member was defined as an integer so that it could be set to the loop iterator variable (an integer).  Since the option enumerator value is directly accessible in the range loop (the key in the iterator), the option member could be properly re-typed as an Option enumeration class variable.  The test name member was changed to a standard string (to match value in the name map) and the arguments to the is option function was changed to standard strings.

[branch stl commit 5c3a7d6141]  (The Qt 4.8.2 memory bug persists.)

Tester – Standard Output Stream

The non-GUI classes use many strings that will be changed from QString to std::string.  In the Tester class, these strings are output using Qt text output streams.  The next step in the STL transition was to change the Tester class to use std::ostream instead of QTextStream.  This also required changes to the Command Line class, which provides the tester instance with the output stream to use for output.

The text stream member of the Command Line class (where either stdout or stderr was opened) was changed to a pointer to an std::ostream (which is now set to a pointer to either std::cout or std::cerr).  The cout() member function was changed to return a reference to the output stream and allowing the output stream member to be set from is argument (defaulting to std::cout).  Since nothing is opened, nothing needs to be closed so the coutClose() member function and destructor was removed.

Where ever a QString is output to the output stream, the toStdString() function was added since QString is not supported by std::ostream.  This is temporary until most of the QString instances are changed to std::string.  The usage string member was changed to a local std::string since no other code used it and it's access function was removed.

The text stream member of the Tester class was changed to an output stream reference.  The toStdString() was added to QString instances (temporary).  All uses of endl were changed to the new line character ('\n').  The endl manipulator (both Qt and STL) not only outputs the new line character, it also flushes the stream and this flush was not needed.

[branch stl commit 5cc2438dd0]

Memory Testing Issues

The next step is to start using C++ smart pointers (described in the next post).  One use for smart pointers will be to replace the complicated code used to track the handling of tokens and the reporting of various errors (for example, memory leaks for tokens not freed).  Smart pointers will make this code unnecessary.  Before starting I thought it was a good idea to make sure the memory test scripts still worked.

There were no memory errors reported when using Qt 4.8.1 (installed with Mint13 or Ubuntu 12.04) or Qt 4.8.2 (installed with the kubuntu backports to get the latest KDE).  However, when building the application with Qt 4.8.6 (the latest Qt 4.8 that was installed from source), memory errors were reported.  The CMake build file was set up to generate the error suppression file from a template using the detected version and directory of Qt.  This file was not tested with newer versions of Qt.

It appeared that with Qt 4.8.6, memory errors were reported differently (which was also the case with Qt 4.8.4, also built from source).  After examining the output for each version of Qt, a common error suppressions file template was created that works with each of these four versions of Qt (other versions were not tested).

As with GDB, valgrind (the memory testing utility) only supports the new debug symbols output from GCC 4.8.1 starting with version 3.8.0 (3.7.0 is installed in Mint13).  Version 3.9.0 is the latest available from source code.  However, version 3.7.0 appears to work fine, including within Qt Creator.  The new errors suppressions file also works with valgrind 3.9.0.  Click Continue... for details on installing 3.9.0 from source, but this shouldn't be necessary on Mint 13 (Unbuntu 12.04).  The valgrind 3.10.0.SVN installed on Mint 17 (Ubuntu 14.04) reports errors differently and does not work with this error suppression file (for now 3.9.0 would have to be installed from source).

[branch cpp11 commit a2069aae24]

Program – Decoder

Before the internal code of a program line can be recreated, the program code needs to be decoded into an RPN list.  Like the encoder, which is part of the program model class because it needs access to the dictionaries, the decoder will also be part of the program model.

The decode routine is given the line information of the line to decode containing the offset of the line within the program code and its size.  A new RPN list is created and for each program word in the line, a new token is created and assigned the code and sub-code of the program word.  If the code has an operand text function in its table entry (implying the code has an operand word), the operand text function is called to get the text for the token from the operand, which is assigned to the string of the token.  The token is added to the RPN list.  After all the words of the line are processed, a pointer to the RPN list is returned.

Like the encode routine, the decode routine is a private function within the program model class.  To access recreated lines of the program, a new line text routine was added.  This routine is given the index to the line and starts be retrieving the information for the line, which is passed to the decode routine.  The pointer to the RPN list returned is passed to the recreate routine of the recreator instance (which was added to the program model class).  The RPN list is deleted and the string returned from the recreate routine is returned.

The temporary check to prevent encoder test files from being used with the recreate output option (-to) was removed from the tester class.  In the tester run routine for encoder test files after outputting the code of the program and the dictionary entries, if the recreate output option was selected, each line of the program is output using the line text routine.

The expected results files for the three encoder tests were created from the encoder test results files with the output of the program added to the end.  All the encoder tests are recreated correctly.  The test script and batch files were updated to also test the encoder test files with the recreate output option.

[commit 1f24a70152]

Recreator – Interactive Testing

Up to now, the only way to test the recreator was by using the expression and translator test files with the batch test mode.  An interactive recreator mode was not implemented since the recreator only supported expressions and creating separate modes for both expressions and commands was unnecessary.  With support for commands (just assignments at the moment), an interactive mode for the recreator could be added.

Before implementing the interactive recreator mode, the translator, program unit and recreator instances (which were local variable in the tester class run routine), were changed to the member variables of the tester class.  As local variables, it was necessary to pass references to them between the various tester routines, which defeated the purpose of having a class.  The output stream is now given to the tester constructor, which is stored in a member variable so that it can be shared by all the class routines, instead of being an argument to the run routine and passed to the other routines.

The tester class was modified to support the new interactive recreator test mode, which is activated with the new "-tr" command line option.

The translate input routine was modified to accept a header string for the list of translated token output.  If this header string is not used, then "Output:" is used as before.  This routine was also modified to return the pointer to the RPN list if the header string is used.  Otherwise, the RPN list is deleted as before and a null pointer is returned, which is also returned when the input line has an error.

The new recreate input routine was added, which starts be calling the translate input routine with the header set to the "Token:" string.  If an RPN list is returned (no error detected), the RPN list is recreated and deleted.  The recreated output is prefixed with the "Output:" string as its header.

[commit 2dc4b17e97] [commit 0ccfb105e7]

Recreator – Testing

The recreator will take an RPN list as input.  Consideration was given to how the recreator will be tested.  There are already many tests for testing the translation of expressions and statements into RPN lists (the expression and translator test input files).  Since these are available, they will also be used for testing the recreator.

To avoid having to duplicate these into "recreator" test files, they will be used as is with a new "-to" command line option to activate the recreator on the translator output.  When this option is used instead of the "-t" option, the expression or translator input file will be translated as before, and then the RPN list will be passed to the recreator.

Since the recreator has not been implemented yet, to verify this code is working correctly, the same RPN text output is used, but prefixed with the "TEST:" string.  Encoder test input files are not supported, which will be added once the recreator is working with the expression and translator tests.

All of the test scripts and Windows batch file were updated to run all the expression and translator test files with the new "-to" test option.  Recreator output files have the ".out" extension to not conflict with the ".txt" extension used for the other test output files.  Recreator development will now commence.

[commit 9b52fc6d83]

Program – Enhanced Line Debug Output

The program view will get the same output as produced at the end of encoder test output, which includes the offset range, and the debug text or the error information (column, length and message) of the line.  Currently a program line with an error does not have any code associated with it (an input error).  However, for a code error, the program line will have been successfully encoded and stored into the program.  An example of a code error is a missing ENDIF to an IF.

The generation of the line offset range and error output was moved from the tester class to the program model debug text routine so that it can also be used for the program view.  This code was also modified to output both the debug text and the error information instead one or the other.  Since input errors have no code, this works as before.

Since the debug text routine is also used by the tester class encode input routine to just obtain the debug text for a line, the debug text routine was given an flag argument for whether to return the full information (offset range, debug text and error information) or just the debug text.

The error information is handled differently by the encode input routine, where the error column and length are used to point to the error.  This was modified to get a pointer to the error item for the line instead of the RPN list using the new error item access function added to the program model class, which returns a null pointer if the line does not have an error.

[commit 3fde3da2e0]

Program – Dictionary Debug Output

The program debug output shows the indexes of dictionary entries, but this is insufficient for showing if the dictionary entries were removed correctly and are placed on the free stack of the dictionary for reused.  Code was added to output the contents of each dictionary.

The debug text routine was added to the dictionary class that takes a header string as an argument.  After appending the header string to the output string, it loops through the dictionary entries and appends the index, use count and string of every entry with a non-zero use count.  After the entries, the indexes in the free stack are appended.  If any free stack item contains a non-zero use count, the use count is appended after the index.  Also, if the item has a non-empty string, the string is also append.  The strings of deleted entries should be cleared.

The debug text dictionaries routine was added to the program model class, which calls the debug text routine of each dictionary and appends each to the output string.  A call to this routine was added to the tester class run routine after the program model debug text function is called to output the program code.  The expected results for encoder test #1 and #2 were updated for the additional dictionary debug output.

[commit 11337cb673]

Program – Operation Testing

With the program line operations (insert, replace and remove) implemented, some automated mechanism was needed to test them using the command line test mode.  The encoder test mode was modified to accept a special syntax at the beginning of each test line to indicate a program operation.

The syntax starts with an optional '+' for insert line and '-' for delete line.  This is followed by a line index number indicating the line that should be inserted or deleted.  If there is just a line index number, the line is replaced.  The number is followed by optional spaces (though the number ends when there are no more digits).  The number must be within the valid range for the lines currently in the program.  The number is optional after a '+' in which case, the line is appended to the end of the program, the same as if the line does not contain this syntax.  After a '-' and its number, there must be no statement.

When using this syntax, the normal "Input:" and "Output:" lines are suppressed.  This is the difference between using a lone '+' (output suppressed) and no syntax (output not suppressed) for appending a line.  After processing this syntax, the characters are removed from the line, and the appropriate call to the update slot routine is made for the specified operation to translate, encode and perform the program operation on the line.  However, if a line does have an error, the outputs are not suppressed to report the error.

Encoder test #1 still operates the way it did before since none of the lines contain this additional operation syntax.  This test was copied into new encoder test #2 where a lone '+' was added to every line.  Several additional lines were added to test #2 to test various program operations.  (Note: this test currently produces a memory error that needs to be resolved.)

[commit e6ac67e6b7]

Additional Memory Issues

Some memory errors were reported when performing memory testing on the current source.  Checking previous commits back to the last tag reported the same memory errors, which was strange because the previous commits successfully passed the memory tests.  The memory errors were reported in libglib2.0.

I remembered that there was just an update for this library within the past week, which explained why these memory errors were previously not reported.  There must be some interaction between the Qt library and the new version of this library.  These errors were added to the error suppression file so that they will no longer be reported.  These extra errors will not affect the memory tests if the update for this library is not applied.

[commit 7ccd9ac05f]

Memory Testing / Minor Memory Leak

Since all of the tests are now working with the new translator (excluding the commands not yet implemented), it seemed appropriate to change the memtestn script to run all of the tests.  After changing this script, two memory leaks were discovered in translator tests #7 (Errors) and #9 (Semicolon Errors).  The memory leak determined to be occurring with sub-string assignment statements that contained an error.

The memory leak occurred because an RPN item was allocated for the sub-string assignment token, which is not appended immediately to the RPN output list.  The RPN item is left on the done stack, which the LET translate routine pops and pushes it's token to the LET stack and then deletes the RPN item.  However, if the next token that should be a comma or equal token is not or a parser error occurred, then this does not occur.  The error clean up code assumes that all RPN items on the done stack have been added to the RPN output list, so only the items in the output list are deleted.

This problem was corrected by slightly rearranging the code in the LET translate routine where if there is an error with the comma or equal token, and the top of the done stack contains a sub-string assignment token (that has not been added to the RPN output list), then the done item on top of the done stack is popped and deleted, which deletes the RPN item and its token(s).  With this change, all of the tests with the new translator have no memory errors.

[commit 229af22a78]

New Translator – Remaining Issues

There are at least two major issues remaining in the new translator routines that are impacting the failures in the other LET tests, though test #10 (Expression Errors) passes.  The first issue is how operators are detected, specifically that some tokens that are considered operators, but are not expression operators (for example, open and closing parentheses, commas, semicolons, colons, the remark operator and end-of-line tokens).
The second issue is part of the design of the new translator has not yet been realized, namely that when getting an expression, if given a particular data type, it should check that the expression is of that type, or can be converted to that type via a hidden conversion code, else return an error.  Currently the data type is only being used for reporting errors with operands.  Correcting and implementing these issues has been a major undertaking, so a bunch of preliminary changes were made leading up to these changes.

Previously, all tokens with a command or operator type were considered operators.  This was necessary for the token-centric old translator, because some commands like THEN and ELSE needed to be considered operators since they can come at the end of expressions.  The first change made was to remove the Token::isOperator() access function along with the static Token::s_op[] array used by the access function.  Uses of the access function were replaced with the Token::isType() access function.  Not all uses required checking for both operator and command token types.  [commit ffd1ba6462]

Since the number of old translator expected results files are growing (because of corrections and changes to translations), the regtest script was modified to look for an old expected results file (ending with an 'o') and comparing to that file if found instead.  An old expected results file for translator test #3 was also added (print functions used in expressions do not report through the closing parentheses.  [commit 7aef54c63e]

The fact that the open parentheses operator token was configured as a unary operator in the table was going to cause issues with the new translator routines.  Therefore, it is now not configured as a unary operand and the old translator routine was modified to look for an open parentheses token before checking if the operator was unary (the check was simply moved from after to before).  [commit e75adeddcb]

The segmentation fault on expression test #3 was becoming a nuisance, and was only caused by the additional of more error tests for the new translator.  Therefore, the problems with old translator routines were corrected, including considering the initial state as an operand state, allowing for an empty command stack in the case of expression mode, and assuming the Any type at the beginning of an expression before any operands or unary operators are received.  The memtest script was also updated to allow for comparing old translator expected results files.  [commit f0f6cb57f7]