Minor Build And Test Issues

A build issue was discovered where CMake does not create a release string if the git command is present but the git repository is not (for instance when building from a downloaded archive).  The git describe command was returning an error and no release string.  CMake now detects this situation and sets the release string the same as if no git command is found.

A test issue was discovered on Windows when building from a downloaded archive, which contains Unix format files (newline only) and not DOS format files (CRLF).  When the program is run from the regression test script (the program builds fine), the output files are in DOS format, but the compares fail because the expected output files are in Unix format.  The cmp command was changed to the diff command, which has an option to ignore the difference in the line separators (the ‑w ignore white space option).

All these changes have been pushed to GitHub and because of the build and test issues found, new tag v0.2‑2 was added.  The changes to the Parser were complete (with the attempt to compile next) before the text stream detour and these other minor issues.

[commit 0e85c83d56] [commit 2be4bd2f91]

Qt Transition – Strings (File Input)

There was an issue in the way the test files were being read.  When the test code was modified to use Qt, the QFile class was used to read the file, where the function used to read a line is actually inherited from the QIODevice class, which QFile is based on.  This function returns the line into a QByteArray, which was easily converted to a character array currently used by the Parser.

However, the Parser is being converted to use the QString class.   The QString class actually contains QChar characters, which supports 16-bit Unicode.  Reading the file as QByteArray would need to be converted to a QString and would not support Unicode text files.  After doing some research, it was found that files can be read as Unicode text using the QTextStream class (into a QString).

Therefore, the file reading code was modified to use a QTextStream (where a pointer to the QFile instance is given in the constructor).  This will also be used for the standard (console) input for the interactive test modes.  The file is opened the same way, but the at end of file check and read line routines from QTextStream are used instead.  This read line routine also strips the line separator from the line (ether newline on Linux or CRLF on Windows).

Temporarily, the QString line is converted to a QByteArray, a null character is added to the end and then converted to character array (constant character pointer) to pass to the Parser.  So that a type cast to a char * was not needed, the argument and variables in the Parser were changed to const char *, which is fine since the Parser does not modify the input line.

[commit 993aa66765]

Qt Transition – Strings (Begin)

The last class to replace is the String class.  The Qt equivalent class is QString and related QByteArray.  As with the List class, the transition will be done in steps.  The Token class would be first.  The first primary user of the Token class is the Parser class, which is also very dense with string operations.  The QString class has many useful functions, and these should help simplify the Parser functions.

Before tackling the Parser, the test code will need to be able to handle the change from String to QString.  The are two other major items in the test code that also needed to be transitioned to Qt, namely file handling, console input and console output.

The Qt QFile class handles file handling and is much simpler to use then the c file handling that was being used.  The QFile class can also handle reading input from the console.  There is also the QFileInfo class that contains many useful functions, but the ones used here were for parsing file names (including the program name).  There are single functions for extracting the path, file name, base file name without extension.

The QTextStream class handles output and is very similar to C++ stream output.  In the main source file, the standard output is opened as a QFile and is attached to a QTextStream named cout.  This text stream is then passed to all the functions that need to do output.

As described in the previous post, the opportunity was taken to rename many of the variables and functions to the Qt naming convention.  Now that the callers to the Parser (and Translator) have been transitioned to Qt, it's time to work on the Parser (and Token) classes.

[commit d94cd3c632]

Qt Transition – Qt Naming Convention

Part of the Qt transition is will be use the Qt naming convention for classes, functions and variables.  Specifically, Qt uses camel casing where the first letter of words in a name are upper case, where the first letter of the name is lower case (except for class names where the first letter is upper case.  There is also a convention for naming memory variables along with there access functions.  This is best shown with this example class:

class MyClass {
    int count;
    int some_value;
public:
    int get_count(void) {
        return count;
    }
    void set_count(int _count) {
        count = _count;
    }
    int get_some_value(void) {
        return some_value;
    }
    int set_some_value(int _some_value) {
        some_value = _some_value;
    }
    bool empty(void);
    bool data(void;
    void process_some_data(int data, int more_data);
}

Note the underline character that is used for separating the words of the variable and function names and the names of the access function.  The arguments on the set functions were also prefixed with an underline to make the name unique from the memory variable.  Using the Qt naming convention, this class definition will look list this:

class MyClass {
    int m_count;
    int m_someValue;
public:
    int count(void) {
        return m_count;
    }
    void setCount(int count) {
        m_count = count;
    }
    int someValue(void) {
        return m_someValue;
    }
    int setSomeValue(int someValue) {
        m_someValue = someValue;
    }
    bool isEmpty(void);
    bool hasData(void);
    void processSomeData(int data, int moreData);
}

Note the "m_" prefix on each member variable, but not on member functions.  Also note that getter functions do not begin with get.  And other than the "m_" prefix, underlines do not appear on any names.  Also note that boolean member functions that return status are prefixed with is or has.  As the Qt transition continues, this naming convention will be used (and this has already started with some of the changes made so far).

Qt Transition – Stacks

The next class to replace is the Stack class.  The Qt equivalent is the QStack class.  Internally the Stack class is implemented with an array that grows as needed.  The QStack class is based on the QVector class, where its members are in contiguous memory (essentially also an array).

The Stack class has two functions not present in QStack (or the underlying QVector).  The first is a push function that takes no arguments, which is used to add an element to the stack without copying a value to the new element (which was shortly followed by setting this new element using the top function).  The second is a null pop function that removes the top element from the stack without actually returning its value (like the regular pop function does).

The QStack (nor the QVector base) class have similar functions, but can be simulated using the resize and size functions of the QVector class.  For example, this line would be used to add an element to the top of the stack:

stack.resize(stack.size() + 1);

Similarly, the top element of the stack can be removed by using subtraction in the line above.

The four stacks using in the Translator were replaced with QStack using the substitutes for the absent functions.  The stack.h header files was removed (along with the program that tested stacks).  The next change in the Qt transition will be more radical, so this would be a good place to add the v0.2-1 development tag (and archives will be available).

[commit 5d7c634713]

Qt Transition – Error Lists

The remaining item that uses the List class are the error lists generated by the Table class constructor (for errors detected in the table entries).  This constructor added errors to an error list, and at the end of the constructor, if there were any errors, the constructor threw an exception containing the pointer to the errors list.  The token initialization routine also previously threw an exception for any token status errors detected, but this code was removed when the enum.awk script was implemented to generate the token status enumeration and it detects any errors.

Because there were two types of errors and both were handling errors the same way, an Error template was implemented that could be used for both code (table) and token status errors.  This template also handled outputting the errors using a print function that was passed to it.  This design was also rather complicated.  I mentioned previously that Qt did not support exceptions, but this was not correct.  Qt can be used with exceptions, but none of the Qt classes and functions throw exceptions themselves.

Since the exception and error template design was complicated, it was removed.  For the table initialization to be able to return errors, the code was moved from the constructor to a new initialization function.  As for the error list to return, the QStringList class was used.  This class is a specialized list class, equivalent to QList<QString>.

Hit Continue... for details of the new table initialization implementation.  All lists have now been replaced with Qt equivalents.  The list.h header file was removed (along with the program that tested lists).  The next replacement will be the various stacks.

[commit a27e456222]

Qt Transition – Token Lists

There were two lists used to keep a list of allocated tokens and a list of tokens that were deleted multiple times.  These lists were used to detect token memory leaks (tokens not released).  To support this detection, the new and delete operators of the Token structure were overloaded.  In addition to allocating the memory (using regular new), the token was also added to the allocated list.  When deleted, the token was removed from the allocated list.  After processing a line, any tokens still in the allocated list was considered a leak.  Similarly the deleted token list kept track of any tokens that were deleted more than once.

Apparently, the Qt classes (including QList, QLinkedList and QVector) do not interface with overloaded new and delete functions (obscure compiler errors result).  Since another method is now being used to detect memory leaks (valgrind), this detection code is not essential.  The advantage of this code was that it would output the exact tokens that were not released or were deleted twice.  This was nice when the Translator was first implemented and debugged, but now that Translator code is fully working, this level of detail is probably (and hopefully) not necessary.  Therefore, these token lists were removed along with the overloaded new and delete operators.

[commit  d21d3d38ad]

Building On Windows With Qt

In order to build the program (that now requires Qt), CMake needs to be able to find the Qt files.  It accomplishes this by looking for the Qmake executable (qmake.exe on Windows) even if it is not actually used, but its location is used to determine where Qt is installed.  To find the Qmake executable, CMake searches the directories in the execution path.

On Linux, the qmake executable is already in the standard directories.  On Windows, the directory for qmake.exe needs to be added to the execution path.  The instructions for adding a directory to the execution path was given in the post on October 20.  The directory C:\QtSDK\Desktop\Qt\4.8.1\mingw\bin needs to be added.  If using the MSYS command line, the directory /c/QtSDK/Desktop/Qt/4.8.1/mingw/bin needs to be added to the path (see post on October 7).

QtCreator can be used to access the git repository (see post on October 20).  To switch to the latest development branch (branch0.2 at the time of this post), the branch needs to retrieved from the repository on GitHub.  On the Tools menu, select Git and then Pull.  Only local branches can be checked out, so a local branch needs to be created.  On the Tools menu, select Git and then Branches....  Select branch0.2 under origin and click the Add... button.  Click OK on the next Dialog.  This will create local branch0.2, which can now be selected and checked out with the Checkout button.  Select Close to dismiss the Branches dialog.

To build, first select Run CMake on the Build menu and make sure the Generator is set to MinGW Generator (MinGW (x86 32bit)).  Now click the Run CMake button.  As mentioned previously, sometimes a bunch of errors occur the first time.  Trying a second time causes no errors.  If at any time there are problems running CMake, deleted every file in the qtcreator-build and try again.

There is a problem in running the regtest script in Windows XP when the build directory is under the current user directory under the C:\Documents and Settings\ directory.  The problem occurs because there are spaces in the directory name, which causes bash to incorrectly process the line.  There is no problem on Windows 7 because the user directories are under the C:\Users\ directory (which has no spaces).  There is also no problem if the build directory is under the user home directory within MSYS (again no spaces).  As for regtest.bat, this batch file requires the build to be performed in the source directory.  Also note that the new memory test script is not available on Windows.

Qt Transition – Translator RPN List

Before embarking on the memory issues, the Qt Transition was started by changing all List to QList and changing all the access functions accordingly.  However, that was going to take while, so those changes were abandoned to start a new approach in only changing one list at a time, and running the new memory test script after each.  This started with the Translator's RPN output list.

Wow, what a learning experience.  One nice feature of the current List class was being to obtain a  pointer to a particular element in the list.  Since the underlining code was implemented as a linked list, the element pointer could be used to access elements previous and following the element.  Though admittedly, the syntax for these element pointers was rather involved and therefore confusing.

The QList class has several ways to access the elements including by index (something List was not capable of) and with iterators (similar to but not the same as the pointers to list elements).  In the Translator, the list element pointer was used in several places, which to modified to use QList.

1. The done stack used to hold pointers to tokens added the output list, which are used when processing operators and functions when checking data types of the operands and arguments.  The done stack actually contained pointers to the output list elements (of RPN items containing the token).  There was no reason to have a pointer to the list element here, so it was changed to just point to the RPN item.

2. The RPN item structure contained an array of list element pointers for each of the operands of the token.  Operands are only kept for identifiers with a parentheses token (until it is determine whether it is an array of function), defined functions with parentheses token (needed to check with the functions arguments), and string operator tokens (until it is determine whether the operands are temporary or not).  There was no reason for these pointers to point to list elements, so the array was changed to an array of RPN item pointers.

3. The command stack used to hold the current command contained a pointer to a list element.  Currently, the INPUT command handler uses this pointer to insert tokens into the output list.  Several unsuccessful attempts were made to use Qt iterators for this pointer, but this only caused weird memory issues to be reported by valgrind (though the code appeared to work).  In the end, an index into the output list was used since QList items can be accessed by index.  In one location of the code, an iterator was used to access the last element in the list, and if a hidden token, stepped back to the previous element.  An iterator worked here because it did not require saving the iterator value and trying to use it later, which appears to cause problems.

One List to QList change is complete; now on to the rest.  The CMakeLists.txt file was also modified to support building with Qt, which involved finding the Qt package and added the Qt libraries to the executable (this works on Linux, but Windows requires some additional steps before it can be compiled).

[commit 34cbd66cd9]

Third Translator Memory Issue

The memory issue reported for translator test 7 was the same message as test 6 except on a different line, which was another if statement.  Again through the process of elimination, the statement causing the error was identified to be MID$(A$ B, which is expected to produce an expected comma error at the B token.

After briefly studying this if statement, the problem was identified and is similar to the previous problem.  This if statement, in the process binary operator routine, was checking if the token (within a sub-string assignment) was not a comma.  Again, it should not have been checking the token code before checking that the code was valid (since not all token types have a code).  The B token is an identifier with no parentheses token type and code is not used.

The if statement was corrected by adding a check if the token has a table entry (and therefore a valid code) and it is not a comma.  Now all the memory issues were resolved.  I also now understand what the Conditional jump or move depends on unintialised values(s) error is indicating.  Apparently, the Analyzer (valgrind) is checking for more than just memory leaks, it is also checking when a variable is being accessed, but it hasn't been initialized, which was the case for these two if statements.

Rechecking all the tests, all the memory issues were resolved.  However, upon running the regression tests, translator test 7 was now failing.  The problem occurred with the statement above, which was now reporting an expected operator or comma error, which was wrong because with a sub-string assignments, an operator is not allowed after the string variable identifier.

The if statement was modified so that either the token does not have a table entry or the token is not a comma.  All tests now pass.  To simplify (and automate) this memory testing, a new memtest script was created, but only for Linux as it requires the valgrind program.  This new script is based on the regtest script and also checks the regression test results along with checking for memory issues.  Now back to replacing the List class with the QList class...

[commit 1d427d7d98] [commit da0014b34d]

Second Translator Memory Issue

While the first memory issue was a simple memory leak, the second issue was much more difficult to resolve.  The issue on translator test 6 was reported as a Conditional jump or move depends on unintialised values(s).  Clicking on this showed the source line and another message reporting Uninitialised value was created by a heap allocation.  The line for this message was in the token new function where the memory for the token is allocated.  Curious that the token allocation checks were not reporting any token leaks.

Through the process of elimination, the statement (out of 42) causing the error was identified to be PRINT A(TAB(10)).  Looking at the line indicated by the first message (an if statement) did not make it clear what the issue was.  So the code was stepped through with the debugger to identify the problem, which was caused by an incorrect check for the item on top of the hold stack, which happen to be the line reported with a problem.

This if statement checks if a print-only function is found in an expression, which should be reported as an error since these functions are only valid in a PRINT command.  The check is for either the current command is not a PRINT command or the token on top of the hold stack is not the null token (any other token indicates the print-only function is in a parentheses, array or function - an error).

The problem was with the null token check as it was only checking if the code of the token was not the null code.  However, not every token type has a valid code, specifically, constants, identifiers (with and without parentheses), and user defined functions (preceded by FN with and without parentheses).  This check was replaced with a call to a new token function that checks to see if the token is a null token, which first checks if the token has a table entry (and therefore a code) and then checks the code.

This corrected the problem with translator test 6.  All the expression and translator tests were rechecked with the Analyzer.  Translator test 7 was still reporting a memory issue.

First Translator Memory Leak

The first memory leak, which occurred on every expression and translator test, was easy to identify and correct as the Analyzer pointed directly to the problem.  The memory issue was reported in the translator start() function with the RPN (reverse polish notation) list allocated there.

For expression test 1, 13 blocks were reported lost, which exactly coincided with the number of test expressions.  This made it obvious that the memory allocated for the RPN list object was not being released.  In the translate input routine, after the resulting tokens in the RPN list were output, the memory allocated for each token was released.  However, the RPN list object itself was not released.

After the correction, all the expression and translator tests were rechecked with the Analyzer.  Translator tests 6 and 7 were still reporting memory issues.

Automatic Building For Debugging

When using Qmake with QtCreator, both Release and Debug builds are configured and it is easy to switch between the two.  CMake supports several build types including Release, Debug, Release With Debug Info and Minimum Size Release.  The default is blank (which is probably similar to Release, but definitely does not turn on debug information).  However, the CMake build types are not directly available in QtCreator except by specifying the CMAKE_BUILD_TYPE variable when running CMake.

During development it is obviously desirable to build the program with debug information.  The CMAKE_BUILD_TYPE variable could simply be set to Debug in the CMakeLists.txt file.  But then this would need to be changed when a release is made, so this is not a good solution.

Instead, the CMakeLists.txt file was setup to look for a CMAKE_BUILD_TYPE environment variable (CMake can access system environment variables).  If this environment variable is set, then the CMake variable is set to the environment variable unless it has already been set.  If the environment variable is not set, it behaves the way is did before.

This environment variable can be set in QtCreator by going to the Projects screen, selecting the Build Settings and adding this environment variable to the Build Environment with the desired Debug value.  The Release With Debug Info (RelWithDebInfo) could also be used, but can make stepping through the program confusing since most of same optimizations are turned on as with Release and the compiler can rearrange statements for efficiency (execution will appear to jump around) or optimize out variables (the values of which cannot be viewed).

[commit 7ee8e7483c]

Finding Memory Leaks (Linux)

QtCreator can be used to find memory leaks using the Analyze mode.  This requires the valgrind program, which can be installed via the valgrind package on Ubuntu based distros.  Unfortunately it looks like this program has not been ported to Windows (MinGW), so Windows developers are out of luck (at least when using MinGW with the Qt SDK).

In order to use valgrind, the program to check must be compiled with debugging information.  To do this with CMake under QtCreator, in the Run CMake dialog, the Arguments line needs to be set to ‑DCMAKE_BUILD_TYPE=Debug.  This is a nuisance because it needs to be done every time CMake is run the first time and there appears to be no way to automate this inside QtCreator.  So an alternate scheme was devised using CMake (will be described in the next post).

Once Analyze mode is selected (by the icon on the side panel or Ctrl+6), the Analyzer panel will appear.  In this panel, the mode needs to be changed from QML Profiler to Valgrind Memory Analyzer.  The program is started using the start (play) icon (left side of Analyzer panel toolbar).  Once the program ends, any memory issues will be reported.

After using the Analyzer to learn about list element deallocation, I thought it might be a good idea to check the ibcp program for any memory issues.  There is already an implementation to detect token memory leaks accomplished by overloading the new and delete operators for the Token structure, but there are many other memory allocation operations in the program.

So this process was started for each parser, expression and translator test.  For each, the run arguments were set (Projects page, Run Settings) and the Analyzer was run.  This were no memory issues on the parser tests, however, some problems were found on the translator tests, which will be discussed in following posts.

List Class Replacement (Begin)

The List class was the first implemented for this project, so it is fitting that it will be the first to be replaced with the transition to Qt.  Qt's list class is named QList.  The functionality is not much different then the home grown List class, though most of the member functions have different names.

The approach being used was to first replace all cases of List with QList and then search for each List class member function name and replace it with QList's version.  In some cases, the code needs to be written a bit since the QList functionality is slightly different.  For example, while QList also has a first() function, it must not be called if the list is empty (must check if it is empty first), while the List class first() function allows for an empty list (returning a null pointer).

The first complication came in how to deallocate items in the QList.  The question was, does this happen automatically when leaving scope, is deleted or via QList's clear() member function.  Neither was the case where the list is a list of pointers to allocated elements.  Each element needs to be deleted (same as the case with the List class).

A small test program was written to evaluate and confirm this behavior.  As part of this evaluation,  a method to detect memory leaks was employed that is kind of built into QtCreator (at least on Linux).  This lead to quite a few detours, which will be the subject of the posts that follow...

Unique Release Number Implementation

To see if the git command is available, a find_program command was added to the CMakeLists.txt file, which sets the CMake variable PROGRAM_GIT.  An if command was also added that checks if this variable is set, which then executes the command using the  execute_process command for the describe sub-command in the project source directory setting the CMake variable ibcp_RELEASE_STRING with the resulting string.  The option OUTPUT_STRIP_TRAILING_WHITESPACE was needed to remove the trailing newline.

If the git program was not found, then the ibcp_RELEASE_STRING variable is set to "v" followed by the major and minor release numbers separated by a period.  If the patch release number is less than zero (development tag), then the patch number is appended to the release string (the dash is present because the number is negative).  Otherwise, the patch number is appended with another period separator.  The final release string, from either source, is output to the CMake output log.

In implementing this if command, it was realized that it was not necessary to test the PROGRAM_GIT variable for the string to be equal to "PROGRAM_GIT‑NOTFOUND" as was done with the awk program because this is one of the tests performed by the CMake if command.  Therefore, the if command for the awk program was simplified.

A define for ibcp_RELEASE_STRING was added to the ibcp_config.h.in file (used to auto-generate ibcp_config.h) with the contents of the CMake variable surrounded by quotes, to form a string constant that can used in the source code.  The ibcp_version() function was modified to output this string constant instead of the major, minor and patch numbers.  The "v" part of the string is not output (because "version" is already being output).  Finally, the GPL header was removed when outputting the version number to be consistent with the output of other programs.

New branch0.2 was created.  The release string produced at this latest commit will be v0.1.16b-1-g1bba2c3, which shows the most recent tag (v0.1.16b), the number of commits beyond this tag (1) and the short commit ID (1bba2c3).  The "g" in front of this ID stands for git (other letters would stand for other software configuration management systems).

[commit 1bba2c335c]

Unique Release Numbers

With a new release numbering scheme defined, there needs to be way to get the current version number into the program for output with the -v command line option (and eventually in the Help/About box once the GUI is implemented).  The version number should also be unique during development since not every commit will be tagged.  The goals are:

1. Use the current tag if at a tagged commit
2. Represent when not at a tagged commit during development
3. Use the current release number assigned for archive downloads
4. Allow for developmental (dash) and patch (period) numbering

The inclusion of the third goal will be clear shortly.  It turns out that the exact desired release string can be obtained using the git describe command, which returns the name of the most recent tag.  When beyond the most recent tag, the number of commits beyond the tag plus the short form of the current commit ID is appended.

This works as desired when the git command is available and the git repository information is present, but for the third goal, the downloaded archives have no git information.  Since the download archives are only available at tagged commits, the release number set in the CMakeLists.txt file can be used and will match the tag at that commit (assuming these variables were set to the same values as the tag).

For the last goal, there are two cases, git repository information present and not present.  When making tags during development, the tag name format will be releaseX.Y‑Z (note the dash).  The git describe command will pick this tag name (and append the rest if beyond that tagged commit).  When the git repository information is not present, the major, minor and patch release numbers set in the  CMakeLists.txt file will be used.  To handle developmental (dash) numbering, negative patch numbers will be used.

New Release Numbering Scheme

In light of the two recent problems discovered after Release 0.1.16 was made, a new release and branch numbering scheme is needed.  Technically, the third number on the release number should be for patches of a release.  Instead of using 0.1.16a and 0.1.16b, 0.1.16.1 and 0.1.16.2 could have been, but wasn't because the version numbering is currently only setup for major, minor and patch numbering, so the "a" and "b" were used (though it wasn't setup for this either).

As for branches, there is no reason to have a branch for every patch number (like with branch0.1.14, branch0.1.15 and branch0.1.16), though these were not really patches though the patch number was increasing.  Therefore, going forward there will only be branch0.2 for the developmental release 0.2 series.  As something worthwhile is completed, a tag with this number plus a dash number will be added, for example 0.2-1, 0.2-2, etc.  This will be the equivalent of the "-pre-" (and "-dev-" before that) that were used previously.

When the release series is completed, the release will be given the number 0.2.0.  (It's too early in development to worry about release candidates, but when that time comes, the "-rcX" format will be used.)  Any patches needed for a given release will then be given patch numbers 0.2.1, 0.2.2, etc.  Now to generate a unique version release number at each commit taking into account using the current tag for a tagged release...

Regression Test Script Problem

Another problem was discovered, this time with the regtest script.  For convenience, my PATH variable contains the current working directory ("."), so I didn't notice this problem.  By default (and for safety), the PATH variable does not contain the current working directory.  So the regtest script fails to find the "ibcp" program.

Therefore, regtest (actually regtest.in) was modified with a "./" in front of ibcp so that the regtest script will run the program in the current working directory.  No new files were uploaded to Sourceforge, but a new commit and tag release0.1.16b was pushed to GitHub.  The Windows batch file regtest.bat is not affected since Windows by default will look in the current working directory for a program.

Build Issues Discovered

While writing the procedures for building with QtCreator, some issues were discovered.  The first was with running CMake on Windows.  The first time it produces errors no matter which generator is selected.  It also appeared that adding "‑G "MSYS Makefiles" caused it to work, but adding this before still caused errors.  It turns out this was not necessary, simply clicking the Run CMake button a second time worked with no error.  The previous post about this was updated.

The next problem was when trying to use the included MinGW (with GCC 4.4) in the Qt SDK.  It was still trying to use the static linking, which is not supported in versions before GCC 4.5.  It turns out there were two problems.  First, the check to whether to add the static linking options should have been greater than 4.4, but was incorrectly less than 4.5.  Second, the add_definitions command used in the CMake file adds compiler options, not linker options.  The command add_target_properties should have been used with the LINK_FLAGS property option.

Due to these linking issues, the binaries posted on Sourceforge were not linked with static linking.  This is not an issue with Linux since the required libraries will be present.  However, for Windows, a missing DLL message will occur on a system without the required libraries.  Therefore, a new binary zip file was updated (labels 0.1.16a).  The update executable was actually tested on a Windows XP system without the require libraries this time. The sources and Linux binary were not updated.  If building from source, the required libraries will be present.  The repository on GitHub was updated with new tag release0.1.16a.

The final problem is in using the MinGW installed with the Qt SDK.  Even though the static linking problems were corrected, the included MinGW cannot (alone) be used the build the project.  The issue is that the awk utility is needed to create the auto-generated header files and there is not included awk utility with MinGW.  It is included with the MSYS package.

Running in QtCreator

After the program has been built it can be run or debugged in QtCreator.  For now, command line arguments need to be added before running the program, otherwise it will only output a usage message.

Command line arguments are set on the Projects screen (the Projects icon on the side or Ctrl+5) by selecting the Run Settings button along the top of the screen.  For example, under Run, enter ../thunder422‑ibcp/test/translator01.dat to run the first translator test (assuming the default directories were used up to now).

Again there are multiple ways to run the program (play icon on lower side, Ctrl+R, Run on the Build menu).  Same for running in the debugger; though note that in order to trace through the program, a breakpoint needs to be set first or the program runs (in the debugger) until it exits.

Building With QtCreator

Now that it has been configured for all the tools, everything can be done inside QtCreator.  Start QtCreator and select New Project... on the File menu.  In the New dialog select Project from Version Control and then Git Repository Clone.  After clicking the Choose... button, the git repository can be selected.  For Clone URL: enter https://github.com/thunder422/ibcp.  If desired, change the Checkout path.  The Checkout directory will default to thunder422-ibcp.  Click the Next button and the repository will be cloned.

Once finished, the CMake Wizard dialog appears asking for the Build Location.  The default directory can be used.  The next dialog of the wizard is to Run CMake.  On Windows, the Generator: needs to be changed to MinGW Generator (MinGW (x86 32bit)) assuming MinGW 4.4 was installed with Qt SDK, otherwise this is the only generator available.

Clicking the Run CMake button at this point causes a bunch of errors.  Adding ‑G "MSYS Makefiles" to the Arguments: line appears to resolve the problem.  Adding this before clicking the Run CMake button the first time does not make it work though clicking a second time does.  Clicking the Run CMake button a second time works.  I have no explanation for this at the moment.  This is a similar problem that caused issues with NetBeans and CMake.  On Linux, clicking the Run CMake button is all that is necessary.

Adding ‑DCMAKE_BUILD_TYPE=Debug to the Arguments: line turns on debug information to that the debugger can be used. After CMake is run, the program can be built.  There are a number of ways to begin the build, including the Build menu, Ctrl+B and the hammer looking icon at the bottom of the tool bar on the left side.  The build can be monitored by clicking the 4 Compile Output button along the bottom of the screen.

Post Qt Installation Setup (Windows)

After installation of the Qt SDK, the tools (git, MinGW and CMake) need to be integrated with QtCreator.  On Linux, no further configuration is needed as all tools will be located where QtCreator can find them.  However, on Windows, this is not the case.

For git, the binary directory for the git tools need to be in the execution path.  On Windows XP, right-click on My Computer and select Properties.  On the Advanced tab, click the Environment Variables button.  On the lower pane under System Variables, find and selected the Path variable.  Click Edit and add C:\Program Files\Git\bin somewhere on the Variable value line (make sure to add the semicolon separator).  On Windows 7 the instructions are similar.  Start by right-clicking Computer and select Properties.  Now select Advanced system settings, go to the Advanced tab and follow the same instructions except add C:\Program Files (x86)\Git\bin to Path.

Now start QtCreator.  QtCreator should have automatically found the external MSYS/MinGW 4.6.2 previously installed.  On the Tools menu select Options, go to the Build & Run page, and select the Tool Chains tab.  Under Auto-detected there will be an entry Mingw as a GCC for Windows targets (if the MinGW 4.4 was installed as part of Qt SDK) and an entry MinGW (x86 32bit), which is the previously installed MinGW 4.6.2.  The g++ path can be seen by selecting this entry, which should show C:\MinGW\bin\g++.exe.

For the final tool, CMake, select the CMake tab next to Tool Chains.  If the correct path was added to the Path under Environment Variables, the path to cmake.exe will already be set.  Otherwise, click Browse and find cmake.exe, which will be found under C:\Program Files\CMake 2.8\bin\cmake.exe (Windows XP) or C:\Program Files (x86)\CMake 2.8\bin\cmake.exe (Windows 7) if CMake was installed in the default location.  Don't select cmake-gui.exe.

Now it's time to see if QtCreator is able to build and debug the ibcp program before we start any Qt related modifications.

Installation For Qt Development

On Windows, when installing the Qt SDK, select Custom then on the Select Components dialog, unselect the following under Documentation: Harmattan, Qt Simulator, Symbian, and Qt Mobility; under APIs: Qt Mobility APIs and Qt Quick Components for Symbian; and under Development Tools: Harmattan, Simulator, Symbian Toolchains, under Desktop Qt: Qt 4.7.4 (Qt 4.8.1 will be used).  Under Miscellaneous, it is unnecessary to select MinGW 4.4 (the previously installed MSYS/MinGW 4.6.2 will be used, see here for instructions).  Also, the Qt Examples are not necessary but can be left in.

These selections will decrease the amount to download when using the on-line installer.  Once installed, any of the unselected components can be added using the Maintain Qt SDK program under the Qt SDK program group using the Package manager.  The Default installation can also be used  Once installed, QtCreator needs to be connected to our tools (git, MinGW and CMake; see next post).

On Linux, the download installers can be used with the same selections as above (make sure the correct installer versions are downloaded, 32-bit or 64-bit to match the version of Linux).  On Ubuntu 12.04 based distros (for example Linux Mint 13), it is not necessary to install the Qt SDK as the packages needed for Qt development are in the Ubuntu repositories.  Most of the Qt libraries should already be installed (especially if KDE is being used since it was developed using Qt).  The only additional packages that need to be installed are qtcreator, libqt4‑dev, qtcreator‑doc and qt4‑doc (use the sudo apt‑get install command or the package manager).  The version of QtCreator installed from the repositories is the same as in the current Qt SDK, specifically 2.4.1.

Preparing For Qt Development

Qt has its own IDE (Integrated Development Environment) called QtCreator.  Qt also requires all the various Qt libraries as well as Qt header files and several utilities.  Qt has its own make system called Qmake, but QtCreator is also compatible with CMake, so Qmake won't be used.

For Windows, the best way to start is to simply install the Qt SDK (Software Development Kit) that contains and installs all the necessary programs and files.  The current SDK, version 1.2.1, which contains Qt libraries version 4.8.1 and QtCreator version 2.4.1 as well as MinGW compiler suite.  However, the version of MinGW installed contains an older GCC, version 4.4.  Although it can easily be made to work with an existing MSYS/MinGW installation like the one the project has been using based on GCC 4.6.2.

Several choices of installers are available, a small on-line installer and a very large off-line installer.  The off-line installer has the advantage of quick reinstalls.  However, the on-line installer only downloads the components selected during installation so potentially there is much less to download (more can be downloaded and installed later if needed).  Many of the components are unnecessary when strictly used for desktop development (for example, all the mobile development files can be ignored).  Next post, installation...

New Focus – Introducing Qt

When this project was started, the plan was to first develop all the internal routines (parser, translator, recreator, etc) and wrap a simple command line interface around it initially for testing before tackling a modern interface (GUI).  I even went as far as starting to learn about using the Console mode on Windows (though this couldn't be used on Linux).  That was the purpose in parsing immediate commands.

Over the past year or so, I have been learning and using Qt, a cross-platform application framework (and it is Open Source).  With Qt, it is fairly easy to develop programs with a sophisticated GUI, plus it does a whole lot more.  So instead of fooling with some antiquated command line interface, the project will be transitioned over to using Qt.  This desire accelerated after discovering the Basic-256 program early this year.  This program is a great platform for teaching programming (was formally called Kid-Basic).  This program also uses Qt.

It is not totally clear all what is necessary for this transition to Qt, but several items come to mind.  The classes implemented towards the beginning of the project, the List and Stack classes, will be removed and the Qt equivalents will used in their place.  Though not necessary, variable and function names will be changed to the Qt way of naming.  The exceptions used during initialization will be removed as there are no exceptions in Qt.

Since this will be a major change in development, the Release 0.1 series will be concluded and development of the Release 0.2 series will begin.  The change in direction mentioned with Release 0.1.15 will continue after the transition to Qt.  First though, the computer needs to set up for Qt Development...

Project – CMake (Release)

It took a little extra time generating the release (it has been a while).  GitHub recommends that the project include both a README and LICENSE file, so these were also added to the project.  GitHub was updated yesterday and all the download files have been uploaded to Sourceforge today.

Since both Windows and Linux natively use different format text files (CRLF vs. newline for line separator), both sets of source files were uploaded.  The Windows download file ibcp_0.1.16‑src.zip is in zip format and the Linux download file ibcp_0.1.16‑src.tar.gz is in tar format compressed with gzip, use the tar xzf <filename> command to uncompress and untar the file with a single command.

The binary download files that were uploaded contain not only the executable program, but also the test input files, expected result files and regression test scripts.  Again the included text files are in the correct file format for the platform.  The Windows binary download also contains a DOS batch file for running the regression tests.  Nothing needs to be installed to run the program and regression tests.

CMake Updated For Test Programs

The old make file also built the various test programs residing in the test sub-directory.  The CMake build system was updated to also build these test programs.  This required quite a bit of experimentation to get working, but the final solution was rather simple.  First, each of the test programs an add_executable command was added to CmakeLists.txt with the list of the source files required to build the program along with any dependent header files.

To create the new make tests target required an add_custom_target command naming the target (tests) and specifying the test programs that this target is dependent on using the DEPENDS option.  However, there was a problem where the test programs were built with the standard make command (the same as make all) in addition to when the make tests command was issued.  This problem was resolved by adding the EXCLUDE_FROM_ALL option to each of the test programs add_executable command.

Each of the test programs were tested to verify that they produced the same output on all three platforms.  Several of the test programs were updated to achieve this, and the expected output files were updated accordingly.  Hit Continue... below for details on the changes made to the test programs.

Now that all the CMake issues have been resolved, it is time to make official release 0.1.16.  It's going to take a little time to prepare the release notes, merge branch0.1.16 to master, generate all the archive files for uploading, etc..  In the mean time, the latest changes have been pushed to GitHub and tagged v0.1.16‑pre‑2.  The should be close to the actual release, since none of the core program files are expected to change.

Updated Auto-Generated Test Header

When correcting the parser code, it was noticed that there were two constant arrays that contain text strings for the names of their associated enumerations, namely the token and data type enumerations.  These are used, along with the code name array (that is already auto-generated) in the parser test code to output the test results.

The issue is that the text strings must match the enumerations.  The code name array was already set up to be auto-generated (getting the text strings by scanning the table.cpp source file).  The token and data type enumerations are contained in the ibcp.h header file.

The awk script test_codes.awk was generating the contents (text string elements) of the code names array.  This script was renamed to test_names.awk and was modified to generate a complete array definition along with scanning ibcp.h header file for the token and data type enumerations to create name arrays for these also.  The generated file was changed from test_codes.h to test_names.h and the CMakeLists.txt was modified accordingly.

Several other minor changes were made along with this commit, for details see the commit log on GitHub.  The remaining issue is to add the test programs to CMake, which were in the older make file that has been removed.

Update (9:06): I should now better then to rush out a change without testing on both major platforms.  It turns out the updated test_names.awk script reported an error on Windows.  The problem was that the awk script was in DOS format and contained line continuations (back-slash at the end of the line).  The awk program saw the CR at the end of these continuation lines and got confused.  These were removed to eliminate the problem and GitHub was updated.

All Platform Parser Issues Resolved

On the Windows platform, the C format specifier "%g" outputs three digits in the exponent always whereas on Linux, only two digits output unless three are needed.  Looking at the C99 standard, it says basically "at least two exponent digits" should be output, so with that criteria, Windows is acceptable.  It also appears that Microsoft decided that three exponent digits should always be output, and the GNU compiler suite decided to follow this on Windows.  There is no way to control the number of exponent digits in the "%g" specifier.

The parser print token routine was modified to output two exponent digits if the first one is zero.  This was accomplished by first outputting the number into a temporary character buffer using the "%g" specifier.  Then it searches for an 'e' character in the string.  If the fourth character from the 'e' is not a zero-terminator, then that means there are three exponent digits.  If the second character from the 'e' is a '0' then the last two digits is moved on top of this zero.

Hit the Continue... for a code snippet of the solution.  This change will now output two exponent digits always if the first one is zero.  This code will do nothing on the Linux platform.  The expected results for parser test #3 were updated for two exponent digits were applicable (five instances).

Since the major CMake issues have been resolved and the 0.1.16 release will be about the build system, the cmake0.1.16 branch was merged to branch0.1.16 branch.  This last commit was also tagged as v0.1.16‑pre‑1.  On GitHub, a zip (or .tar.gz) files can be downloaded directly for a tag, which contain all the files at that tag, so it is not necessary to install git to retrieve the files.  This file does not contain the git repository however.  There are still a few more build issues to take care of.

Resolving Platform Parser Issues

The first parser issue is that on the Windows 7 (64-bit) platform, values as small as around 1e‑323 are accepted without a range error, whereas on the Linux (64‑bit) and Windows XP (32‑bit), values smaller than around 1e-308 produce a range error.  This is equivalent to the large value around 1e+308.

I thought possibly this had to do with Intel (what Windows 7 is running) on and AMD (what Linux and the Windows XP virtual machine are running on).  However, in testing the limits on a Scientific Linux 6.2 (equivalent to RedHat Enterprise 6.2) 64-bit virtual machine on the Intel, the limit was around 1e‑308 (though it was with the older GCC 4.4 vs. 4.6 on the others).

The float and long double conversions were also tested on all three platforms.  The float had a limit of around 1e‑38 on all three platforms as well as the long double having a limit of around 1e‑4931.  So I have no explanation why double conversions are different on Windows 7.  Both 7 and XP have identical versions of MinGW installed.

So instead of fighting this problem any longer, the 1.234e‑308 test value in parser test #3 was changed to 1.234e‑324, a value that will produce a range error on all three platforms.  Next to deal with the differences in the number of exponent digits output.

Updated Regression Test Scripts

The regression test scripts were modified for the changes in the test code (and new command line).  The test numbering was removed and the scipts now determine which tests to run by the presence of the parser*.dat, expression*.dat and translator*.dat files in the test directory.  The regtest script (for Windows/MSYS and Linux) no longer references ibcp.exe but simply ibcp, which will work on both Windows and Linux, with the ".exe" part added automatically on Windows.

To be able to handle any build directory (in source or out of source), the regtest script uses the source directory directly.  This was accomplished by making CMake configure this, so regtest was renamed to regtest.in and references the CMake ibcp_SOURCE_DIR variable to create regtest in the binary (build) directory that uses the correct source directory.  Now a build and test consists of just three steps, running cmake (with the proper options and path to the build directory), running make and running regtest (in the build directory).

I considered dropping the Windows Command window regtest.bat batch file, but decided this should still be included so that the regression tests could be run from the downloaded binary along with the source directory without having to build the program.  It is still a crude alternative to the regtest bash script, but it gets the job done.  When run, the "Compare more files (Y/N)?" question still has to be answered with an N now three times with the addition of the expression test compares.

The tests with the regression tests script work on all three platforms, except one problem remains: the problem with parser test #3 giving different results on the three platforms (the Windows XP results are in the repository), which still needs to be resolved.  All the testing changes has be pushed to the repository on github, including a new .gitignore file, which tells git to ignore any files in the build and nbproject sub-directories.

New Test Code

The new test code design outlined on October 10 has been implemented and tested.  The old test_parser() and test_translator() functions with their test inputs, were replaced with a single test_ibcp() function that reads an input file and calls the appropriate function based on the type.  The ibcp program command lines options have changed as shown:

ibcp -t <testfile>
ibcp -tp|-te|-tt

The first command form specifies the test file, which contains the tests inputs.  The test code determines the type of file by the beginning of its name (parser, expression, or translator).  The file name may contain anything after the type, including any extension.  Blank lines and comment lines (first character '#') are ignored.  The second command form activates the interactive mode when the program asks for each input for one of the three types.  The new expression type needs further explanation. 

Early in the translator development, the translator only processed expressions.  The first four tests only have expressions.  This implementation was left in the code and was changed to be activated only if an expression only flag is passed to the translator.  The old test code set this flag for the first four tests.  To keep things simple, since the file name is being used to detect the type, this was extended by the new expression type.

The expected result files were renamed accordingly.  Also, the translator tests were given two digit test numbers, with a leading 0 for the first nine tests, so that these would be processed in order by the regression test scripts.  The new test input files were given a ".dat" extension.  The code was also modified to not output the ".exe" with the program name in the header of the output so that Windows and Linux output matches.  Now the regression test scripts need to be updated.

CMake with NetBeans

There were problems getting the new test code to work properly.  Using gdb from the command line was not the first choice (more time is spent looking up help on commands than actually debugging).  So this left NetBeans, which I was hoping to delay if not permanently postpone the use of (for the reason the change in development alluded to earlier, more on this soon).  NetBeans is supposed to work with CMake and it was time to find out. 

In the spirit of staying current, the latest version (7.2) was downloaded from NetBeans.org.  The C/C++ bundle was selected for both Windows and Linux (x86/x64).  The defaults for the installer were used on all three platforms (XP, 7 and Mint 13) and available updates were installed.

On Windows XP and 7, getting NetBeans to work with CMake was problematic, in fact, I never did figure out how to get it to work correctly.  NetBeans appeared to give CMake incorrect options and CMake said that the compiler was not able to be used.  All attempts to get the options that worked were unsuccessful.

The solution found searching the Internet was to run CMake outside of NetBeans, then point NetBeans to the Makefile produced and this worked.  CMake would have to be run outside of NetBeans from then on when needed - not ideal.  After a bit of work, debugging was enabled and the program could be stepped through.

On Mint 13, NetBeans worked with CMake the way it was supposed to.  NetBeans was able to run CMake and build the program.  The program was built with debugging enabled by default.  Debugging the problems in the new test code went without a hitch.  Hit Continue... for details of the procedures on both platforms.

Testing Implementation Change

In considering how to modify the CMake build system to handle regression testing, I realized that there was a better design in how testing should be handled.  Currently all the test inputs are contained in a source file with the testing code.  The disadvantage is that every time a new test input or test is added, this source file needs to be modified.  Adding a new test requires additional work including several changes to the test source file and modifying both regression test scripts.  And finally, strange looking C string quoting is required for double quotes and back-slashes in the test strings.

A better design would be for the tests and their inputs to be in separate test files that would be read and processed by the test code.  To change or add test inputs, these test files would be modified instead of the source file.  To add a new test, a new test file would be added.  The regression test scripts would simply run the program for each test file present.  Another advantage is that the test code does not need to be recompiled when tests are modified or added.

The test files will also support simple comments.  Lines that start with the pound '#' character and blanks lines will be ignored by the test code.  This method of commenting was chosen since no BASIC lines will start with this character and editors that support syntax highlighting (for example, vim, Kate and probably notepad++) show these comments in a different color and/or font when the file is treated as a configuration file (which vim does automatically, and Kate remembers after the option is set once).

Additional CMake Corrections

Back at the end of September when Mint 12 (Ubuntu 11.10) with GCC 4.4 (needed for CUDA development) was still being used, a change to the CMake file to allow (with a warning) GCC versions less than 4.5, which do not support linking static libraries.  This option was used to create an executable that did not require dynamic libraries (so the binaries posted on SourceForge would run without having to install any other software). 

This change was considered temporary and the warnings produced with older GCC versions could be ignored and any executables produced couldn't be posted.  To eliminate this compiler warning, an if statement was added around the statement that added the static library link options in the CMake file, so that this option is used for GCC versions 4.5 and later.

The problem with in source directory builds was due to the local string.h header file name conflicting with the standard header file of the same name, which was included in the local string.h header file - the local header file was being included instead of the standard header file.

There is a statement in the CMake file that adds the project's binary directory to the include directories searched so that the auto-generated include files can be found.  So, with an in source directory build, the local string.h file was being found even with angle brackets on the include statement since the project directory was now being searched and is apparently being search before the standard directories.

With an in source directory build, if the project directory is not added to the include directories, the correct header file is found.  Since it is not necessary to add the project directory if building in the source directory (the auto-generated header files will be in the same directory as the source files including them), so to correct this problem, a check was added to the CMake file that if the project binary (build) directory is the same as the project source directory, then it is not added to the include directories.

The old make file was also removed since it now being generated by CMake and there are problems when doing in source directory builds with it present since CMake modifies the make file that is under source control (git then indicates that is was modified).  These two commits are now up on github.

Building The Latest With Command Line CMake

To use CMake from the command line on Windows, the directory containing cmake.exe must be added to the path in MSYS just like for the git programs.  The directory /c/Program Files/Cmake 2.8/bin (Windows XP) or /c/Program Files (x86)/Cmake 2.8/bin (Windows 7) needs to be added.  One thing I forgot to mention in the October 5, 2012 post was that in order for the "/c" part to work in MSYS, the /etc/fstab (C:\MinGW\msys\1.0\etc\fstab) file system table file needs to be modified by adding a new line containing "C:\  /c" to give access to the entire C: drive.

To start, first create the build directory.  For this procedure, create ibcp-build in the same directory as the ibcp source directory and change to it.  On Windows, just like in the CMake GUI, the generator needs to be selected from the command line.  This is done with the ‑G option.  So, use this command to generate the make file:

cmake -G "MSYS Makefiles" ../ibcp 

On Linux, the default generator will be used, so use the cmake ../ibcp command.  To build, use the make command.  By the way, to see the complete compile lines that make is using, use the make VERBOSE=1 command to build (which can be helpful in debugging compile problems like header files that can't be found).

Now some additional build issues need to be resolved, including correcting the issues with parser test #3 on the different platforms, finding the problem with in-source directory builds, and correcting regression testing so that it can be run from the build directory without have to copy the executable to the source directory.

So far, Release 0.1.16 development has been mostly about the build system, but a couple of other minor changes were made, namely, parentheses are no longer stored with identifiers and support for negative constants was added to the parser (instead of treating the minus character as a unary operator).  Therefore, before continuing with development (and a change is coming, stay tuned), a release will be made with the new CMake build system once the remaining issues are resolved.

Building The Latest With CMake-GUI

Instructions for building with CMake were previously given on June 23, 2011 for Windows and June 21, 2011 for Linux, but will be briefly given here again using the CMake GUI for both.

On Windows, the CMake program needs to be installed.  To keep up to date, the current version (2.8.9) was downloaded from Kitware and installed.  There is only a 32-bit version, but this works fine on Windows 7 64-bit.  The defaults of the installer were used.  This version is close to the packages (2.8.7) mentioned on October 3, 2012 installed on Mint 13 (Ubuntu 12.04).

Start the CMake GUI, on Windows: Start, All Programs, CMake 2.8, CMake (cmake-gui), and on Linux: CMake from the launcher menu under Applications/Development or the cmake-gui command from a terminal in any directory).  Once started, enter or browse to the ibcp directory where the source code is (on Windows, this will be C:/MinGW/msys/1.0/home/username/ibcp, and on Linux where it was cloned to, probably /home/username/ibcp).  Then select the directory to build the binaries in (recommend using C:/MinGW/msys/1.0/home/username/ibcp-build or /home/username/ibcp-build).  Click the Configure button, which brings up a dialog for which generator to use.

On Windows, the generator needs to be changed to MSYS Makefiles, and on Linux, the default Unix Makefiles is used.  Click the Finish button to configure.  The CmakeLists.txt file will be processed and the output will be written to the lower pane.  Click the Generate button to create the make file.  That's it, the program can now be built.  From the command line, change to the ibcp-build directory and use the make command.

To test, copy the executable (ibcp.exe on Windows or ibcp on Linux) to the ibcp directory (on Linux, copy it as ibcp.exe).  From the ibcp directory, the regtest script is used to run the tests.  Currently, all tests succeed on Windows XP.  On Windows 7, one test in parser test #3 fails due to how small a floating point number can be (see here).  On Linux, several (5) tests in parser test #3 don't compare because exponents are output with only 2 digits instead of 3 on Windows.  Next up, using CMake from the command line...