Thursday, October 25, 2012

Third Translator Memory Issue

The memory issue reported for translator test 7 was the same message as test 6 except on a different line, which was another if statement.  Again through the process of elimination, the statement causing the error was identified to be MID$(A$ B, which is expected to produce an expected comma error at the B token.

After briefly studying this if statement, the problem was identified and is similar to the previous problem.  This if statement, in the process binary operator routine, was checking if the token (within a sub-string assignment) was not a comma.  Again, it should not have been checking the token code before checking that the code was valid (since not all token types have a code).  The B token is an identifier with no parentheses token type and code is not used.

The if statement was corrected by adding a check if the token has a table entry (and therefore a valid code) and it is not a comma.  Now all the memory issues were resolved.  I also now understand what the Conditional jump or move depends on unintialised values(s) error is indicating.  Apparently, the Analyzer (valgrind) is checking for more than just memory leaks, it is also checking when a variable is being accessed, but it hasn't been initialized, which was the case for these two if statements.

Rechecking all the tests, all the memory issues were resolved.  However, upon running the regression tests, translator test 7 was now failing.  The problem occurred with the statement above, which was now reporting an expected operator or comma error, which was wrong because with a sub-string assignments, an operator is not allowed after the string variable identifier.

The if statement was modified so that either the token does not have a table entry or the token is not a comma.  All tests now pass.  To simplify (and automate) this memory testing, a new memtest script was created, but only for Linux as it requires the valgrind program.  This new script is based on the regtest script and also checks the regression test results along with checking for memory issues.  Now back to replacing the List class with the QList class...

[commit 1d427d7d98] [commit da0014b34d]

Second Translator Memory Issue

While the first memory issue was a simple memory leak, the second issue was much more difficult to resolve.  The issue on translator test 6 was reported as a Conditional jump or move depends on unintialised values(s).  Clicking on this showed the source line and another message reporting Uninitialised value was created by a heap allocation.  The line for this message was in the token new function where the memory for the token is allocated.  Curious that the token allocation checks were not reporting any token leaks.

Through the process of elimination, the statement (out of 42) causing the error was identified to be PRINT A(TAB(10)).  Looking at the line indicated by the first message (an if statement) did not make it clear what the issue was.  So the code was stepped through with the debugger to identify the problem, which was caused by an incorrect check for the item on top of the hold stack, which happen to be the line reported with a problem.

This if statement checks if a print-only function is found in an expression, which should be reported as an error since these functions are only valid in a PRINT command.  The check is for either the current command is not a PRINT command or the token on top of the hold stack is not the null token (any other token indicates the print-only function is in a parentheses, array or function - an error).

The problem was with the null token check as it was only checking if the code of the token was not the null code.  However, not every token type has a valid code, specifically, constants, identifiers (with and without parentheses), and user defined functions (preceded by FN with and without parentheses).  This check was replaced with a call to a new token function that checks to see if the token is a null token, which first checks if the token has a table entry (and therefore a code) and then checks the code.

This corrected the problem with translator test 6.  All the expression and translator tests were rechecked with the Analyzer.  Translator test 7 was still reporting a memory issue.

First Translator Memory Leak

The first memory leak, which occurred on every expression and translator test, was easy to identify and correct as the Analyzer pointed directly to the problem.  The memory issue was reported in the translator start() function with the RPN (reverse polish notation) list allocated there.

For expression test 1, 13 blocks were reported lost, which exactly coincided with the number of test expressions.  This made it obvious that the memory allocated for the RPN list object was not being released.  In the translate input routine, after the resulting tokens in the RPN list were output, the memory allocated for each token was released.  However, the RPN list object itself was not released.

After the correction, all the expression and translator tests were rechecked with the Analyzer.  Translator tests 6 and 7 were still reporting memory issues.

Wednesday, October 24, 2012

Automatic Building For Debugging

When using Qmake with QtCreator, both Release and Debug builds are configured and it is easy to switch between the two.  CMake supports several build types including Release, Debug, Release With Debug Info and Minimum Size Release.  The default is blank (which is probably similar to Release, but definitely does not turn on debug information).  However, the CMake build types are not directly available in QtCreator except by specifying the CMAKE_BUILD_TYPE variable when running CMake.

During development it is obviously desirable to build the program with debug information.  The CMAKE_BUILD_TYPE variable could simply be set to Debug in the CMakeLists.txt file.  But then this would need to be changed when a release is made, so this is not a good solution.

Instead, the CMakeLists.txt file was setup to look for a CMAKE_BUILD_TYPE environment variable (CMake can access system environment variables).  If this environment variable is set, then the CMake variable is set to the environment variable unless it has already been set.  If the environment variable is not set, it behaves the way is did before.

This environment variable can be set in QtCreator by going to the Projects screen, selecting the Build Settings and adding this environment variable to the Build Environment with the desired Debug value.  The Release With Debug Info (RelWithDebInfo) could also be used, but can make stepping through the program confusing since most of same optimizations are turned on as with Release and the compiler can rearrange statements for efficiency (execution will appear to jump around) or optimize out variables (the values of which cannot be viewed).

[commit 7ee8e7483c]

Finding Memory Leaks (Linux)

QtCreator can be used to find memory leaks using the Analyze mode.  This requires the valgrind program, which can be installed via the valgrind package on Ubuntu based distros.  Unfortunately it looks like this program has not been ported to Windows (MinGW), so Windows developers are out of luck (at least when using MinGW with the Qt SDK).

In order to use valgrind, the program to check must be compiled with debugging information.  To do this with CMake under QtCreator, in the Run CMake dialog, the Arguments line needs to be set to ‑DCMAKE_BUILD_TYPE=Debug.  This is a nuisance because it needs to be done every time CMake is run the first time and there appears to be no way to automate this inside QtCreator.  So an alternate scheme was devised using CMake (will be described in the next post).

Once Analyze mode is selected (by the icon on the side panel or Ctrl+6), the Analyzer panel will appear.  In this panel, the mode needs to be changed from QML Profiler to Valgrind Memory Analyzer.  The program is started using the start (play) icon (left side of Analyzer panel toolbar).  Once the program ends, any memory issues will be reported.

After using the Analyzer to learn about list element deallocation, I thought it might be a good idea to check the ibcp program for any memory issues.  There is already an implementation to detect token memory leaks accomplished by overloading the new and delete operators for the Token structure, but there are many other memory allocation operations in the program.

So this process was started for each parser, expression and translator test.  For each, the run arguments were set (Projects page, Run Settings) and the Analyzer was run.  This were no memory issues on the parser tests, however, some problems were found on the translator tests, which will be discussed in following posts.

List Class Replacement (Begin)

The List class was the first implemented for this project, so it is fitting that it will be the first to be replaced with the transition to Qt.  Qt's list class is named QList.  The functionality is not much different then the home grown List class, though most of the member functions have different names.

The approach being used was to first replace all cases of List with QList and then search for each List class member function name and replace it with QList's version.  In some cases, the code needs to be written a bit since the QList functionality is slightly different.  For example, while QList also has a first() function, it must not be called if the list is empty (must check if it is empty first), while the List class first() function allows for an empty list (returning a null pointer).

The first complication came in how to deallocate items in the QList.  The question was, does this happen automatically when leaving scope, is deleted or via QList's clear() member function.  Neither was the case where the list is a list of pointers to allocated elements.  Each element needs to be deleted (same as the case with the List class).

A small test program was written to evaluate and confirm this behavior.  As part of this evaluation,  a method to detect memory leaks was employed that is kind of built into QtCreator (at least on Linux).  This lead to quite a few detours, which will be the subject of the posts that follow...

Tuesday, October 23, 2012

Unique Release Number Implementation

To see if the git command is available, a find_program command was added to the CMakeLists.txt file, which sets the CMake variable PROGRAM_GIT.  An if command was also added that checks if this variable is set, which then executes the command using the  execute_process command for the describe sub-command in the project source directory setting the CMake variable ibcp_RELEASE_STRING with the resulting string.  The option OUTPUT_STRIP_TRAILING_WHITESPACE was needed to remove the trailing newline.

If the git program was not found, then the ibcp_RELEASE_STRING variable is set to "v" followed by the major and minor release numbers separated by a period.  If the patch release number is less than zero (development tag), then the patch number is appended to the release string (the dash is present because the number is negative).  Otherwise, the patch number is appended with another period separator.  The final release string, from either source, is output to the CMake output log.

In implementing this if command, it was realized that it was not necessary to test the PROGRAM_GIT variable for the string to be equal to "PROGRAM_GIT‑NOTFOUND" as was done with the awk program because this is one of the tests performed by the CMake if command.  Therefore, the if command for the awk program was simplified.

A define for ibcp_RELEASE_STRING was added to the ibcp_config.h.in file (used to auto-generate ibcp_config.h) with the contents of the CMake variable surrounded by quotes, to form a string constant that can used in the source code.  The ibcp_version() function was modified to output this string constant instead of the major, minor and patch numbers.  The "v" part of the string is not output (because "version" is already being output).  Finally, the GPL header was removed when outputting the version number to be consistent with the output of other programs.

New branch0.2 was created.  The release string produced at this latest commit will be v0.1.16b-1-g1bba2c3, which shows the most recent tag (v0.1.16b), the number of commits beyond this tag (1) and the short commit ID (1bba2c3).  The "g" in front of this ID stands for git (other letters would stand for other software configuration management systems).

[commit 1bba2c335c]

Unique Release Numbers

With a new release numbering scheme defined, there needs to be way to get the current version number into the program for output with the -v command line option (and eventually in the Help/About box once the GUI is implemented).  The version number should also be unique during development since not every commit will be tagged.  The goals are:
  1. Use the current tag if at a tagged commit
  2. Represent when not at a tagged commit during development
  3. Use the current release number assigned for archive downloads
  4. Allow for developmental (dash) and patch (period) numbering
The inclusion of the third goal will be clear shortly.  It turns out that the exact desired release string can be obtained using the git describe command, which returns the name of the most recent tag.  When beyond the most recent tag, the number of commits beyond the tag plus the short form of the current commit ID is appended.

This works as desired when the git command is available and the git repository information is present, but for the third goal, the downloaded archives have no git information.  Since the download archives are only available at tagged commits, the release number set in the CMakeLists.txt file can be used and will match the tag at that commit (assuming these variables were set to the same values as the tag).

For the last goal, there are two cases, git repository information present and not present.  When making tags during development, the tag name format will be releaseX.Y‑Z (note the dash).  The git describe command will pick this tag name (and append the rest if beyond that tagged commit).  When the git repository information is not present, the major, minor and patch release numbers set in the  CMakeLists.txt file will be used.  To handle developmental (dash) numbering, negative patch numbers will be used.

New Release Numbering Scheme

In light of the two recent problems discovered after Release 0.1.16 was made, a new release and branch numbering scheme is needed.  Technically, the third number on the release number should be for patches of a release.  Instead of using 0.1.16a and 0.1.16b, 0.1.16.1 and 0.1.16.2 could have been, but wasn't because the version numbering is currently only setup for major, minor and patch numbering, so the "a" and "b" were used (though it wasn't setup for this either).

As for branches, there is no reason to have a branch for every patch number (like with branch0.1.14, branch0.1.15 and branch0.1.16), though these were not really patches though the patch number was increasing.  Therefore, going forward there will only be branch0.2 for the developmental release 0.2 series.  As something worthwhile is completed, a tag with this number plus a dash number will be added, for example 0.2-1, 0.2-2, etc.  This will be the equivalent of the "-pre-" (and "-dev-" before that) that were used previously.

When the release series is completed, the release will be given the number 0.2.0.  (It's too early in development to worry about release candidates, but when that time comes, the "-rcX" format will be used.)  Any patches needed for a given release will then be given patch numbers 0.2.1, 0.2.2, etc.  Now to generate a unique version release number at each commit taking into account using the current tag for a tagged release...

Monday, October 22, 2012

Regression Test Script Problem

Another problem was discovered, this time with the regtest script.  For convenience, my PATH variable contains the current working directory ("."), so I didn't notice this problem.  By default (and for safety), the PATH variable does not contain the current working directory.  So the regtest script fails to find the "ibcp" program.

Therefore, regtest (actually regtest.in) was modified with a "./" in front of ibcp so that the regtest script will run the program in the current working directory.  No new files were uploaded to Sourceforge, but a new commit and tag release0.1.16b was pushed to GitHub.  The Windows batch file regtest.bat is not affected since Windows by default will look in the current working directory for a program.

Sunday, October 21, 2012

Build Issues Discovered

While writing the procedures for building with QtCreator, some issues were discovered.  The first was with running CMake on Windows.  The first time it produces errors no matter which generator is selected.  It also appeared that adding "‑G "MSYS Makefiles" caused it to work, but adding this before still caused errors.  It turns out this was not necessary, simply clicking the Run CMake button a second time worked with no error.  The previous post about this was updated.

The next problem was when trying to use the included MinGW (with GCC 4.4) in the Qt SDK.  It was still trying to use the static linking, which is not supported in versions before GCC 4.5.  It turns out there were two problems.  First, the check to whether to add the static linking options should have been greater than 4.4, but was incorrectly less than 4.5.  Second, the add_definitions command used in the CMake file adds compiler options, not linker options.  The command add_target_properties should have been used with the LINK_FLAGS property option.

Due to these linking issues, the binaries posted on Sourceforge were not linked with static linking.  This is not an issue with Linux since the required libraries will be present.  However, for Windows, a missing DLL message will occur on a system without the required libraries.  Therefore, a new binary zip file was updated (labels 0.1.16a).  The update executable was actually tested on a Windows XP system without the require libraries this time. The sources and Linux binary were not updated.  If building from source, the required libraries will be present.  The repository on GitHub was updated with new tag release0.1.16a.

The final problem is in using the MinGW installed with the Qt SDK.  Even though the static linking problems were corrected, the included MinGW cannot (alone) be used the build the project.  The issue is that the awk utility is needed to create the auto-generated header files and there is not included awk utility with MinGW.  It is included with the MSYS package.

Saturday, October 20, 2012

Running in QtCreator

After the program has been built it can be run or debugged in QtCreator.  For now, command line arguments need to be added before running the program, otherwise it will only output a usage message.

Command line arguments are set on the Projects screen (the Projects icon on the side or Ctrl+5) by selecting the Run Settings button along the top of the screen.  For example, under Run, enter ../thunder422‑ibcp/test/translator01.dat to run the first translator test (assuming the default directories were used up to now).

Again there are multiple ways to run the program (play icon on lower side, Ctrl+R, Run on the Build menu).  Same for running in the debugger; though note that in order to trace through the program, a breakpoint needs to be set first or the program runs (in the debugger) until it exits.

Building With QtCreator

Now that it has been configured for all the tools, everything can be done inside QtCreator.  Start QtCreator and select New Project... on the File menu.  In the New dialog select Project from Version Control and then Git Repository Clone.  After clicking the Choose... button, the git repository can be selected.  For Clone URL: enter https://github.com/thunder422/ibcp.  If desired, change the Checkout path.  The Checkout directory will default to thunder422-ibcp.  Click the Next button and the repository will be cloned.

Once finished, the CMake Wizard dialog appears asking for the Build Location.  The default directory can be used.  The next dialog of the wizard is to Run CMake.  On Windows, the Generator: needs to be changed to MinGW Generator (MinGW (x86 32bit)) assuming MinGW 4.4 was installed with Qt SDK, otherwise this is the only generator available.

Clicking the Run CMake button at this point causes a bunch of errors.  Adding ‑G "MSYS Makefiles" to the Arguments: line appears to resolve the problem.  Adding this before clicking the Run CMake button the first time does not make it work though clicking a second time does.  Clicking the Run CMake button a second time works.  I have no explanation for this at the moment.  This is a similar problem that caused issues with NetBeans and CMake.  On Linux, clicking the Run CMake button is all that is necessary.

Adding ‑DCMAKE_BUILD_TYPE=Debug to the Arguments: line turns on debug information to that the debugger can be used. After CMake is run, the program can be built.  There are a number of ways to begin the build, including the Build menu, Ctrl+B and the hammer looking icon at the bottom of the tool bar on the left side.  The build can be monitored by clicking the 4 Compile Output button along the bottom of the screen.

Post Qt Installation Setup (Windows)

After installation of the Qt SDK, the tools (git, MinGW and CMake) need to be integrated with QtCreator.  On Linux, no further configuration is needed as all tools will be located where QtCreator can find them.  However, on Windows, this is not the case.

For git, the binary directory for the git tools need to be in the execution path.  On Windows XP, right-click on My Computer and select Properties.  On the Advanced tab, click the Environment Variables button.  On the lower pane under System Variables, find and selected the Path variable.  Click Edit and add C:\Program Files\Git\bin somewhere on the Variable value line (make sure to add the semicolon separator).  On Windows 7 the instructions are similar.  Start by right-clicking Computer and select Properties.  Now select Advanced system settings, go to the Advanced tab and follow the same instructions except add C:\Program Files (x86)\Git\bin to Path.

Now start QtCreatorQtCreator should have automatically found the external MSYS/MinGW 4.6.2 previously installed.  On the Tools menu select Options, go to the Build & Run page, and select the Tool Chains tab.  Under Auto-detected there will be an entry Mingw as a GCC for Windows targets (if the MinGW 4.4 was installed as part of Qt SDK) and an entry MinGW (x86 32bit), which is the previously installed MinGW 4.6.2.  The g++ path can be seen by selecting this entry, which should show C:\MinGW\bin\g++.exe.

For the final tool, CMake, select the CMake tab next to Tool Chains.  If the correct path was added to the Path under Environment Variables, the path to cmake.exe will already be set.  Otherwise, click Browse and find cmake.exe, which will be found under C:\Program Files\CMake 2.8\bin\cmake.exe (Windows XP) or
C:\Program Files (x86)\CMake 2.8\bin\cmake.exe (Windows 7) if CMake was installed in the default location.  Don't select cmake-gui.exe.

Now it's time to see if QtCreator is able to build and debug the ibcp program before we start any Qt related modifications.

Installation For Qt Development

On Windows, when installing the Qt SDK, select Custom then on the Select Components dialog, unselect the following under Documentation: Harmattan, Qt Simulator, Symbian, and Qt Mobility; under APIs: Qt Mobility APIs and Qt Quick Components for Symbian; and under Development Tools: Harmattan, Simulator, Symbian Toolchains, under Desktop Qt: Qt 4.7.4 (Qt 4.8.1 will be used).  Under Miscellaneous, it is unnecessary to select MinGW 4.4 (the previously installed MSYS/MinGW 4.6.2 will be used, see here for instructions).  Also, the Qt Examples are not necessary but can be left in.

These selections will decrease the amount to download when using the on-line installer.  Once installed, any of the unselected components can be added using the Maintain Qt SDK program under the Qt SDK program group using the Package manager.  The Default installation can also be used  Once installed, QtCreator needs to be connected to our tools (git, MinGW and CMake; see next post).

On Linux, the download installers can be used with the same selections as above (make sure the correct installer versions are downloaded, 32-bit or 64-bit to match the version of Linux).  On Ubuntu 12.04 based distros (for example Linux Mint 13), it is not necessary to install the Qt SDK as the packages needed for Qt development are in the Ubuntu repositories.  Most of the Qt libraries should already be installed (especially if KDE is being used since it was developed using Qt).  The only additional packages that need to be installed are qtcreator, libqt4‑dev, qtcreator‑doc and qt4‑doc (use the sudo apt‑get install command or the package manager).  The version of QtCreator installed from the repositories is the same as in the current Qt SDK, specifically 2.4.1.

Preparing For Qt Development

Qt has its own IDE (Integrated Development Environment) called QtCreator.  Qt also requires all the various Qt libraries as well as Qt header files and several utilities.  Qt has its own make system called Qmake, but QtCreator is also compatible with CMake, so Qmake won't be used.

For Windows, the best way to start is to simply install the Qt SDK (Software Development Kit) that contains and installs all the necessary programs and files.  The current SDK, version 1.2.1, which contains Qt libraries version 4.8.1 and QtCreator version 2.4.1 as well as MinGW compiler suite.  However, the version of MinGW installed contains an older GCC, version 4.4.  Although it can easily be made to work with an existing MSYS/MinGW installation like the one the project has been using based on GCC 4.6.2.

Several choices of installers are available, a small on-line installer and a very large off-line installer.  The off-line installer has the advantage of quick reinstalls.  However, the on-line installer only downloads the components selected during installation so potentially there is much less to download (more can be downloaded and installed later if needed).  Many of the components are unnecessary when strictly used for desktop development (for example, all the mobile development files can be ignored).  Next post, installation...