Accessing and Building on Linux

The git clone command is used to obtain a git repository.  To obtain the IBCP git repository from github.com use the command:

git clone https://github.com/thunder422/ibcp.git

This will create the ibcp directory and populate ot with the source files for the master branch (which is currently Release 0.1.15).  However, the entire repository is present (there is a hidden .git directory).  The commits can be viewed using the git log command (there are many options, which is why installing the git documentation is strongly recommended).  The repository can also be viewed in GUI form using the gitk command if installed.

At this point, the program can be built using the make command in the ibcp directory.  There are a couple of compiler warnings, which were corrected in the 0.1.6 development branch and can be ignored.  The parser and translator tests can be run with the ./regtest command.  Currently on Mint 13 (64-bit) on Release 0.1.5 (master), parser tests 1 and 3 fail.

Parser test 1 fails due to a bug with parsing integers, which did not occur on 32-bit Windows.  This problem was corrected in the 0.1.16 development branch.  Parser test 3 also fails because of an integer parsing error and due to a change in output of exponentiation numbers (64-bit Linux outputs only 2 digit exponents, but Windows outputs 3, which may also be due to the particular compiler or library versions).  The exponent issue has yet to be corrected.

Next up, how to build the latest development branch, which is also contained in the git repository and uses cmake to create a make file...

Linux Prerequisites For IBCP & Git

Several software packages are required to obtain and build the IBCP program from the git repository, which include:

g++    Needed to compile C++ programs (brings libstdc++6 with it)
git    Needed to retrieve the IBCP Git repository
cmake  Needed for the CMake utility

These can either be installed from the GUI package manager (for example the Synaptic Package Manager or Muon Package Manager) or from the command line using the apt-get command, as in sudo apt-get install g++ cmake git.  There are several other packages that are nice to have, but are not required, which include:

gitk              a GUI for displaying a git repository
git-gui           a GUI for working with the git repository
git-man           the manual pages for the git commands
git-doc           alternate documentation for git
cmake-curses-gui  a terminal GUI for cmake (ccmake)
cmake-qt-gui      a nice Qt GUI for cmake (cmake-gui)

I should note that I actually obtained git by cloning the official git source repository and building the latest version (1.7.12.1) myself though git needs to be installed before this can be done.  Quite a few more packages are required to build all of the git package and documentation.  However, the version in the repositories (1.7.9.5 in Mint 13/Ubuntu 12.04) is sufficient for working with the IBCP git repository.  Next up, how to obtain and build the IBCP program.

Using Linux Mint

Linux Mint 10 KDE Julia proved to be much more stable that Kubuntu 10.10, however, after about week or so of running, it would either freeze or get so sluggish that it was unusable.  This appeared to occur when memory usage got up over 85% or more.  I suspected the problem was due to problems in Linux when using swap space.  After adding another 4 GB to the component to bring the total to 8 GB, Mint 10 was rock solid, at times running over a month with no issues.

Mint 10 KDE was a good platform and the CUDA SDK (version 4.0) worked on this system.  CUDA was not needed for this project, but is needed for a work project, the reason for going to Linux in the first place (which was also required).  Mint 12 KDE Lisa (based on Ubuntu 11.10 Oneiric Ocelot) was released (there was no Linux Mint 11 KDE Katja).  I installed Mint 12, but there were a number of issues.  Since I made the system dual boot (or should say multiple boot because the original Windows XP system and even the Kubuntu 10.10 install was still available), I just went back to Mint 10 as it was stable and met my needs.

However, sometime early last spring when the then next version of Firefox and Thunderbird was released, I thought it strange after many weeks the new versions were not showing up in the repositories.  When I investigated, I learned that Mint 10 was at end-of-life (meaning no more updates).  So it was time to move to Mint 12.  Some time was spent getting Mint 12 working they way Mint 10 was.

Public Git Repository

Initially CVS (concurrent versions system) was used for this project and the repository was converted to git using the git cvsimport command.  Before making the git repository public, I wanted to correct some issues that occurred during the conversion, specifically branches were not converted correctly.  Merges to the trunk (CVS nomenclature) did not show correctly as a merge to master (git nomenclature) - in fact there was no actual merge, just a hanging branch which a merge commit from where the branch first branched off the master branch.

These issues plus several other issues were corrected and the public git can now be accessed at github, where the repository can be explored, including the commits, source files, and branches.  Hit the Continued... for more details of the repairs.  There is also a Wiki on github, but for now it only points back to here.  The current branch is cmake0.1.16 (which is off of the branch0.1.16 development branch, but the CMAKE implementation is not quite finished yet).  Next up, some details on my current Linux development platform (there have been some changes).

Long Time No Posts

It has been a long time with no posts.  I have been busy with other things, but it is time to get this project started back up with a change in focus (more on this shortly).  The primary purpose of this project has been a platform for learning.  Initially this was to get back into C++, something that I hadn't been using for nearly a decade (I had been using C, but there is quite a different between the two).  Along the way git (software revision control) and cmake (cross platform make) was introduced along with the Netbeans development environment.

Speaking of git, the first thing I wanted to accomplish in this restart was to make the git repository for this project public.  Sourceforge allows a public SVN (subversion) repository, but I am not familiar with SVN and wasn't really interested in learning SVN since I otherwise wouldn't be using it (I use git extensively for work and have got quite fond of it - for more on git, read the excellent book Pro Git by Scott Chacon).  But first, I wanted to clean up the existing git repository before making it public, which I spend the last several weeks working on...

Switch in Linux Distribution

The 64-bit Kubuntu 10.10 installation has been problematic. Several comments around the Internet have indicated this to be the case. The reason Kubuntu was chosen was that it contained the KDE desktop, and 10.10 was chosen because of it is the newest on the supported host operating system list for VMWare Workstation (being used for the Windows XP virtual machine). Standard Ubuntu comes with the GNome interface, which I am not a fan of.

The problems with Kubuntu mainly include general sluggishness, a problem that was never resolved after a little over a month of use (something not expected on a Phenom II quad code running at 3.2 GHz with 4GB of memory with a NDIVIA GTS-450). It would completely hose up at times (not lock up or crash) when doing even simple tasks like uncompressing files. Even after all the updates were applied (less the update to 11.04 Natty Narwhal), including KDE and all the fancy desktop effects were turned off, the problems persisted. Mounting a bunch of NTFS partitions (from the previous Windows XP installation) was also suspected to be a cause.

Therefore, the Linux Mint 10 (Julia) with KDE distribution, which is also based on Ubuntu 10.10 (Maverick Meerkat) was chosen for the next installation. Linux Mint is another distribution based on Ubuntu/Debian. All the various commands mentioned in previous posts for Kubuntu also apply to Mint.

However, several issues were discovered in the various required tools that were not mentioned previously. There are some additional packages required for building GCC 4.6.0 and its prerequisite libraries (the GNU autotools) and for building CMake (specifically the ccmake GUI). The Linker Problem on Linux and New Make System posts were updated with new information about these dependencies. After a day of using Linux Mint 10, it is running as expected for a modern operating system on a fairly decent (though not cutting edge) system.

Using CMake on Windows

The latest version of CMake (2.8.4) for Windows was installed, which can be obtained from here (the Win32 Installer). The same version used on Kubuntu 10.10. On Windows, there is a CMake (cmake-gui) program installed in the CMake 2.8 start menu group and it took some effort to figure out how to use it. Here is the procedure for building the program, starting in the CMake (cmake-gui):

1. Select "Browse Source..." and point to where the source is
2. Select "Browse Build..." and point to where the build will be (which should not be the source directory)
3. Optionally there is a "Make New Directory" to create a "build" directory
4. The next step is to select the "Configure" button, however, if the "Current Generator:" does not say "None" then select File/Delete Cache to make it change to None.

Before doing these steps, CMake needs to be able to find the GCC compilers and other tools. The Windows environment variable PATH needs to be modified, which can be done by inserting "C:\MinGW\bin;C:\MinGW\msys\1.0\bin;" at the beginning of PATH (assuming the default directories were used in the MinGW installer). Upon selecting "Configure" in Step 4, CMake will pop up a dialog. Change the generator for the project to "MSYS Makefiles" and select the "Finish" button. If all goes well, some lines will be output in the bottom section with no errors (errors are output in red text).

Now select "Generate" and the make file will be created. In an MSYS window, change to the build directory and do a "make" command. The program should now be built. All the generated files will be in the build directory including the executable, the auto-generated header files and the codes.txt file.

Using the CMake System

There are a few simple steps for building using the CMake system. It's worth noting that when using CMake, all the output files are put into a separate build directory. This prevents the source directory from getting cluttered with files. Assuming that the source files are in the ibcp sub-directory, an ibcp-build directory will be created. These commands are used for building the project:

1. mkdir ibcp-build
2. cd ibcp-build
3. cmake ../ibcp
4. make

If no errors occur from Step 3 (like the correct version of GCC was not found), then Step 4 will build the program. All auto-generated include files, object files and the program will be located in the ibcp-build directory. To enable debug information in the executable, the ccmake program is used with these steps:

1. ccmake ../ibcp
2. Press Enter on the CMAKE_BUILD_TYPE line
3. Type in "Debug" and press Enter
4. Press [c] to configure
5. Press [g] to generate and exit
6. Now entering make will build with debug information

Setting the string to "Release" in Step 3 will turn off the debugging information. There is also a "make clean" option. If the build is not performed in the source directory (possible but not recommended), the entire contents of the build directory can also be deleted and cmake rerun. Now that CMake is working on Linux, it is now time to make it work under Windows...

The CMake System

The CMake system consists of a CMakeLists.txt file that describes how to generate the make file that will be used for building a project. The initial CMakeLists.txt file consisted of these major sections:

1. Check that the GCC version is 4.5 or later.
2. Find the awk program.
3. Pass version information to the IBCP source.
4. Create custom commands for the auto-generated header files.
5. Define the lists of header and source files.
6. Add linker definitions for the static library linking.
7. Define the executable.

Step 1 is needed to make sure the needed version of the GCC compiler is installed and setup correctly. Step 2 checks if the awk program is available and gets it's path - the awk program is needed to create the auto-generated header files. Step 3 is to pass program version and copyright information to the source program. This is accomplished by an ibcp_config.h.in file with definitions set to CMake variables to create an ibcp_config.h file that will be included in the source.

Step 4 contains two custom commands to run awk to create the two auto-generated header files. The awk scripts were modified to take a command line argument for the path of where the output files are to written to since the awk scripts will be run in the build directory, not in the source directory. This path argument was made optional and if not provided, the awk scripts assume the current directory. In the CMakeLists.txt custom commands, the source directory path is passed to the awk scripts. The auto-generated header files are now written into the build directory (CMake calls this the binary directory), not the source directory, so a statement was also needed in CMakeLists.txt to include the binary directory to the list of paths that are searched for header files.

In Step 5, the list of header and source files are put into CMake variables. These variables are used to define what the dependencies are and are used to build the executable. Step 6 adds the ‑static‑libgcc and ‑static‑libstc++ linker options. Finally, Step 7 defines the executable, which depends on the auto-generated header files, the project header files and the source files. The ".exe" is not needed on the executable program name as CMake will automatically add it for Windows, but not for Linux (CMake is aware of which system is being used for building).

New Make System

The standalone make file was originally generated by a feature in NetBeans, and was heavily modified to make it work, but it never looked very clean. Some consideration was given to rewriting it using the proper make rules (having individual build rules for each individual source file is not the way make is suppose to be utilized).

Continuing to use the make system within NetBeans was not desirable, though it allows the setup to deal with multiple platforms (Linux and Windows). A reason for not using it is that releasing all the associated files is not really an ideal option. And it never really worked right for the auto-generated include files from the awk scripts - warnings were generated for every build and no solution for eliminating these warnings was ever found (perhaps this was a bug in 6.9.1 that may have been fixed in 7.0).

Using the GNU autotools was out of the question - way too complicated and too much time coming up to speed with it. Instead the CMake system (Cross platform make generator) will be used as it is far simpler to setup and use. Plus NetBeans is supposed to understand CMake files. Adding CMake on Kubuntu installed version 2.8.2, however, this version did not contain the ccmake user interface for configuring the build. So the Linux source for version 2.8.4 (the latest) was downloaded (from here) and installed.

There are two prerequisite packages that are required to build the text CCMake GUI (more on this later), which are the libncurses5 and libncurses5-dev packages. These can be installed with this command:

sodu apt-get install <package-name>

Upgrade to NetBeans 7.0

Netbeans 7.0 was recently released (version 6.9.1 was being used for development on Windows), which contains support for git via a plug-in. Getting that to work took some research and experimentation. It appeared that the git plug-in only allowed creating a new git repository, not opening an existing one. However, it was discovered that if a new project is created from existing source and standalone make file in a directory with a git repository, NetBeans will then see the repository.

There was a problem building the project. The compiler complained that it didn't recognize the -static-libstdc++ option. NetBeans was still executing GCC 4.4.5, even though it was correctly pointing to the GCC 4.6.0 directory. Setting the path to include GCC 4.6.0 before executing NetBeans from the command line did not help.

Next an attempt was made to run the debugger on the project. There was no debug option in the standalone make file. Previously, the project was set up under NetBeans and it generated its own complex and convoluted set of files for generating what it needed to build the project. Both a Release and a Debug configuration was created by NetBeans. To release the project, NetBeans was used to create a standalone make file instead of releasing all the NetBeans files, which would have required NetBeans to build.

This time around, NetBeans was given this (heavily modified) make file when the project was created, though there was no Debug configuration. The make file was temporarily modified to turn on debug compiling. The make system needs to be reworked to allow both a release and debug configuration without using the system built into NetBeans.

Linker Problem on Linux

Now that the program builds and runs on Windows, it was time to make sure it still worked on Linux. However, the linker failed because the compiler/linker did not understand that ‑static‑libstdc++ option. Upon research, it was discovered that this option was only in GCC version 4.5 and later, so it was not available in the GCC 4.4.5 on Kubuntu.

The latest version of GCC (4.6.0) was downloaded and installed on Kubuntu using the source code. This required quite a bit of work to accomplish (for complete details, hit the continue link). Now that the project was building on both Linux and Windows, it was time to introduce it to the NetBeans IDE (which was now at version 7.0, version 6.9.1 was being used for development on Windows).

Running on Windows

There were no problems building the modified project back on Windows, however, when running the regression test script on Windows, all tests failed. This occurred because the expected test results files were in Unix (LF only) format, but the freshly generated output files were in DOS (CRLF) format.  The files otherwise appeared to be identical.

To verify that the output was the same, the test script was modified where the cmp ‑s command (‑s for silent) was replaced with a diff ‑bq command (‑b for ignore white-space, ‑q for quiet) to have it ignore the differences between CRLF and LF, both considered white-space. Now all tests compared to the expected results. A better solution will need to be found to deal with the differences between Windows and Linux.

One other problem was found when trying to run on Windows, which was discovered when trying to make the program run. Upon the first run, an error was reported that the application failed to start because libstdc++‑6.dll was not found. Once the environment variables were adjusted to include the MinGW binary path, the program ran fine.

This was the same problem as before with libgcc_s_dw2‑1.dll. This problem was resolved by adding a -static-libgcc option to the linker step, which links in the libgcc.a library. It turns out that there is similar option ‑static‑libstdc++ for linking in the libstdc++.a library. The program no ran without requiring any external library. This was tested by moving the program to another computer that did not have MinGW installed.

Setting up and Building on Windows

Now that the code was building and working on Linux, it was time to setup the Windows XP virtual machine for building.  On the previously Windows XP installation, MinGW (with GCC 3.4.6) and MSYS (1.0.11) were installed with a number of packages installed later (like GCC 4.4.0). It was a very hacked together setup in attempt to make things work right.

This time around, the Automated MinGW Installer was used. The current 2011-05-30 version was used, which contains the latest versions of MSYS, MinGW and all the other utilities.  The latest version can be obtained here. When installing this package (which actually downloads all the latest versions before installation), both the MSYS and MinGW options were selected. The version of GCC installed was 4.5.2.

Moving the project software back over to Windows, the build succeeded. Turns out the awk scripts still work in Windows. There was also no warning about the gets() function. This may be because the version implemented with MinGW contains more features, which were mentioned back on June 15, 2010 and are not implemented in the Linux version.

Integers on 64-bit Linux

After a successful compile, the regression test script aborted with errors. The errors occurred because it was trying to execute ibcp but the make file was building ibcp.exe. This worked on Windows because it will execute ibcp.exe when the command is ibcp. The script was modified to run ibcp.exe instead (this issue will be resolved shortly). Now only two of the parser tests and one of the translator test failed.

The first parser test (immediate commands) was not working on the inputs containing large values for line numbers or increments that were suppose to be reported as an error, but now were being accepted and the value output was not the same as the input. A problem in the third parser test (numbers) was a negative integer one beyond the limit of an integer, was now also being accepted as an integer (it should have become a double since it was out of range for an integer). And a problem on the 17th translator test (negative values) was another negative number out of range of an integer, was also being accepted.

These problems were caused by the strtol() function calls used to convert strings to an integer. An overflow was expected for these large numbers. This function returns a long integer and on 32-bit Windows (or any 32-bit OS) this is the same as integer values, which is 32 bits, so an overflow was reported. However, on 64-bit Linux (or any 64-bit OS), long integers are 64 bits, so these large values were now being accepted, but when converting to an integer, were being truncated from 64 bits to 32 bits.

To correct this problem, in additional to the ERANGE (overflow) error occurred condition, a condition was added to check if beyond the maximum integer using the predefined constant INT_MAX (and INT_MIN when the value could also be negative).

The remaining problem in the third parser test was that only two digits were output for floating point exponents on Linux if a third (the hundreds) digit was not required. On Windows, three digits are always output for the exponent. This must be due to slightly different builds or implementation of the standard C library functions. How to deal with this problem for testing will be dealt with a little later.

Resolving 64-bit Linux Build Problems

There were problems with the awk scripts - they were not generating the auto-generated includes files. This was caused by the RS (Record Separator) and ORS (Output Record Separator) variables being set to CRLF to work with Windows. The files were now all in Unix (LF) format and so the input source files not being read correctly. So the RS and ORS lines were commented.

The git configuration option autocrlf was set to false, causing git upon checkout to change any DOS (CRLF) format files in the repository to be converted to Unix format (LF), or the CRLF was removed when the CVS repository was converted to git. There were previous problems with CRLF on Windows when switching between cvs with MSYS to CVSNT (all the files were changed to DOS format since CVSNT didn't seem to like Unix format files).

Next there were warnings about the gets() function calls in the test routines, which were not present on Windows (using GCC 4.4.0 on MinGW/MSYS). After updating with the package manager, Kubuntu had GCC 4.4.5, so perhaps this was a change between 4.4.0 and 4.4.5 or a difference between the GCC build on MinGW vs. Kubuntu. The warning complained that the returned value from gets() was being ignored. To remove the warnings, statements were add to check the return value.

The linker also warned that the gets() function is dangerous and should not be used. This is probably because gets() does not check for a buffer overflow of the allocated buffer given to it. The buffer allocated is plenty big for testing here and this is just the test code anyway, so this warning will be ignored from the time being.