Saturday, October 6, 2012

Another Windows AWK Script Problem

I spoke too soon, in doing some final testing with CMake building, the other awk script enums.awk (used to generate autoenums.h from table.cpp for the code enumeration and ibcp.cpp for the token status enumeration) also did not work correctly on Windows.

Again, the solution used in winfix0.1.15 with setting the record separator variables can't be used because it doesn't work on Linux.  Therefore, a similar solution used for test_codes.awk was used in both parts of enums.awk, namely looking for a CR character and removing it if present and doing nothing if not.  This change was committed and push to github.

While testing, another CMake problem was also discovered.  For some reason, a compile error occurs when building in the ibcp directory (instead of in a different build directory).  I'm not sure why this is happening.  This is not recommended, but it should still work.  It was also noticed that the old Makefile was still in the directory and needs to be removed.

Obtaining New Commits

If the IBCP repository has not been cloned yet, then there is no reason to read this post.  New commits, like the fix mentioned in the last post, can be obtained using the Git GUI.  The fix commit was made on the cmake0.1.16 branch, so this branch should be checked out first.  On the Remote menu, select Fetch from and select origin.  This should fetch the new commit.

The fetch reads the new commit and updates the remotes/origin/cmake0.1.16 branch, but not the local cmake0.1.16.  The remote branch needs to be merged with the local branch.  This is a fast-forward merge (only the branch pointer is moved).  On the Merge menu select Local Merge... and make sure it says Merge Into cmake0.1.16 (in other words, this branch is checked out).  Select origin/cmake0.1.16 and click the Merge button.

For me, this worked on Linux and Windows 7, however, on Windows XP there was an error that the user name and email address was not set (it wasn't set on Windows 7 either, but no error occured).  So if this error does occurs, on the Edit menu select Options... and enter a User Name and Email Address on at least the ibcp Repository (left) side .  These are used when making commits.  The merge then worked without incident.

From the command line, use the git merge origin/cmake0.1.16 command assuming cmake0.1.16 if currently checkout (which can be verified using the git status command).  Next up, finally building this latest development branch with CMake... 

Windows AWK Script Problem Revisited

Before moving on how to build with CMake, A problem was discovered on Windows while writing the build procedure.  Specifically with the test_codes.awk script that generates the test_codes.h header file (containing the strings of the code names used in the parser test code).  This is the same problem corrected with the winfix0.1.15 branch off of the master branch.

The same fix can't be used for Linux as the script no longer works on Linux.  The issue is that when git checks out files are Windows, all the source files are automatically converted to DOS (CRLF) format while on Linux, they are left in Unix (newline) format.  Because the table.cpp source file (the source of the code names) is in DOS format on Windows, there is an extra CR at the end of each line, which is treated as a normal character.  The way this was fixed in the winfix0.1.15 branch was to set the record separator (RS) variable to "\r\n"  in the awk script so that CRLF characters are properly handled as line separators.

However, setting RS to "\r\n" on Linux breaks the awk script because now it sees a single huge line since there are no "\r\n" to be interpreted as line separators in the table.cpp source file that is in Unix format.  Currently  the script was subtracting 5 from the length of the field containing the "Xxx_Code" characters, removing the "_Code" part to get the "Xxx" part.  With the extra CR character, the "_" was left on the string leaving "Xxx_" instead.

The length function was changed to the index function to search for the position of the "_Code" part.  One was subtracted from this value to calculate the length of the "Xxx" part.  This change works with both DOS and Unix format files.  This change was committed and pushed to the github repository.  Next up, how to obtain this latest commit for an already cloned repository...

Getting The Latest Development Branch

Using gitk, it can be seen that the cloned IBCP git repository contains all of the commits (shown with blue circles), branches (shown in green boxes), and releases (tags, shown in yellow boxes) created during the project's existence.  The only local branch is master (shown in bold).  The rest of the branches are known as remote branches (shown in tan with the orange branch boxes).  The current commit is shown with a yellow circle (the currently selected commit is highlighted in blue).

All remote branches are preceded with a remotes name, followed by origin, denoting which remote the branch is from - origin is the name given to the remote repository the repository was cloned from.  These remote branches show where the branches were the last time the remote repository was accessed.  To switch to a branch using gitk, right-click on the branch and select Check out this branch.  However, only local branches can be checked out this way.

To switch to the latest development branch, which is currently remote branch remotes/origin/cmake0.1.16, a local branch needs to be created.  All branches besides master are remote branches (called tracking branches in the Git GUI).  To create a local branch for cmake0.1.16, on the Branch menu, select Create..., enter cmake0.1.16 for the Name, select Tracking Branch, select origin/cmake0.1.15, uncheck the Fetch Tracking Branch option (this options causes the remote repository to be read, but this branch is already in the local repository, so it's not necessary to read again).

The Current Branch will now be local branch cmake0.1.16.  In gitk, you can see that this branch is now checked out, by selecting Reload on the File menu (or pressing Ctrl+F5) and cmake0.1.16 will now be bold (previously master was bold).

Hit Continue... for instructions on doing these operations from the command line (for now on when command line is mentioned, this will mean either MSYS on Windows, or a Terminal on Linux and any differences between the too will be described).  Next up, building this latest development branch with CMAKE...

Building On Windows

Once the IBCP git repository has been cloned, the program can be built using the MSYS command line.  If the Git GUI was used to clone the repository as instructed, once the MSYS command line is started, there will be the ibcp directory.  Change to this directory and build the same as Linux with the make command.

However, there a problem on Windows with the awk scripts - the record separator (RS and ORS) assignments to "\r\n" lines, that are needed to work with DOS (CRLF) text file format, were removed during the clean up and repair of the git repository so that they would work on correctly on Linux.  However, this broke the scripts on Windows.  This showed up when running the regtest script and all the parser tests failed (the translator tests did all succeed).

These assignments were put back and a new commit was made to new branch winfix0.1.15 off of master.  So, to see the parser tests succeed on Windows, checkout out this branch with the git checkout winfix0.1.15 command.  If the make was already performed, do a git clean ‑df command first and run make again.  Now all tests should succeed when running the ./regtest script - at least on Windows XP.

However, on Windows 7, parser test 3 fails.  The specific test that fails is the smallest constant out of range check for the value 1.234e‑308, which does not return an error on Windows 7, whereas on Window XP and Linux it does.  Briefly investigating this, Windows 7 appears to allow even smaller values before reporting an out of range error, which don't occur until the value gets to around 1e‑323.  This will be investigated further later on the latest development branch (assuming the problem still exists). 

Making the awk scripts work on all platforms will be resolved on the latest development branch.  Next up, building the latest development branch...

Friday, October 5, 2012

Obtain IBCP Repository on Windows

There are two ways to obtain the IBCP repositories on Windows.  The first is using the MSYS command line, simply use the git clone command the same as on Linux:
git clone https://github.com/thunder422/ibcp.git
The second is using the Git GUI, first start the GUI by selecting Start, All Programs, Git, Git GUI.  Select Clone Existing Repository.  Enter https://github.com/thunder422/ibcp.git in the Source Location.  Enter C:/MinGW/msys/1.0/home/username (where username is your user name) in the Target Directory (or browse you MSYS home directory and add "/ibcp" to the end in the Target Directory entry, but don't use the Make New Folder in the location browser dialog to create the ibcp directory or an error will cloning).  Now click the Clone button.

Once the repository is cloned, the Git Gui (ibcp) window is presented, which will be mostly empty and just below the menu bar, there is a line that has the Current Branch: master indicating the current branch.  While it doesn't appear that anything was downloaded, you can see the repository by selecting Visualize All Branch History in the Repository menu.  This will start the gitk GUI.

From the gitk GUI, all the commits are in the upper pane, the differences of the currently selected commit is shown in the lower left pane and the files modified for this commit is shown in the lower right pane.  Though not obvious, the sizes of each pane can be adjusted by maneuvering the mouse cursor to the area between the panes until the mouse changes to the double arrow, then click, hold and drag to the desired size.  For the upper and lower panes, this area is between the Find next/prev line (used for finding commits) and the Search line (for searching the differences).  The sizes of the three columns in the upper pane can also be adjusted.

The gitk GUI also has several options for searching through the commits and how to display the differences.  Next up, building on Windows...

Windows Prerequisites For IBCP+Git

A couple of programs need to be installed on Windows to be able to obtain and build the IBCP program.  Instructions for both Windows XP (32-bit) and Windows 7 (64-bit) will be given - adjust accordingly for other versions (these are the two versions of Windows I have available).

First the MinGW (Minimalist GNU for Windows) package needs to be installed.  The version I am using, which is the latest as of this date, can be obtained from Sourceforge/MinGW.  Download and run the mingw‑get‑inst-20120426.exe installer.  In the installer, use the defaults (especially "Use pre-packaged repository catalogs" on the third dialog).  On the seventh dialog "Select Components", select C++ Compiler, MSYS Basic System, and MinGW Developer Toolkit in addition to the C Compiler.  This will download and install the GCC and MSYS programs.  The version of GCC installed is 4.6.2 (close to the version 4.6.3 on Mint 13).

Next, the Git for Windows package needs to be installed.  The Git‑1.7.11‑preview20120710.exe installer can be downloaded from msysgit Downloads.  When installing this program at the Select Components dialog, under Windows Explorer integration select Simple context menu to install Git Bash and Git GUI.  Select the defaults for the rest of the dialogs.

Now the git package needs to be integrated with the MSYS package.  All that is necessary is that the PATH environment variable needs to be modified to include where the git programs are installed.  For Windows XP, this is C:Program Files/Git/bin and for Windows 7, this is C:Program Files (x86)/Git/bin.  Within the MSYS command line, edit the file /etc/profile (or with Notepad - the file is located at C:\MinGW\msys\1.0\etc\profile) and add the following line after all other lines that modify PATH:
Windows XP: export PATH="$PATH:/c/Program Files/Git/bin"
Windows 7:   export PATH="$PATH:/c/Program Files (x86)/Git/bin"
(Update October 6, 2012) In order for the "/c" part to work in MSYS, the /etc/fstab (C:\MinGW\msys\1.0\etc\fstab) file system table file needs to be modified by adding a new line containing "C:\  /c" to give access to the entire C drive.

If necessary, restart the MSYS command window, and confirm git is there by using the git ‑‑version command within the MSYS command window.  Next up, obtaining the IBCP repository on Windows.

Thursday, October 4, 2012

Accessing and Building on Linux

The git clone command is used to obtain a git repository.  To obtain the IBCP git repository from github.com use the command:
git clone https://github.com/thunder422/ibcp.git
This will create the ibcp directory and populate ot with the source files for the master branch (which is currently Release 0.1.15).  However, the entire repository is present (there is a hidden .git directory).  The commits can be viewed using the git log command (there are many options, which is why installing the git documentation is strongly recommended).  The repository can also be viewed in GUI form using the gitk command if installed.

At this point, the program can be built using the make command in the ibcp directory.  There are a couple of compiler warnings, which were corrected in the 0.1.6 development branch and can be ignored.  The parser and translator tests can be run with the ./regtest command.  Currently on Mint 13 (64-bit) on Release 0.1.5 (master), parser tests 1 and 3 fail.

Parser test 1 fails due to a bug with parsing integers, which did not occur on 32-bit Windows.  This problem was corrected in the 0.1.16 development branch.  Parser test 3 also fails because of an integer parsing error and due to a change in output of exponentiation numbers (64-bit Linux outputs only 2 digit exponents, but Windows outputs 3, which may also be due to the particular compiler or library versions).  The exponent issue has yet to be corrected.

Next up, how to build the latest development branch, which is also contained in the git repository and uses cmake to create a make file...

Wednesday, October 3, 2012

Linux Prerequisites For IBCP & Git

Several software packages are required to obtain and build the IBCP program from the git repository, which include:
g++    Needed to compile C++ programs (brings libstdc++6 with it)
git    Needed to retrieve the IBCP Git repository
cmake  Needed for the CMake utility
These can either be installed from the GUI package manager (for example the Synaptic Package Manager or Muon Package Manager) or from the command line using the apt-get command, as in sudo apt-get install g++ cmake git.  There are several other packages that are nice to have, but are not required, which include:
gitk              a GUI for displaying a git repository
git-gui           a GUI for working with the git repository
git-man           the manual pages for the git commands
git-doc           alternate documentation for git
cmake-curses-gui  a terminal GUI for cmake (ccmake)
cmake-qt-gui      a nice Qt GUI for cmake (cmake-gui)
I should note that I actually obtained git by cloning the official git source repository and building the latest version (1.7.12.1) myself though git needs to be installed before this can be done.  Quite a few more packages are required to build all of the git package and documentation.  However, the version in the repositories (1.7.9.5 in Mint 13/Ubuntu 12.04) is sufficient for working with the IBCP git repository.  Next up, how to obtain and build the IBCP program.

Tuesday, October 2, 2012

Using Linux Mint

Linux Mint 10 KDE Julia proved to be much more stable that Kubuntu 10.10, however, after about week or so of running, it would either freeze or get so sluggish that it was unusable.  This appeared to occur when memory usage got up over 85% or more.  I suspected the problem was due to problems in Linux when using swap space.  After adding another 4 GB to the component to bring the total to 8 GB, Mint 10 was rock solid, at times running over a month with no issues.

Mint 10 KDE was a good platform and the CUDA SDK (version 4.0) worked on this system.  CUDA was not needed for this project, but is needed for a work project, the reason for going to Linux in the first place (which was also required).  Mint 12 KDE Lisa (based on Ubuntu 11.10 Oneiric Ocelot) was released (there was no Linux Mint 11 KDE Katja).  I installed Mint 12, but there were a number of issues.  Since I made the system dual boot (or should say multiple boot because the original Windows XP system and even the Kubuntu 10.10 install was still available), I just went back to Mint 10 as it was stable and met my needs.

However, sometime early last spring when the then next version of Firefox and Thunderbird was released, I thought it strange after many weeks the new versions were not showing up in the repositories.  When I investigated, I learned that Mint 10 was at end-of-life (meaning no more updates).  So it was time to move to Mint 12.  Some time was spent getting Mint 12 working they way Mint 10 was.

Monday, October 1, 2012

Public Git Repository

Initially CVS (concurrent versions system) was used for this project and the repository was converted to git using the git cvsimport command.  Before making the git repository public, I wanted to correct some issues that occurred during the conversion, specifically branches were not converted correctly.  Merges to the trunk (CVS nomenclature) did not show correctly as a merge to master (git nomenclature) - in fact there was no actual merge, just a hanging branch which a merge commit from where the branch first branched off the master branch.

These issues plus several other issues were corrected and the public git can now be accessed at github, where the repository can be explored, including the commits, source files, and branches.  Hit the Continued... for more details of the repairs.  There is also a Wiki on github, but for now it only points back to here.  The current branch is cmake0.1.16 (which is off of the branch0.1.16 development branch, but the CMAKE implementation is not quite finished yet).  Next up, some details on my current Linux development platform (there have been some changes).

Saturday, September 29, 2012

Long Time No Posts

It has been a long time with no posts.  I have been busy with other things, but it is time to get this project started back up with a change in focus (more on this shortly).  The primary purpose of this project has been a platform for learning.  Initially this was to get back into C++, something that I hadn't been using for nearly a decade (I had been using C, but there is quite a different between the two).  Along the way git (software revision control) and cmake (cross platform make) was introduced along with the Netbeans development environment.

Speaking of git, the first thing I wanted to accomplish in this restart was to make the git repository for this project public.  Sourceforge allows a public SVN (subversion) repository, but I am not familiar with SVN and wasn't really interested in learning SVN since I otherwise wouldn't be using it (I use git extensively for work and have got quite fond of it - for more on git, read the excellent book Pro Git by Scott Chacon).  But first, I wanted to clean up the existing git repository before making it public, which I spend the last several weeks working on...

Saturday, June 25, 2011

Switch in Linux Distribution

The 64-bit Kubuntu 10.10 installation has been problematic. Several comments around the Internet have indicated this to be the case. The reason Kubuntu was chosen was that it contained the KDE desktop, and 10.10 was chosen because of it is the newest on the supported host operating system list for VMWare Workstation (being used for the Windows XP virtual machine). Standard Ubuntu comes with the GNome interface, which I am not a fan of.

The problems with Kubuntu mainly include general sluggishness, a problem that was never resolved after a little over a month of use (something not expected on a Phenom II quad code running at 3.2 GHz with 4GB of memory with a NDIVIA GTS-450). It would completely hose up at times (not lock up or crash) when doing even simple tasks like uncompressing files. Even after all the updates were applied (less the update to 11.04 Natty Narwhal), including KDE and all the fancy desktop effects were turned off, the problems persisted. Mounting a bunch of NTFS partitions (from the previous Windows XP installation) was also suspected to be a cause.

Therefore, the Linux Mint 10 (Julia) with KDE distribution, which is also based on Ubuntu 10.10 (Maverick Meerkat) was chosen for the next installation. Linux Mint is another distribution based on Ubuntu/Debian. All the various commands mentioned in previous posts for Kubuntu also apply to Mint.

However, several issues were discovered in the various required tools that were not mentioned previously. There are some additional packages required for building GCC 4.6.0 and its prerequisite libraries (the GNU autotools) and for building CMake (specifically the ccmake GUI). The Linker Problem on Linux and New Make System posts were updated with new information about these dependencies. After a day of using Linux Mint 10, it is running as expected for a modern operating system on a fairly decent (though not cutting edge) system.

Thursday, June 23, 2011

Using CMake on Windows

The latest version of CMake (2.8.4) for Windows was installed, which can be obtained from here (the Win32 Installer). The same version used on Kubuntu 10.10. On Windows, there is a CMake (cmake-gui) program installed in the CMake 2.8 start menu group and it took some effort to figure out how to use it. Here is the procedure for building the program, starting in the CMake (cmake-gui):
  1. Select "Browse Source..." and point to where the source is
  2. Select "Browse Build..." and point to where the build will be (which should not be the source directory)
  3. Optionally there is a "Make New Directory" to create a "build" directory
  4. The next step is to select the "Configure" button, however, if the "Current Generator:" does not say "None" then select File/Delete Cache to make it change to None.
Before doing these steps, CMake needs to be able to find the GCC compilers and other tools. The Windows environment variable PATH needs to be modified, which can be done by inserting "C:\MinGW\bin;C:\MinGW\msys\1.0\bin;" at the beginning of PATH (assuming the default directories were used in the MinGW installer). Upon selecting "Configure" in Step 4, CMake will pop up a dialog. Change the generator for the project to "MSYS Makefiles" and select the "Finish" button. If all goes well, some lines will be output in the bottom section with no errors (errors are output in red text).

Now select "Generate" and the make file will be created. In an MSYS window, change to the build directory and do a "make" command. The program should now be built. All the generated files will be in the build directory including the executable, the auto-generated header files and the codes.txt file.

Tuesday, June 21, 2011

Using the CMake System

There are a few simple steps for building using the CMake system. It's worth noting that when using CMake, all the output files are put into a separate build directory. This prevents the source directory from getting cluttered with files. Assuming that the source files are in the ibcp sub-directory, an ibcp-build directory will be created. These commands are used for building the project:
  1. mkdir ibcp-build
  2. cd ibcp-build
  3. cmake ../ibcp
  4. make
If no errors occur from Step 3 (like the correct version of GCC was not found), then Step 4 will build the program. All auto-generated include files, object files and the program will be located in the ibcp-build directory. To enable debug information in the executable, the ccmake program is used with these steps:
  1. ccmake ../ibcp
  2. Press Enter on the CMAKE_BUILD_TYPE line
  3. Type in "Debug" and press Enter
  4. Press [c] to configure
  5. Press [g] to generate and exit
  6. Now entering make will build with debug information
Setting the string to "Release" in Step 3 will turn off the debugging information. There is also a "make clean" option. If the build is not performed in the source directory (possible but not recommended), the entire contents of the build directory can also be deleted and cmake rerun. Now that CMake is working on Linux, it is now time to make it work under Windows...

Monday, June 20, 2011

The CMake System

The CMake system consists of a CMakeLists.txt file that describes how to generate the make file that will be used for building a project. The initial CMakeLists.txt file consisted of these major sections:
  1. Check that the GCC version is 4.5 or later.
  2. Find the awk program.
  3. Pass version information to the IBCP source.
  4. Create custom commands for the auto-generated header files.
  5. Define the lists of header and source files.
  6. Add linker definitions for the static library linking.
  7. Define the executable.
Step 1 is needed to make sure the needed version of the GCC compiler is installed and setup correctly. Step 2 checks if the awk program is available and gets it's path - the awk program is needed to create the auto-generated header files. Step 3 is to pass program version and copyright information to the source program. This is accomplished by an ibcp_config.h.in file with definitions set to CMake variables to create an ibcp_config.h file that will be included in the source.

Step 4 contains two custom commands to run awk to create the two auto-generated header files. The awk scripts were modified to take a command line argument for the path of where the output files are to written to since the awk scripts will be run in the build directory, not in the source directory. This path argument was made optional and if not provided, the awk scripts assume the current directory. In the CMakeLists.txt custom commands, the source directory path is passed to the awk scripts. The auto-generated header files are now written into the build directory (CMake calls this the binary directory), not the source directory, so a statement was also needed in CMakeLists.txt to include the binary directory to the list of paths that are searched for header files.

In Step 5, the list of header and source files are put into CMake variables. These variables are used to define what the dependencies are and are used to build the executable. Step 6 adds the ‑static‑libgcc and ‑static‑libstc++ linker options. Finally, Step 7 defines the executable, which depends on the auto-generated header files, the project header files and the source files. The ".exe" is not needed on the executable program name as CMake will automatically add it for Windows, but not for Linux (CMake is aware of which system is being used for building).