Translator Exceptions – Development Strategy

The next incremental change was initially difficult to see for adding exception throws throughout the translator routines.  Starting at the bottom by changing the get token function to throw exceptions was problematic because all callers would have to catch the errors and create an error token to hold the column and length of the error.  Starting at the top by changing both the get expressions (used for testing) and get commands functions to throw exceptions was problematic because all functions in between would also need to be modified.  An attempt to change the entire translator was made.

After the code was modified and corrected for compile errors, the tests were run, but there were many problems, which was not unexpected considering the large number of changes made.  Instead of trying to debug, the changes were committed to a temporary work branch (though not pushed to the official GitHub repository).  This is a scheme to use git to temporary save work:

git checkout -b work           (create a temporary work branch)
git commit -a                  (stages all changed files and commits them)
git checkout -b misc-cpp-stl   (restore all files before changes)

This saved the changes of the failed attempt and restored the original files.  Changes from the work branch could now be transferred to the working directory piecemeal.  For example, several changes were made to the Token Error structure in the token header file (details in the next post).  For more details on using this scheme, click Continue...

Transitioning to the New Branching Model

The current topic in development was integrating the program model with recreator into the edit box, which has been going well, but there is at least one issue remaining.  When the text for a line entered in the edit box is recreated line replaced back into the edit box, extra undo commands are added to the undo stack.  So upon undoing the line, it firsts undoes the recreation and then undoes the change.  This issue will probably not be trivial to correct.

I want to start working on improving the C++ usage including using features that are part of the C++11 standard.  This implies a new topic branch for this work.  A topic branch is also needed for the recreator to edit box integration, somewhere on the current branch0.6 branch.  Finally for the new branching model, the develop branch is needed somewhere.

Since v0.6.1 was the last tag, the develop branch should go at this tag.  For the new C++11 branch, it should branch off of the develop branch (at this tag).  However, a lot of improvements are on the current branch0.6 branch, so I've decided to split branches further on from the last tag.

Therefore, the new develop branch was created at branch0.6.  The current branch0.6 was dead ended with an appropriate abandoned commit.  There will be no more of these branches.  Normally, git does not allow an empty commit with just a message, but this can be circumvented by using the --allow-empty option on the commit command.

A new recreator-editbox topic branch was placed at where the failing encoder test expected results were corrected before the latest two commits where the various Windows build issues were corrected since these commits have nothing to with this topic.  When work commences on this topic, changes can be merged from the develop branch.

[commit d3bb314f24] [commit 727f7d223f]

Project – New Git Branching Model

I found a better Git branching model.  Currently, a branch is created for the development release series.  For example, branch0.5 was created for all the 0.5.x versions (0.5.0 through 0.5.3).  Once this development series was completed, the branch was merged back to the master branch.  Since there were no other changes on the master branch, the pointer to master was simply moved from 0.4.6 (the last of the 0.4 development series) to 0.5.3 - what is known as a fast-forward merge.  In the end though, the repository is just a long row of linear commits.

The new Git branch model is detailed in this blog post.  I'm going to adopt this branching model.  The main development branch will be named develop, which will replace the branchX.X branches that has been used for development.  When development is ready for a release, a releaseX.X branch will be created from the develop branch.  There will be at least one commit used for updating the version number, and possibly another to do minor clean up (like fixing comments), plus any for possible platform related bug fixes discovered during testing.  This will replace the "updated files for release X.X.X" commits just before a release.  If testing goes well (on all platforms), then this branch will be merged to master and tagged for the release (as vX.X.X).

When a new feature, function or change is being added (what is called a "topic"), a topic branch will be created off of the develop branch and will be named for the topic.  This will more clearly state what is being developed (branchX.X states nothing except the release series).  When the topic is complete, it will be merged back to the develop branch.  If for some reason the topic doesn't work out, then the branch will abandoned and an empty commit will be made with the reason for the abandonment.  These topic branches will allow more when one topic to be worked on at a given time.

This model also allows for hit fixes to the master branch, where a hotfix-X.X.X branch will be created for the fix (which includes the proposed hot fix version number).  Once complete, the hot fix branch will be merged backed to the master branch, and it can be merged to the develop branch as needed.

As the blog post explains, all merges will use the no-fast-forward option (--no-ff), meaning that Git will make a merge commit showing the branch being merged instead of just moving the branch pointer.  Using this option will allow it to be cleary seen where the branch started and where it was merged.  With the current model, there is no way to see where the branch originally started, only where it ended up (assuming it isn't deleted, which is why the older development branches were not deleted).

Installing Git on Windows 8.1

Finally, to obtain this project from this project's Git repository on GitHub, the Git utility is required.  The latest Git for Windows can be obtained from their home page (which has changed since I previously described how to install Git).  Click the Download button to download the latest version.

Open the download file (Git‑1.9.4‑preview20140611) and accept the User Control Access if asked.  Click through and accept the license.  On the Select Components dialog, select additional icons if desired and click through accepting the defaults.  It's not necessary to view the release notes, so disable and click Finish.

The paths to these programs along with Qt needs to be added to Windows.  From Windows Explorer, select This PC, right-click and select Properties, Advanced system settings, and on the Advanced tab, click Environment Variables...  Under System variables, scroll down to Path and add:

;C:\mingw32\bin;C:\Qt\4.8.6\bin;C:\Program Files (x86)\Git\bin

Select Path, click Edit..., add the string above to the end of Variable value, and click OK (the whole way out).  This will allow the utilities git, g++ (compiler), gdb (debugger) and qmake (how it is determined where Qt is installed) to be found on the command line or by a program that is started under Windows.  The path to the CMake directory should already have been added during CMake installation.

New Git Repository Tagging Convention

The error highlighting implementation is now complete and it is time for a development release.  I made the decision to change the naming convention of repository tags from releaseX.X.X to vX.X.X, which is the convention used by many other projects under git revision control (like the git source repository itself).  All the "release" named tags have been replaced with the new "v" named tags.  These can be removed from a local repository using the command git tag -d `git tag -l release*` (backward single quotes) or the repository can be re-cloned.

To support this change, the CMake build configuration file CMakeLists.txt was modified for this naming convention.  A change was also required in the building of the version string in the command line class, which previously added seven to the release string pointer, seven being the length of the "release" part of the version string.  This could have simply been changed to one, the length of the "v" part of the string, but instead a little code was added to look for the start of the version number part of the string (by using a regular expression) so it doesn't matter what string precedes the version number.

The change in tag names will not cause a problem when building one of the archive files downloaded for a given tag because CMake will generate the version string being that the git repository will not be available.  Though the version announced as being built will still have the "release" string in the name.

However, if building at an older tag with the new tag naming convention, then these changes are needed, which is not an issue with any of the versions prior to 0.2.  CMake will not complain from versions 0.2-1 to 0.2-6 but the version number output will be messed up.  The changes need to be applied manually.  Starting with the final version 0.2.0 through the 0.3.X versions, the changes can be applied by git using the command git cherry-pick fa89 command where fa89 is the abbreviated commit ID of the commit with the changes.  To preserve this commit, a new branch can be created using the command git checkout -b xxx where xxx is the desired branch name.

Finally, previous blog entries were edited to reflect the new tag naming convention.  All blog previously given the Pre-Release blog tag was given the Tag blog tag.  Major releases still contain the Release blog tag (and not the Tag blog tag so they can be differentiated).

[commit fa89de8a0d]

Minor Build And Test Issues

A build issue was discovered where CMake does not create a release string if the git command is present but the git repository is not (for instance when building from a downloaded archive).  The git describe command was returning an error and no release string.  CMake now detects this situation and sets the release string the same as if no git command is found.

A test issue was discovered on Windows when building from a downloaded archive, which contains Unix format files (newline only) and not DOS format files (CRLF).  When the program is run from the regression test script (the program builds fine), the output files are in DOS format, but the compares fail because the expected output files are in Unix format.  The cmp command was changed to the diff command, which has an option to ignore the difference in the line separators (the ‑w ignore white space option).

All these changes have been pushed to GitHub and because of the build and test issues found, new tag v0.2‑2 was added.  The changes to the Parser were complete (with the attempt to compile next) before the text stream detour and these other minor issues.

[commit 0e85c83d56] [commit 2be4bd2f91]

Unique Release Number Implementation

To see if the git command is available, a find_program command was added to the CMakeLists.txt file, which sets the CMake variable PROGRAM_GIT.  An if command was also added that checks if this variable is set, which then executes the command using the  execute_process command for the describe sub-command in the project source directory setting the CMake variable ibcp_RELEASE_STRING with the resulting string.  The option OUTPUT_STRIP_TRAILING_WHITESPACE was needed to remove the trailing newline.

If the git program was not found, then the ibcp_RELEASE_STRING variable is set to "v" followed by the major and minor release numbers separated by a period.  If the patch release number is less than zero (development tag), then the patch number is appended to the release string (the dash is present because the number is negative).  Otherwise, the patch number is appended with another period separator.  The final release string, from either source, is output to the CMake output log.

In implementing this if command, it was realized that it was not necessary to test the PROGRAM_GIT variable for the string to be equal to "PROGRAM_GIT‑NOTFOUND" as was done with the awk program because this is one of the tests performed by the CMake if command.  Therefore, the if command for the awk program was simplified.

A define for ibcp_RELEASE_STRING was added to the ibcp_config.h.in file (used to auto-generate ibcp_config.h) with the contents of the CMake variable surrounded by quotes, to form a string constant that can used in the source code.  The ibcp_version() function was modified to output this string constant instead of the major, minor and patch numbers.  The "v" part of the string is not output (because "version" is already being output).  Finally, the GPL header was removed when outputting the version number to be consistent with the output of other programs.

New branch0.2 was created.  The release string produced at this latest commit will be v0.1.16b-1-g1bba2c3, which shows the most recent tag (v0.1.16b), the number of commits beyond this tag (1) and the short commit ID (1bba2c3).  The "g" in front of this ID stands for git (other letters would stand for other software configuration management systems).

[commit 1bba2c335c]

Unique Release Numbers

With a new release numbering scheme defined, there needs to be way to get the current version number into the program for output with the -v command line option (and eventually in the Help/About box once the GUI is implemented).  The version number should also be unique during development since not every commit will be tagged.  The goals are:

1. Use the current tag if at a tagged commit
2. Represent when not at a tagged commit during development
3. Use the current release number assigned for archive downloads
4. Allow for developmental (dash) and patch (period) numbering

The inclusion of the third goal will be clear shortly.  It turns out that the exact desired release string can be obtained using the git describe command, which returns the name of the most recent tag.  When beyond the most recent tag, the number of commits beyond the tag plus the short form of the current commit ID is appended.

This works as desired when the git command is available and the git repository information is present, but for the third goal, the downloaded archives have no git information.  Since the download archives are only available at tagged commits, the release number set in the CMakeLists.txt file can be used and will match the tag at that commit (assuming these variables were set to the same values as the tag).

For the last goal, there are two cases, git repository information present and not present.  When making tags during development, the tag name format will be releaseX.Y‑Z (note the dash).  The git describe command will pick this tag name (and append the rest if beyond that tagged commit).  When the git repository information is not present, the major, minor and patch release numbers set in the  CMakeLists.txt file will be used.  To handle developmental (dash) numbering, negative patch numbers will be used.

All Platform Parser Issues Resolved

On the Windows platform, the C format specifier "%g" outputs three digits in the exponent always whereas on Linux, only two digits output unless three are needed.  Looking at the C99 standard, it says basically "at least two exponent digits" should be output, so with that criteria, Windows is acceptable.  It also appears that Microsoft decided that three exponent digits should always be output, and the GNU compiler suite decided to follow this on Windows.  There is no way to control the number of exponent digits in the "%g" specifier.

The parser print token routine was modified to output two exponent digits if the first one is zero.  This was accomplished by first outputting the number into a temporary character buffer using the "%g" specifier.  Then it searches for an 'e' character in the string.  If the fourth character from the 'e' is not a zero-terminator, then that means there are three exponent digits.  If the second character from the 'e' is a '0' then the last two digits is moved on top of this zero.

Hit the Continue... for a code snippet of the solution.  This change will now output two exponent digits always if the first one is zero.  This code will do nothing on the Linux platform.  The expected results for parser test #3 were updated for two exponent digits were applicable (five instances).

Since the major CMake issues have been resolved and the 0.1.16 release will be about the build system, the cmake0.1.16 branch was merged to branch0.1.16 branch.  This last commit was also tagged as v0.1.16‑pre‑1.  On GitHub, a zip (or .tar.gz) files can be downloaded directly for a tag, which contain all the files at that tag, so it is not necessary to install git to retrieve the files.  This file does not contain the git repository however.  There are still a few more build issues to take care of.

Obtaining New Commits

If the IBCP repository has not been cloned yet, then there is no reason to read this post.  New commits, like the fix mentioned in the last post, can be obtained using the Git GUI.  The fix commit was made on the cmake0.1.16 branch, so this branch should be checked out first.  On the Remote menu, select Fetch from and select origin.  This should fetch the new commit.

The fetch reads the new commit and updates the remotes/origin/cmake0.1.16 branch, but not the local cmake0.1.16.  The remote branch needs to be merged with the local branch.  This is a fast-forward merge (only the branch pointer is moved).  On the Merge menu select Local Merge... and make sure it says Merge Into cmake0.1.16 (in other words, this branch is checked out).  Select origin/cmake0.1.16 and click the Merge button.

For me, this worked on Linux and Windows 7, however, on Windows XP there was an error that the user name and email address was not set (it wasn't set on Windows 7 either, but no error occured).  So if this error does occurs, on the Edit menu select Options... and enter a User Name and Email Address on at least the ibcp Repository (left) side .  These are used when making commits.  The merge then worked without incident.

From the command line, use the git merge origin/cmake0.1.16 command assuming cmake0.1.16 if currently checkout (which can be verified using the git status command).  Next up, finally building this latest development branch with CMake... 

Windows AWK Script Problem Revisited

Before moving on how to build with CMake, A problem was discovered on Windows while writing the build procedure.  Specifically with the test_codes.awk script that generates the test_codes.h header file (containing the strings of the code names used in the parser test code).  This is the same problem corrected with the winfix0.1.15 branch off of the master branch.

The same fix can't be used for Linux as the script no longer works on Linux.  The issue is that when git checks out files are Windows, all the source files are automatically converted to DOS (CRLF) format while on Linux, they are left in Unix (newline) format.  Because the table.cpp source file (the source of the code names) is in DOS format on Windows, there is an extra CR at the end of each line, which is treated as a normal character.  The way this was fixed in the winfix0.1.15 branch was to set the record separator (RS) variable to "\r\n"  in the awk script so that CRLF characters are properly handled as line separators.

However, setting RS to "\r\n" on Linux breaks the awk script because now it sees a single huge line since there are no "\r\n" to be interpreted as line separators in the table.cpp source file that is in Unix format.  Currently  the script was subtracting 5 from the length of the field containing the "Xxx_Code" characters, removing the "_Code" part to get the "Xxx" part.  With the extra CR character, the "_" was left on the string leaving "Xxx_" instead.

The length function was changed to the index function to search for the position of the "_Code" part.  One was subtracted from this value to calculate the length of the "Xxx" part.  This change works with both DOS and Unix format files.  This change was committed and pushed to the github repository.  Next up, how to obtain this latest commit for an already cloned repository...

Getting The Latest Development Branch

Using gitk, it can be seen that the cloned IBCP git repository contains all of the commits (shown with blue circles), branches (shown in green boxes), and releases (tags, shown in yellow boxes) created during the project's existence.  The only local branch is master (shown in bold).  The rest of the branches are known as remote branches (shown in tan with the orange branch boxes).  The current commit is shown with a yellow circle (the currently selected commit is highlighted in blue).

All remote branches are preceded with a remotes name, followed by origin, denoting which remote the branch is from - origin is the name given to the remote repository the repository was cloned from.  These remote branches show where the branches were the last time the remote repository was accessed.  To switch to a branch using gitk, right-click on the branch and select Check out this branch.  However, only local branches can be checked out this way.

To switch to the latest development branch, which is currently remote branch remotes/origin/cmake0.1.16, a local branch needs to be created.  All branches besides master are remote branches (called tracking branches in the Git GUI).  To create a local branch for cmake0.1.16, on the Branch menu, select Create..., enter cmake0.1.16 for the Name, select Tracking Branch, select origin/cmake0.1.15, uncheck the Fetch Tracking Branch option (this options causes the remote repository to be read, but this branch is already in the local repository, so it's not necessary to read again).

The Current Branch will now be local branch cmake0.1.16.  In gitk, you can see that this branch is now checked out, by selecting Reload on the File menu (or pressing Ctrl+F5) and cmake0.1.16 will now be bold (previously master was bold).

Hit Continue... for instructions on doing these operations from the command line (for now on when command line is mentioned, this will mean either MSYS on Windows, or a Terminal on Linux and any differences between the too will be described).  Next up, building this latest development branch with CMAKE...

Obtain IBCP Repository on Windows

There are two ways to obtain the IBCP repositories on Windows.  The first is using the MSYS command line, simply use the git clone command the same as on Linux:

git clone https://github.com/thunder422/ibcp.git

The second is using the Git GUI, first start the GUI by selecting Start, All Programs, Git, Git GUI.  Select Clone Existing Repository.  Enter https://github.com/thunder422/ibcp.git in the Source Location.  Enter C:/MinGW/msys/1.0/home/username (where username is your user name) in the Target Directory (or browse you MSYS home directory and add "/ibcp" to the end in the Target Directory entry, but don't use the Make New Folder in the location browser dialog to create the ibcp directory or an error will cloning).  Now click the Clone button.

Once the repository is cloned, the Git Gui (ibcp) window is presented, which will be mostly empty and just below the menu bar, there is a line that has the Current Branch: master indicating the current branch.  While it doesn't appear that anything was downloaded, you can see the repository by selecting Visualize All Branch History in the Repository menu.  This will start the gitk GUI.

From the gitk GUI, all the commits are in the upper pane, the differences of the currently selected commit is shown in the lower left pane and the files modified for this commit is shown in the lower right pane.  Though not obvious, the sizes of each pane can be adjusted by maneuvering the mouse cursor to the area between the panes until the mouse changes to the double arrow, then click, hold and drag to the desired size.  For the upper and lower panes, this area is between the Find next/prev line (used for finding commits) and the Search line (for searching the differences).  The sizes of the three columns in the upper pane can also be adjusted.

The gitk GUI also has several options for searching through the commits and how to display the differences.  Next up, building on Windows...

Windows Prerequisites For IBCP+Git

A couple of programs need to be installed on Windows to be able to obtain and build the IBCP program.  Instructions for both Windows XP (32-bit) and Windows 7 (64-bit) will be given - adjust accordingly for other versions (these are the two versions of Windows I have available).

First the MinGW (Minimalist GNU for Windows) package needs to be installed.  The version I am using, which is the latest as of this date, can be obtained from Sourceforge/MinGW.  Download and run the mingw‑get‑inst-20120426.exe installer.  In the installer, use the defaults (especially "Use pre-packaged repository catalogs" on the third dialog).  On the seventh dialog "Select Components", select C++ Compiler, MSYS Basic System, and MinGW Developer Toolkit in addition to the C Compiler.  This will download and install the GCC and MSYS programs.  The version of GCC installed is 4.6.2 (close to the version 4.6.3 on Mint 13).

Next, the Git for Windows package needs to be installed.  The Git‑1.7.11‑preview20120710.exe installer can be downloaded from msysgit Downloads.  When installing this program at the Select Components dialog, under Windows Explorer integration select Simple context menu to install Git Bash and Git GUI.  Select the defaults for the rest of the dialogs.

Now the git package needs to be integrated with the MSYS package.  All that is necessary is that the PATH environment variable needs to be modified to include where the git programs are installed.  For Windows XP, this is C:Program Files/Git/bin and for Windows 7, this is C:Program Files (x86)/Git/bin.  Within the MSYS command line, edit the file /etc/profile (or with Notepad - the file is located at C:\MinGW\msys\1.0\etc\profile) and add the following line after all other lines that modify PATH:

Windows XP: export PATH="$PATH:/c/Program Files/Git/bin"
Windows 7:   export PATH="$PATH:/c/Program Files (x86)/Git/bin"

(Update October 6, 2012) In order for the "/c" part to work in MSYS, the /etc/fstab (C:\MinGW\msys\1.0\etc\fstab) file system table file needs to be modified by adding a new line containing "C:\  /c" to give access to the entire C drive.

If necessary, restart the MSYS command window, and confirm git is there by using the git ‑‑version command within the MSYS command window.  Next up, obtaining the IBCP repository on Windows.

Accessing and Building on Linux

The git clone command is used to obtain a git repository.  To obtain the IBCP git repository from github.com use the command:

git clone https://github.com/thunder422/ibcp.git

This will create the ibcp directory and populate ot with the source files for the master branch (which is currently Release 0.1.15).  However, the entire repository is present (there is a hidden .git directory).  The commits can be viewed using the git log command (there are many options, which is why installing the git documentation is strongly recommended).  The repository can also be viewed in GUI form using the gitk command if installed.

At this point, the program can be built using the make command in the ibcp directory.  There are a couple of compiler warnings, which were corrected in the 0.1.6 development branch and can be ignored.  The parser and translator tests can be run with the ./regtest command.  Currently on Mint 13 (64-bit) on Release 0.1.5 (master), parser tests 1 and 3 fail.

Parser test 1 fails due to a bug with parsing integers, which did not occur on 32-bit Windows.  This problem was corrected in the 0.1.16 development branch.  Parser test 3 also fails because of an integer parsing error and due to a change in output of exponentiation numbers (64-bit Linux outputs only 2 digit exponents, but Windows outputs 3, which may also be due to the particular compiler or library versions).  The exponent issue has yet to be corrected.

Next up, how to build the latest development branch, which is also contained in the git repository and uses cmake to create a make file...

Linux Prerequisites For IBCP & Git

Several software packages are required to obtain and build the IBCP program from the git repository, which include:

g++    Needed to compile C++ programs (brings libstdc++6 with it)
git    Needed to retrieve the IBCP Git repository
cmake  Needed for the CMake utility

These can either be installed from the GUI package manager (for example the Synaptic Package Manager or Muon Package Manager) or from the command line using the apt-get command, as in sudo apt-get install g++ cmake git.  There are several other packages that are nice to have, but are not required, which include:

gitk              a GUI for displaying a git repository
git-gui           a GUI for working with the git repository
git-man           the manual pages for the git commands
git-doc           alternate documentation for git
cmake-curses-gui  a terminal GUI for cmake (ccmake)
cmake-qt-gui      a nice Qt GUI for cmake (cmake-gui)

I should note that I actually obtained git by cloning the official git source repository and building the latest version (1.7.12.1) myself though git needs to be installed before this can be done.  Quite a few more packages are required to build all of the git package and documentation.  However, the version in the repositories (1.7.9.5 in Mint 13/Ubuntu 12.04) is sufficient for working with the IBCP git repository.  Next up, how to obtain and build the IBCP program.

Public Git Repository

Initially CVS (concurrent versions system) was used for this project and the repository was converted to git using the git cvsimport command.  Before making the git repository public, I wanted to correct some issues that occurred during the conversion, specifically branches were not converted correctly.  Merges to the trunk (CVS nomenclature) did not show correctly as a merge to master (git nomenclature) - in fact there was no actual merge, just a hanging branch which a merge commit from where the branch first branched off the master branch.

These issues plus several other issues were corrected and the public git can now be accessed at github, where the repository can be explored, including the commits, source files, and branches.  Hit the Continued... for more details of the repairs.  There is also a Wiki on github, but for now it only points back to here.  The current branch is cmake0.1.16 (which is off of the branch0.1.16 development branch, but the CMAKE implementation is not quite finished yet).  Next up, some details on my current Linux development platform (there have been some changes).