Translator – More Operand Processing Debugging

Continuing with debugging, the next difference was a PRINT command that contained sub-string functions where the expected results needed to be updated (sub-string functions don't get attached tokens, tokens are attached to next token).

Finally in the Tenth (PRINT statements) test, there were three “debug #1” bug errors, which should have been an “expected numeric expression” errors. These test lines were incomplete statements (e.g. PRINT A+). This bug error was put in to fix later because the existing code was not correct. The code was modified to return the expected type error for the operand of the operand token on top of the hold stack (taking into account whether it is the first operand of a unary operator or second operand for a binary operator). It was previously set to return the variable type for the command on top of the command stack (which was wrong – the PRINT command has no data type).

Now on the the Eleventh (error statements) tests, which has quite of few incorrect error messages...

Windows, NetBeans and CVS (CVSNT)

This post is not strictly related to the IBCP project, but will be helpful explaining how NetBeans (6.9.1) on Windows (XP) was made to work with CVS since it took nearly a day or searching and experimenting to accomplish (and many others were having similar problems).

The initial hope was that NetBeans for Windows would play nice with the CVS installed with MSYS (Minimum SYStem), but it did not. Searching indicated that NetBeans requires a CVS server even for a local repository and apparently, the CVS that comes with MSYS is not sufficient, though when doing a cvs ‑v command, it does respond … 1.11 (client/server). Further searching uncovered The CvsGui project and the WinCVS 2.1.1.1 package (includes WinCVS and CVSNT) was downloaded from http://sourceforge.net/projects/cvsgui/files/. Initially only the CVSNT package was installed (and this may have been sufficient).

Not realizing that there was a Service control panel under the Start menu (under CVSNT) that is used to configure CVSNT, the WinCVS package was installed. Normally WinCVS will also install the required CVSNT (the CVS server), but can be skipped if CVSNT is already installed. WinCVS is a very nice GUI for looking at checked out software (including nice version and branch graphs), however, it does not contain any configuration of CVSNT. And exactly how to check out a module from the repository was not discovered. This was not important as either the command line or NetBeans can be used to check out a module.

Once CVSNT is installed (indicates version 2.0.51d), it requests that Windows be restarted. This is necessary to install and start the CVS services, which occurred automatically upon reboot. Next the CVS server needs to be configured by using the Service control panel. The first tab reported that the services (CVS Service and CVS Lock service) were both running (a good sign).

The CVS repository needs to be identified using the Repositories tab by selecting Add. Using the browse button on the next dialog (the ... button), the repository that had been previously placed at C:/msys/1.0/home/cvsroot by using the MSYS CVS command was selected and entered in the Location field. The Name field was set to /msys/1.0/home/cvsroot.

After selecting Checkout... in NetBeans, the CVS Root was set to :local:/msys/1.0/home/cvsroot using the Edit... button, setting the Access Method to local and setting the Repository path to /msys/1.0/home/cvsroot (the same name as entered in the Name field in the CVSNT Service control panel Repositories tab). If all goes well, selecting Next> will immediately advance to the Module to Checkout page where the Module, Branch and Location Folder fields can be set. If this doesn't appear immediately, NetBeans either responds with an error, or spins its wheels looking for something (don't bother waiting – it's not going to work).

Several environment variables were set in the attempt to get it work and may or may not be necessary. The environment variables can be accessed by right-clicking on My Computer and selecting Properties, then going to the Advanced tab and selecting Environment Variables. Under System variables the following was entered:

PATH – Edited with C:\Program Files\cvsnt; at the beginning
CVSROOT – New set to C:\msys\1.0\home\cvsroot
CVS_EXE – New set to C:\Program Files\cvsnt\cvs.exe
CVS_RSH – New set to C:\Program Files\cvsnt\cvs.exe

One last warning about using CVSNT on Windows. CVSNT expects text files in the repository to be in DOS format (lines are terminated by CR/LF characters). If the text files are in Unix format (terminated by only LF characters), upon checkout, CVSNT will mess up the files by putting extra CR characters on each line. This is probably due to being a Windows program and expecting that text files to be in the expected Windows (DOS) format.

NetBeans and Debugging – Conclusions

Enough time has now been spent debugging with NetBeans to make some conclusions. After some adjusting, using the GDB debugger under NetBeans appears to be easier then using Insight, the  GUI front-end to GDB. The two are definitely different and initially it appeared that is was more difficult looking inside classes with NetBeans, but this was not the case. NetBeans puts the active local variables for each code line executed in the Variables window automatically and it's very easy to step into them to look at the contents.

With Insight, the variables first need to be added to the Watch window. Insight is never aware of allocated arrays for pointer members – it treats them as simply pointers. If you want to look at the contents of a particular array element, another variable needs to be added to the Watch window. And if too many variables are added to the Watch window, Insight becomes unstable (crashes). NetBeans usually appears to be aware of arrays and lets you descend into which ever element you want to look at.

In Insight there is no way to edit a Watch variable name, say to look at a different array element – it must be deleted and a new one entered (a lot of typing). In NetBeans, the watch variables can be edited (using Customize). Though disconcerting, with the jVi plugin installed, NetBeans requires the use of Vi commands in the New Watch and it starts in command mode – this will take a little getting used to. One last thing about the jVi plugin – sometimes it just stops working (NetBeans returns to its default editor). Restarting NetBeans corrects the problem.

In conclusion, NetBeans nicely integrates everything, therefore, development will be transferred to the NetBeans IDE with CVSNT (see next post on how this was made to work). The next release of the source will include the necessary files to build with NetBeans (development with VIDE2 will now be abandoned).

Translator – Operand Processing Debugging

The Ninth (LET command) test contained multiple equal assignments, so the expected results needed to be updated since the multiple equal assignments feature was removed. Once the expected results were corrected, including removing the comma sub-code that was only needed to tell the difference between multiple equal and comma assignments, this test was successful. There was no necessity to used debugging here, but on to the Tenth (PRINT command) test that is crashing...

The Tenth test was crashing because of an invalid status code was being returned. This was tracked down to the add print code routine that was calling the process final_operand routine incorrectly (passed status variable as an argument instead of setting it to the return value).

Now all the PRINT commands were generating a “done stack not empty” bug error. This was caused by the hidden print codes (PrintDbl, PrintInt and PrintStr) not being popped from the done stack. These should not have been pushed on to the done stack and this was caused because these codes did not have the Print flag set in the table, which would have prevented them from being pushed on to the done stack.

Next the PRINT commands were generating a “invalid use of print function” error at the PRINT command. This was caused during the processing of the last hidden print code on the statement when the top of the command stack was checked for the PRINT command (the stack was empty, so the check was not performed). The PRINT command had been popped by the End-of-line command handler that called the PRINT command handler, which handles the last print code by calling the add print code routine. The add print code calls the process final operand routine, which checked for the print command. The solution was for the End-of-line command handle to leave the command on top of the stack when the command's handler is called and then pop the command off upon returning.

NetBeans and Debugging

Now with the CVS/CR problem resolved (the NetBeans editor will show modifications from the CVS repository as changes are made) and the program builds and runs, it was time to start the debugger.

First off, NetBeans gave a warning that GDB 6.3 (installed with MinGW) was not supported. A later version (7.2) was found at the MingGW Sourceforge page  http://sourceforge.net/projects/mingw/files/MinGW/BaseSystem/GDB/GDB‑7.2/ and the gdb‑7.2‑1‑mingw32‑bin.tar.lzma file was downloaded. This file consisted of three files, gdb.exe, gdbserver.exe and gdb‑python27.exe. The current files at \MinGw\bin were renamed to gdb‑6.3.exe and gdbserver-6.3.exe just in case they were needed and the three new files were copied in. Insight appears to still run, but I'm not sure if it using GDB 6.3 or GDB 7.2. NetBeans now seemed to be happy.

The next thing to figure out was how to specify command line arguments for the program being debugged. With Insight, this was done with the Console window using the “set args” command. The first thing attempted was to enable the GDB console in NetBeans, but there is not need to explain how to do that because the console isn't available until the program is started and by then it is too late for the program to see the arguments set because it had already read them. The solution was discovered under Project Properties in the Run section – there is an Arguments setting.  Now with the arguments set to “‑t 9” for Ninth Translator test, debugging can begin...

NetBeans – Building the Program

Before debugging, the program needed to be built. Within NetBeans, the IBCP project was created and the IBCP source code was checked out of the CVS repository. The .h include files were added in the project under Header files and the .cpp source files were added under Source files. If NetBeans proves satisfactory, the necessary NetBeans files will be included with the source files.

First, some options were set in NetBeans to simply the default directory structure, which assumes a multiple platform product - only Windows is being used here. By default, programs built are put into the directory “dist\Debug\MinGW-Windows” (for the debug configuration, “Release” for the release configuration). This is unnecessary and inconvenient.  To make it build ibcp.exe in the root IBCP directory, the option in Project Properties, Linker, Output was changed to just ibcp from the more involved default ${CND_DISTDIR}/${CND_CONF}/${CND_PLATFORM}/ibcp.

Selecting build project immediately generated errors. The first file was opened and it looked as if it was double spaced (back-slash terminated line continuations caused the errors since the blank line prematurely terminated the line). Somehow in most of the source files, a control-M got added to the end of each line. This is a Unix vs. DOS test file format issue – Unix puts a single newline (control-J or linefeed) at the end of each line and DOS (Windows) puts a CRLF (control-M/control-J) at the end of each line. Vim deals with this invisibly and the GCC compilers ignore it.

I first noticed a week ago that these CRLF were in some of the source files when I did a “cvs diff” command it showed the whole file had changed. Outputting the diff into a file and looking at it with Vim showed the problem was all the lines from one file had “^M” characters at the end of each line, therefore causing no lines to compare. This was causing no problems with either Vim or the compiler.

After some playing around, the problem was discovered to be caused by CVSNT needed by NetBeans (more on this later), which seems to get confused by Unix format files in the repository. The first attempt was to convert all the files to Unix format, but this did not help. After all the files were converted to DOS format, the problem appeared to be solved. Further checkouts were OK. The program then built with no issues. Tests ran as last released. Now on to debugging...

NetBeans and Vim (jVi plugin)

The version of NetBeans installed is the latest 6.9.1 and can be obtained for free at http://netbeans.org/.  The C/C++ pack was installed.  JDK 6 (Java Development Kit) also needs to be installed, but that was already installed on the computer for something else.  The jVi (vi editor clone) plugin can be downloaded at http://sourceforge.net/projects/jvi/.  The downloaded nbvi‑1.3.0.x1.zip file was unzipped, which contains a directory that was put into the Program Files\NetBeans 6.9.1 directory. To install this plugin package, once NetBeans was installed and running, go to Tools/Plugins and the Downloaded tab. Select “Add Plugins...” and find the nbvi-1.3.0.x1 directory and select the two .nbm files. Once both are selected, select the Install button.

Before going into what was needed to make CVS work with NetBeans (which is not necessary for building the IBCP program from the source, but I want to give details on how to do it since it may help someone else in a similar situation as it took a bit of searching to figure it out), I wanted to see what was needed to make debugging work within NetBeans (it works very nice under Linux). This can be done by actually debugging starting with the Ninth Translator (Commands) test...

New Development Environment

I decided to try out a new IDE I recently learned about – NetBeans, which has integration with CVS, an integrated debugger (front end to GDB), and there is an available plugin to get vim editor capability. Getting the Windows version to work properly was going to be a challenge through.

Currently for development, MSYS (Minimum SYStem) with MinGW (Minimum GNU for Windows) is being used with CVS for MSYS installed for version control.  MinGW was updated to version 4.4.0 of the GNU Compiler Suite. For controlling (building) the project, VIDE2 (a simple IDE) is being used, but it contains a very primitive editor (gVim is being used instead for editing). For debugging, Insight, a GUI front end for GDB (version 6.3) is installed. Insight is not exactly stable (crashes a lot), and the latest version 6.8.1 could not be made to work with MSYS.

I'm not sure I will be able to get NetBeans working satisfactory. So far, after a lot of struggle, NetBeans is working with CVS, though not the CVS that was installed with MSYS. Complete details will be in a future post. The IBCP was successful checked out of the CVS repository with NetBeans, the IBCP project was created NetBeans and was built successfully. Next will be to get the debugger working.

Translator – Sub-String Pre-Release

Now the first eight Translator tests succeed. There are more problems to correct with tests 9 through 12 (test 10 crashes). But, since it has been half a year since code has been released, it's time to commit and tag the changes to CVS and make another pre-release.

I was distracted during the last two months with a work project and didn't get much time to work on this project. I made the mistake of not committing the last good working code to CVS (mainly because I wanted to fix just one more bug).  It took almost a week to recover, figure out what I was in the middle of, make the code compile again, and get the most recent changes working (almost). With that accomplished, a pre-release can be made. The file ibcp_0.1.14-pre-2-src.zip has been uploaded at Sourceforge IBCP Project along with the binary for the program. Now on to more debugging to get the rest of the tests working again...

Translator – Sub-String Debugging

To implement the sub-string change, not pushing the sub-string function to the done stack leaving its string operand on the stack (to carry to the next token), is accomplished by using the same mechanism that prevents print function from being pushed on the done stack (set the done push flag to false). With the number of operands being left set to zero, the sub-string function's string operand will be left on the done stack.

As debugging continued, nothing was getting attached to some AssignStr tokens. The assign command handler was only attaching tokens of the String data type, not the TmpStr data type returned (by the concatenate operator and the non-sub-string functions). In these cases, the AssignStr code will be changed to the AssignStrTmp code, but none of the associated codes for TmpStr arguments have been implemented yet (that will come next). Therefore, this will be left as is for now (TmpStr tokens will not be attached to AssignStr).

The new sub-string implementation (not pushing them on the done stack), made sub-string assignments stop working (produced “done stack empty” errors). The variable being assigned was popped from the done stack because it was not needed since it doesn't need to be attached to a token. When the assignment operator (equal) went to look for what was being assigned to determine which assignment code was needed, the done stack was empty. In this case, the sub-string function needs to be on the done stack. Therefore, a check was added to still push the sub-string function token if its reference flag is set.

The sub-string change also affected error reporting - the “expected numeric expression” error for a statement like Z=B+MID$(A$,2,3) was now incorrectly pointing to the A$ instead of the MID$ token (because the A$ carried forward and the MID$ token was not on top of the done stack).

To correct this problem, the code at the end of the find code routine that returns the appropriate error when an operand of the wrong data type was modified. If the last token appended to the output list doesn't match the top token on the done stack and the last token is a sub-string function, then the token returned is set to the sub-string token instead of the top token.

Translator – Sub-String Implementation

Implementation of the transfer of operands of sub-strings will mostly take place in the process final operand function. But first, some information needs to be added to the Table, namely the number of string arguments each code has. This value could be calculated on the fly as needed, but it will make for a simpler design to just put these values in the table. The table initialization code (Table constructor) was modified to calculate these values automatically.

In the process final operand routine, the internal function and operator section, when the number of operands passed in is zero, needs to set the number of operands to the number of string arguments that are on the done stack. These operands will be popped from the done stack and attached to the internal function or operand token.  Only string operands were left on the done stack (other numeric operands were popped by the find code routine as they do not need to be attached to any token).

However, for sub-string functions, which have only one string operand, the string operand will be left on the done stack, to be attached to the next token. If the next token is another sub-string function, then the string operand argument will continue to be carried forward. Therefore, the number of (string) operands to pop from the done stack will only be set to the code's number of string arguments if the code is not a sub-string. For sub-string function, the number of operands will be left set to zero, so nothing will be popped and attached to the sub-string function token.

Translator – Operands of Sub-Strings

One final issue was noticed with the eighth (Sub-String Assignments) Translator test that needed to be corrected, which can be seen with these sample statements (and their current translations):

A$ = LEFT$(B$, 1)  A$<ref> B$ 1 LEFT$([B$] Assign$
A$ = LEFT$(B$+C$, 2)    A$<ref> B$ C$ +$[B$,C$] 2 LEFT$([+$] Assign$

In these examples, remember that the Translator cannot determine if B$ and C$ are variables or functions, therefore these operands are attached to the code they belong to so that the Encoder can adjust the operators and functions accordingly with associated codes with the appropriate arguments. However, sub-string functions add a wrinkle to this scheme.

During run-time, sub-string functions work the same whether their string operand is a reference or a temporary string – they will simply modify the string pointer and length of the string on top of the evaluation stack. Temporary strings are not deleted since the sub-string refers to the temporary string – whichever code has the sub-string function as an operand will take care of deleting the temporary strings. Only be one code for each sub-string function is needed (not two, one with a reference string and one with a temporary string as an operand). The correct translations of the statements above should be:

A$ = LEFT$(B$, 1)  A$<ref> B$ 1 LEFT$( Assign$[B$]
A$ = LEFT$(B$+C$, 2)  A$<ref> B$ C$ +$[B$,C$] 2 LEFT$( Assign$[+$]

In the first statement, the B$ should be attached to the Assign$ token so that the Encoder can change it to an AssignStrTmp code if B$ turns out to be a function (results of functions are  temporary strings) or leave it the AssignStr code if it is a variable. In the second statement, the Translator can change it to an AssignStrTmp code with no attached token, but until temporary strings are fully support, it will attach the +$ to an AssignStr code.

Since the result of a sub-string function is the same as its operand (reference or temporary string), the operands of the sub-string function simply get transferred to the token that has the sub-string function as an operand.