Translator – Errors with a Range of Tokens

Currently when the Translator detects an error, the error code is returned along with the token where the error was detected. The caller uses the column and length values in the token to point to the token. (Later the token will be highlighted in a different color.)

Now it is desired that a range of tokens be highlighted to show an entire sub-expression that contains an error. While it is possible to modify all the functions within the Translator for a second return token, there is an easier solution.

The column and length members of the token are used to output the error – where the token starts and how long it is. The Translator, upon detecting an error with a range of tokens, only needs to modify the length token where the beginning of the error is detected, to include all the tokens. The Translator can do this by using the column of the first token, the column of the last token and the length of the last token. The equation would be:

first->length = last->column – first->column + last->length

So when an error is detected and a range of tokens are involved, the length of the token being returned will be set using the equation above.

Translator – First Operand (Updated Design)

Upon considering the first operand design (see posts starting on July 8, 2010), There still be some confusion with the reported error message and the token at which it is pointing. Consider this simple statement (and the error produced currently):

Z$ = A$ + B$ > C$
             ^-- expected string expression

This does not make sense, because first, the integer expression is valid. The error occurs because this integer expression cannot be assigned to a string variable. And it could be confusing because a string expression cannot be placed where the greater than operator is. Alternatively it could indicate that a string operator is expected at the greater than (later why this was ruled out). The previously proposed first operand design would produce this error:

Z$ = A$ + B$ > C$
     ^^-- expected string expression

The error is now pointing to the beginning of the integer expression. But, one could say wait a minute, A$+B$ is a string expression, why is this an error? A better solution would produce this error:

Z$ = A$ + B$ > C$
     ^^^^^^^^^^^^-- expected string expression

This is much better as it highlights the entire expression, an integer expression, and says that a string expression is expected here. This can be accomplished by, in addition to keeping track of the first operand (the A$), also keeping track of the last operand (the C$). The error would point to all tokens between the first and the last of the invalid expression. Now, how can these two tokens be passed back where currently only one token is being returned when an error occurs? Next, the solution to this...

Translator – Expression Type Pre-Release

There are two main problems with the Twelfth (more error statements) test, namely the handling of the first operand of operators first mentioned on July 8, 2010, and recognizing tokens with parentheses before assignment operators as arrays only (implying the operands are numeric subscripts).

All of the other tests are succeeding, so it's probably a good idea to kick another pre-release first. This is the first release since development was switched to NetBeans, so there are some changes in the released files. NetBeans is not required for building the project. There is now full Makefile for building the program. NetBeans was used to initially generate this file, but it was modified to simplify it. The VIDE2 project file ibcp.vpj will no longer be part of the source files.

Also no longer part of the source files is codes.txt and test_codes.h. Both of these files are now be generated by the new Makefile using the codes.awk and test_codes.awk awk scripts. The MinGW and MSYS packages both need to be installed in order to use the Makefile. The program build has only been tested with GCC 4.4.0 version for MinGW. Also, the awk program is needed for the awk scripts (I don't remember if this is part of MSYS or whether it needs to be installed separately).

The Makefile is currently configured for building the release program with no symbols or debugging information. Debugging can be enabled by changing BASICOPTS from "-O -s" to "-g" in the Makefile. The file ibcp_0.1.14-pre-3-src.zip has been uploaded at Sourceforge IBCP Project along with the binary for the program. Now on to implementing the first operand design...

Translator – More Expression Type Debugging

The next problem was occurring when there was an unexpected operand when an operator, comma or closing parentheses was expected (for example: Z$=MID$(A$,B C). This was only occurring for functions with multiple arguments (ASC, MID$ and INSTR) when at the last argument of the shorter form. The “expected operator or closing parentheses” error was produced, but a comma could follow for the optional argument.

A checked was added to the parentheses status routine when at the last argument. If the function has the Multiple flag set, indicating there is another form with more arguments, then the “expected operator, comma or closing parentheses” error is produced instead.

One problem remained, the statement Z$=MID$(A$,B, produced an “expected string expression” at the end of the line where it should have been an “expected numeric expression” error. The problem also occurred on the last optional argument of the INSTR and ASC functions. The problem was caused because the top of the count stack contained the index to the first form of these functions instead of to the second form with the optional argument.

When the code went to get the argument's data type, it ran of the end off the operand data type array incorrectly picking up a string data type. The coding error was on the line that incremented the index of the token on the hold stack to the next form of the function and assigned it to the index on top of the count stack. A post-increment C operator was used instead of the pre-increment operator and the index on the count stack was not actually changed.

Now all the test statements in the Eleventh Translator test are working (and on expected error needed to be corrected on the Third test because of the last fixed – it was missed previously). On to the Twelfth (more error statements) test...

Translator – More Expression Type Debugging

Now that the data type was moved back from the expression information structure back to the main table entry structure, so that it is available for commands as well as operators and functions, and the code was working again the way it was, it was back to debugging the Eleventh (error tests) Translator test (after committing the working code to CVS) to continue fine tuning the error messages produced.

The next problem to tackle was the generic “expected expression” error being produced when a binary operator occurred in an expression when only a unary operator was expected. This error needed to be changed to the more specific “expected <numeric/string> expression” errors. This was accomplish by calling the new get expression data type routine to get the current data type and producing the appropriate error.

Next there were several error test statements that contained an unexpected comma in an expression, which were producing “expected operator or end-of-statement” errors. This was the correct for some statements (for example: Z=A+B, and Z=A,B=1 and Z=A,), but, not for others (for example: Z=, and Z,Y=, and Z$=,) where the correct error should have been “expected <numeric/string> expression” errors.

These errors are being handled in the comma token handler. These conditions can be determined by looking at the done stack. If the done stack is not empty, the expression up to the comma is valid, so expecting another operator or end-of-statement is appropriate. If the done stack is empty, then there is no valid expression yet for the assignment, so expecting an expression is appropriate. Again, to determine the type of expression, the new get expression data type routine is used.

Translator – Command Expression Type

There was a problem getting the data type for the token on top of the command stack. There is no issue with assignment tokens as they contain a data type (they are just assignment operators put on the command stack).

A while back, a new field was added to the Table entries for the expression type with possible values of None, Numeric, String and Any to be used with command entries. The idea at the time was that this would be used for the expression type checking (for the generation of the correct error messages). However, this field was not being used (the original expression type code added at the time was abandoned because it had too many problems – it is currently commented).

This first attempt at the current problem was to use this expression type and convert it into a data type (None/Any to None, Numeric to Double, and String to String). The only place this was used was in the new get expression data type routine for commands. But this wasn't working for assignment commands (which were operators and their entries didn't have the expression type set – the data type in the expression information structure had the return value).

So it was concluded that the expression type field should be removed and the data type moved back from the expression information structure into the main entry structure (where it was originally). When the Parser finds and creates a command token, it sets the data type of the token to data type from the table entry. Currently for all tokens, the Parser was getting the data type from the expression information structure or none if there was no structure (as is the case for commands).

Translator – Expression Type Implementation

A new routine was implemented to handle getting the current data type of the expression processed so far. This routine is called when the unexpected end of expression occurs to get the data type so that the correct “expected ...” error can be reported, and is also called before pushing an opening parentheses token to the hold stack to set its data type (so that it can be carried through the rest of the expression).

Once the routine was working, it was simplified. First, the token on top of the hold stack is checked for an operator before looking for the other conditions (parentheses, internal functions, and tokens with parentheses). This reduced the number of tests overall. An empty hold stack causes a bug error since at least the Null token should be on the stack.

If the top hold stack token is the Null token, then the data type is obtained from the token on top of the command stack. This caused another problem with commands (more on this in the next post). If the top token is an opening parentheses, then its data type is used (which could be none). Otherwise the data type of the operator's operand is used (first operand for a unary operator, second operand for binary operator).

If the top token on the hold stack is not an operator, then the top of the count stack is used to determine where to get the data type. If the count stack is empty, a bug error is returned since this situation should not happen. If the top is an internal function, then the data type of the current operand of the function is used. If the top is a parentheses, a bug error is returned since this situation should also not happen (parentheses token should have been on top of the hold stack). Otherwise the top contains a token with parentheses (an array or non-internal function) where the data type cannot be determined, so is set to None.

Translator - Expression Type With Parentheses

Determining the expression type to report the appropriate error within a parenthetical expression without another operator on the hold stack involves a little more work. To accomplish this, the expression type needs to carried from before the opening parentheses.

To prevent having to scan backwards into the expression, when the opening parentheses is pushed on to the hold stack, the opening parentheses token will be assigned a data type, if possible, otherwise the data type will be left at the default data type of none (which will cause the generic “expected expression” error if an error occurs).

Here are several possibility of tokens that will be on the hold stack when an opening parentheses token is about to be pushed onto the hold stack and how to determine the data type to put into the parentheses token:

Operator – get the expected operand data type of operator
Internal function – get the expected argument operand data type of function
Token with parentheses – can not determine data type
Opening parentheses – inherit this token's data type
Null – Look at data type for item on top of command stack

The first four possibilities are the same as before, so a common function will probably be implemented to handle determining the data type for the opening parentheses token. The last possibilities will require looking for the expected data type for the command on the command stack. Here are some examples of commands and expected data types for expressions:

LET A = (  - assign token on command stack will have data type of variable
PRINT (  - PRINT can take any type of expression, therefore none
INPUT PROMT (  - INPUT PROMPT expects string expression
IF (  - IF expects numerical expression

The IF statement has not been defined as of yet, but most likely will expect an integer expression (the result of all logical operators). However, a double expression will be accepted either with a different IF code expecting a double expression or the more likely, a hidden integer conversion code will be inserted. The IF will be defined later. Now on to the implementation of this...

Translator - Expression Type Determination

In the Eleventh (error statements) tests, the are many statements for testing the correct error message in a line that contains something unexpected, like either an unexpected comma or end-of-line. One of these was just corrected (see last post) – the situation when the Translator is not in a parenthetical expression including inside array subscripts or function arguments. This was detected when the count stack is empty.

The situation when in an internal function has also been handled when the operand processing was implemented. The code simply checks for the expected operand type for the current operand. An internal function is detected when the number of expected operands for the item on top of the count stack is non-zero. The current operand number is in the number of operands in top item on the count stack.

For the remaining situation, the code was simply returning an “expected expression” error, since it was assumed that the type of expression could not be determined. Turns out, this is not entirely true. Two sub-conditions to this situation are inside a parentheses and inside an array or non-internal function (a token with a parentheses). Each of these can be further sub-divided into whether this is an operator or not. Here are examples of each of these four conditions (terminated with an unexpected end-of-line):

Z = (  in parentheses, no operator
Z = (A+  in Parentheses, with operator
Z = A(  in token with parentheses, no operator
Z = A(B+  in token with parentheses, with operator

In the third case, the expected expression type can't be determined because the Translator does not know if A is an array (expression must be numeric for a subscript) or a function (expression can be any type), therefore the “expected expression” error is appropriate. In the other cases, the expression type can be determined and in these examples, this would mean an “expected numeric expression” error.

For the two cases with an operator, the data type of the operand or the operator on top of the hold stack can be used to determine the expected type of expression to follow (just like for a non-parenthetical expression just corrected). For the first case, with just a parentheses, determining the expression type is a little more involved...

Translator – More Operand Processing Debugging

Continuing with debugging, the next difference was a PRINT command that contained sub-string functions where the expected results needed to be updated (sub-string functions don't get attached tokens, tokens are attached to next token).

Finally in the Tenth (PRINT statements) test, there were three “debug #1” bug errors, which should have been an “expected numeric expression” errors. These test lines were incomplete statements (e.g. PRINT A+). This bug error was put in to fix later because the existing code was not correct. The code was modified to return the expected type error for the operand of the operand token on top of the hold stack (taking into account whether it is the first operand of a unary operator or second operand for a binary operator). It was previously set to return the variable type for the command on top of the command stack (which was wrong – the PRINT command has no data type).

Now on the the Eleventh (error statements) tests, which has quite of few incorrect error messages...

Windows, NetBeans and CVS (CVSNT)

This post is not strictly related to the IBCP project, but will be helpful explaining how NetBeans (6.9.1) on Windows (XP) was made to work with CVS since it took nearly a day or searching and experimenting to accomplish (and many others were having similar problems).

The initial hope was that NetBeans for Windows would play nice with the CVS installed with MSYS (Minimum SYStem), but it did not. Searching indicated that NetBeans requires a CVS server even for a local repository and apparently, the CVS that comes with MSYS is not sufficient, though when doing a cvs ‑v command, it does respond … 1.11 (client/server). Further searching uncovered The CvsGui project and the WinCVS 2.1.1.1 package (includes WinCVS and CVSNT) was downloaded from http://sourceforge.net/projects/cvsgui/files/. Initially only the CVSNT package was installed (and this may have been sufficient).

Not realizing that there was a Service control panel under the Start menu (under CVSNT) that is used to configure CVSNT, the WinCVS package was installed. Normally WinCVS will also install the required CVSNT (the CVS server), but can be skipped if CVSNT is already installed. WinCVS is a very nice GUI for looking at checked out software (including nice version and branch graphs), however, it does not contain any configuration of CVSNT. And exactly how to check out a module from the repository was not discovered. This was not important as either the command line or NetBeans can be used to check out a module.

Once CVSNT is installed (indicates version 2.0.51d), it requests that Windows be restarted. This is necessary to install and start the CVS services, which occurred automatically upon reboot. Next the CVS server needs to be configured by using the Service control panel. The first tab reported that the services (CVS Service and CVS Lock service) were both running (a good sign).

The CVS repository needs to be identified using the Repositories tab by selecting Add. Using the browse button on the next dialog (the ... button), the repository that had been previously placed at C:/msys/1.0/home/cvsroot by using the MSYS CVS command was selected and entered in the Location field. The Name field was set to /msys/1.0/home/cvsroot.

After selecting Checkout... in NetBeans, the CVS Root was set to :local:/msys/1.0/home/cvsroot using the Edit... button, setting the Access Method to local and setting the Repository path to /msys/1.0/home/cvsroot (the same name as entered in the Name field in the CVSNT Service control panel Repositories tab). If all goes well, selecting Next> will immediately advance to the Module to Checkout page where the Module, Branch and Location Folder fields can be set. If this doesn't appear immediately, NetBeans either responds with an error, or spins its wheels looking for something (don't bother waiting – it's not going to work).

Several environment variables were set in the attempt to get it work and may or may not be necessary. The environment variables can be accessed by right-clicking on My Computer and selecting Properties, then going to the Advanced tab and selecting Environment Variables. Under System variables the following was entered:

PATH – Edited with C:\Program Files\cvsnt; at the beginning
CVSROOT – New set to C:\msys\1.0\home\cvsroot
CVS_EXE – New set to C:\Program Files\cvsnt\cvs.exe
CVS_RSH – New set to C:\Program Files\cvsnt\cvs.exe

One last warning about using CVSNT on Windows. CVSNT expects text files in the repository to be in DOS format (lines are terminated by CR/LF characters). If the text files are in Unix format (terminated by only LF characters), upon checkout, CVSNT will mess up the files by putting extra CR characters on each line. This is probably due to being a Windows program and expecting that text files to be in the expected Windows (DOS) format.

NetBeans and Debugging – Conclusions

Enough time has now been spent debugging with NetBeans to make some conclusions. After some adjusting, using the GDB debugger under NetBeans appears to be easier then using Insight, the  GUI front-end to GDB. The two are definitely different and initially it appeared that is was more difficult looking inside classes with NetBeans, but this was not the case. NetBeans puts the active local variables for each code line executed in the Variables window automatically and it's very easy to step into them to look at the contents.

With Insight, the variables first need to be added to the Watch window. Insight is never aware of allocated arrays for pointer members – it treats them as simply pointers. If you want to look at the contents of a particular array element, another variable needs to be added to the Watch window. And if too many variables are added to the Watch window, Insight becomes unstable (crashes). NetBeans usually appears to be aware of arrays and lets you descend into which ever element you want to look at.

In Insight there is no way to edit a Watch variable name, say to look at a different array element – it must be deleted and a new one entered (a lot of typing). In NetBeans, the watch variables can be edited (using Customize). Though disconcerting, with the jVi plugin installed, NetBeans requires the use of Vi commands in the New Watch and it starts in command mode – this will take a little getting used to. One last thing about the jVi plugin – sometimes it just stops working (NetBeans returns to its default editor). Restarting NetBeans corrects the problem.

In conclusion, NetBeans nicely integrates everything, therefore, development will be transferred to the NetBeans IDE with CVSNT (see next post on how this was made to work). The next release of the source will include the necessary files to build with NetBeans (development with VIDE2 will now be abandoned).

Translator – Operand Processing Debugging

The Ninth (LET command) test contained multiple equal assignments, so the expected results needed to be updated since the multiple equal assignments feature was removed. Once the expected results were corrected, including removing the comma sub-code that was only needed to tell the difference between multiple equal and comma assignments, this test was successful. There was no necessity to used debugging here, but on to the Tenth (PRINT command) test that is crashing...

The Tenth test was crashing because of an invalid status code was being returned. This was tracked down to the add print code routine that was calling the process final_operand routine incorrectly (passed status variable as an argument instead of setting it to the return value).

Now all the PRINT commands were generating a “done stack not empty” bug error. This was caused by the hidden print codes (PrintDbl, PrintInt and PrintStr) not being popped from the done stack. These should not have been pushed on to the done stack and this was caused because these codes did not have the Print flag set in the table, which would have prevented them from being pushed on to the done stack.

Next the PRINT commands were generating a “invalid use of print function” error at the PRINT command. This was caused during the processing of the last hidden print code on the statement when the top of the command stack was checked for the PRINT command (the stack was empty, so the check was not performed). The PRINT command had been popped by the End-of-line command handler that called the PRINT command handler, which handles the last print code by calling the add print code routine. The add print code calls the process final operand routine, which checked for the print command. The solution was for the End-of-line command handle to leave the command on top of the stack when the command's handler is called and then pop the command off upon returning.

NetBeans and Debugging

Now with the CVS/CR problem resolved (the NetBeans editor will show modifications from the CVS repository as changes are made) and the program builds and runs, it was time to start the debugger.

First off, NetBeans gave a warning that GDB 6.3 (installed with MinGW) was not supported. A later version (7.2) was found at the MingGW Sourceforge page  http://sourceforge.net/projects/mingw/files/MinGW/BaseSystem/GDB/GDB‑7.2/ and the gdb‑7.2‑1‑mingw32‑bin.tar.lzma file was downloaded. This file consisted of three files, gdb.exe, gdbserver.exe and gdb‑python27.exe. The current files at \MinGw\bin were renamed to gdb‑6.3.exe and gdbserver-6.3.exe just in case they were needed and the three new files were copied in. Insight appears to still run, but I'm not sure if it using GDB 6.3 or GDB 7.2. NetBeans now seemed to be happy.

The next thing to figure out was how to specify command line arguments for the program being debugged. With Insight, this was done with the Console window using the “set args” command. The first thing attempted was to enable the GDB console in NetBeans, but there is not need to explain how to do that because the console isn't available until the program is started and by then it is too late for the program to see the arguments set because it had already read them. The solution was discovered under Project Properties in the Run section – there is an Arguments setting.  Now with the arguments set to “‑t 9” for Ninth Translator test, debugging can begin...

NetBeans – Building the Program

Before debugging, the program needed to be built. Within NetBeans, the IBCP project was created and the IBCP source code was checked out of the CVS repository. The .h include files were added in the project under Header files and the .cpp source files were added under Source files. If NetBeans proves satisfactory, the necessary NetBeans files will be included with the source files.

First, some options were set in NetBeans to simply the default directory structure, which assumes a multiple platform product - only Windows is being used here. By default, programs built are put into the directory “dist\Debug\MinGW-Windows” (for the debug configuration, “Release” for the release configuration). This is unnecessary and inconvenient.  To make it build ibcp.exe in the root IBCP directory, the option in Project Properties, Linker, Output was changed to just ibcp from the more involved default ${CND_DISTDIR}/${CND_CONF}/${CND_PLATFORM}/ibcp.

Selecting build project immediately generated errors. The first file was opened and it looked as if it was double spaced (back-slash terminated line continuations caused the errors since the blank line prematurely terminated the line). Somehow in most of the source files, a control-M got added to the end of each line. This is a Unix vs. DOS test file format issue – Unix puts a single newline (control-J or linefeed) at the end of each line and DOS (Windows) puts a CRLF (control-M/control-J) at the end of each line. Vim deals with this invisibly and the GCC compilers ignore it.

I first noticed a week ago that these CRLF were in some of the source files when I did a “cvs diff” command it showed the whole file had changed. Outputting the diff into a file and looking at it with Vim showed the problem was all the lines from one file had “^M” characters at the end of each line, therefore causing no lines to compare. This was causing no problems with either Vim or the compiler.

After some playing around, the problem was discovered to be caused by CVSNT needed by NetBeans (more on this later), which seems to get confused by Unix format files in the repository. The first attempt was to convert all the files to Unix format, but this did not help. After all the files were converted to DOS format, the problem appeared to be solved. Further checkouts were OK. The program then built with no issues. Tests ran as last released. Now on to debugging...

NetBeans and Vim (jVi plugin)

The version of NetBeans installed is the latest 6.9.1 and can be obtained for free at http://netbeans.org/.  The C/C++ pack was installed.  JDK 6 (Java Development Kit) also needs to be installed, but that was already installed on the computer for something else.  The jVi (vi editor clone) plugin can be downloaded at http://sourceforge.net/projects/jvi/.  The downloaded nbvi‑1.3.0.x1.zip file was unzipped, which contains a directory that was put into the Program Files\NetBeans 6.9.1 directory. To install this plugin package, once NetBeans was installed and running, go to Tools/Plugins and the Downloaded tab. Select “Add Plugins...” and find the nbvi-1.3.0.x1 directory and select the two .nbm files. Once both are selected, select the Install button.

Before going into what was needed to make CVS work with NetBeans (which is not necessary for building the IBCP program from the source, but I want to give details on how to do it since it may help someone else in a similar situation as it took a bit of searching to figure it out), I wanted to see what was needed to make debugging work within NetBeans (it works very nice under Linux). This can be done by actually debugging starting with the Ninth Translator (Commands) test...