Upgrade to NetBeans 7.0

Netbeans 7.0 was recently released (version 6.9.1 was being used for development on Windows), which contains support for git via a plug-in. Getting that to work took some research and experimentation. It appeared that the git plug-in only allowed creating a new git repository, not opening an existing one. However, it was discovered that if a new project is created from existing source and standalone make file in a directory with a git repository, NetBeans will then see the repository.

There was a problem building the project. The compiler complained that it didn't recognize the -static-libstdc++ option. NetBeans was still executing GCC 4.4.5, even though it was correctly pointing to the GCC 4.6.0 directory. Setting the path to include GCC 4.6.0 before executing NetBeans from the command line did not help.

Next an attempt was made to run the debugger on the project. There was no debug option in the standalone make file. Previously, the project was set up under NetBeans and it generated its own complex and convoluted set of files for generating what it needed to build the project. Both a Release and a Debug configuration was created by NetBeans. To release the project, NetBeans was used to create a standalone make file instead of releasing all the NetBeans files, which would have required NetBeans to build.

This time around, NetBeans was given this (heavily modified) make file when the project was created, though there was no Debug configuration. The make file was temporarily modified to turn on debug compiling. The make system needs to be reworked to allow both a release and debug configuration without using the system built into NetBeans.

Linker Problem on Linux

Now that the program builds and runs on Windows, it was time to make sure it still worked on Linux. However, the linker failed because the compiler/linker did not understand that ‑static‑libstdc++ option. Upon research, it was discovered that this option was only in GCC version 4.5 and later, so it was not available in the GCC 4.4.5 on Kubuntu.

The latest version of GCC (4.6.0) was downloaded and installed on Kubuntu using the source code. This required quite a bit of work to accomplish (for complete details, hit the continue link). Now that the project was building on both Linux and Windows, it was time to introduce it to the NetBeans IDE (which was now at version 7.0, version 6.9.1 was being used for development on Windows).

Running on Windows

There were no problems building the modified project back on Windows, however, when running the regression test script on Windows, all tests failed. This occurred because the expected test results files were in Unix (LF only) format, but the freshly generated output files were in DOS (CRLF) format.  The files otherwise appeared to be identical.

To verify that the output was the same, the test script was modified where the cmp ‑s command (‑s for silent) was replaced with a diff ‑bq command (‑b for ignore white-space, ‑q for quiet) to have it ignore the differences between CRLF and LF, both considered white-space. Now all tests compared to the expected results. A better solution will need to be found to deal with the differences between Windows and Linux.

One other problem was found when trying to run on Windows, which was discovered when trying to make the program run. Upon the first run, an error was reported that the application failed to start because libstdc++‑6.dll was not found. Once the environment variables were adjusted to include the MinGW binary path, the program ran fine.

This was the same problem as before with libgcc_s_dw2‑1.dll. This problem was resolved by adding a -static-libgcc option to the linker step, which links in the libgcc.a library. It turns out that there is similar option ‑static‑libstdc++ for linking in the libstdc++.a library. The program no ran without requiring any external library. This was tested by moving the program to another computer that did not have MinGW installed.

Setting up and Building on Windows

Now that the code was building and working on Linux, it was time to setup the Windows XP virtual machine for building.  On the previously Windows XP installation, MinGW (with GCC 3.4.6) and MSYS (1.0.11) were installed with a number of packages installed later (like GCC 4.4.0). It was a very hacked together setup in attempt to make things work right.

This time around, the Automated MinGW Installer was used. The current 2011-05-30 version was used, which contains the latest versions of MSYS, MinGW and all the other utilities.  The latest version can be obtained here. When installing this package (which actually downloads all the latest versions before installation), both the MSYS and MinGW options were selected. The version of GCC installed was 4.5.2.

Moving the project software back over to Windows, the build succeeded. Turns out the awk scripts still work in Windows. There was also no warning about the gets() function. This may be because the version implemented with MinGW contains more features, which were mentioned back on June 15, 2010 and are not implemented in the Linux version.

Integers on 64-bit Linux

After a successful compile, the regression test script aborted with errors. The errors occurred because it was trying to execute ibcp but the make file was building ibcp.exe. This worked on Windows because it will execute ibcp.exe when the command is ibcp. The script was modified to run ibcp.exe instead (this issue will be resolved shortly). Now only two of the parser tests and one of the translator test failed.

The first parser test (immediate commands) was not working on the inputs containing large values for line numbers or increments that were suppose to be reported as an error, but now were being accepted and the value output was not the same as the input. A problem in the third parser test (numbers) was a negative integer one beyond the limit of an integer, was now also being accepted as an integer (it should have become a double since it was out of range for an integer). And a problem on the 17th translator test (negative values) was another negative number out of range of an integer, was also being accepted.

These problems were caused by the strtol() function calls used to convert strings to an integer. An overflow was expected for these large numbers. This function returns a long integer and on 32-bit Windows (or any 32-bit OS) this is the same as integer values, which is 32 bits, so an overflow was reported. However, on 64-bit Linux (or any 64-bit OS), long integers are 64 bits, so these large values were now being accepted, but when converting to an integer, were being truncated from 64 bits to 32 bits.

To correct this problem, in additional to the ERANGE (overflow) error occurred condition, a condition was added to check if beyond the maximum integer using the predefined constant INT_MAX (and INT_MIN when the value could also be negative).

The remaining problem in the third parser test was that only two digits were output for floating point exponents on Linux if a third (the hundreds) digit was not required. On Windows, three digits are always output for the exponent. This must be due to slightly different builds or implementation of the standard C library functions. How to deal with this problem for testing will be dealt with a little later.

Resolving 64-bit Linux Build Problems

There were problems with the awk scripts - they were not generating the auto-generated includes files. This was caused by the RS (Record Separator) and ORS (Output Record Separator) variables being set to CRLF to work with Windows. The files were now all in Unix (LF) format and so the input source files not being read correctly. So the RS and ORS lines were commented.

The git configuration option autocrlf was set to false, causing git upon checkout to change any DOS (CRLF) format files in the repository to be converted to Unix format (LF), or the CRLF was removed when the CVS repository was converted to git. There were previous problems with CRLF on Windows when switching between cvs with MSYS to CVSNT (all the files were changed to DOS format since CVSNT didn't seem to like Unix format files).

Next there were warnings about the gets() function calls in the test routines, which were not present on Windows (using GCC 4.4.0 on MinGW/MSYS). After updating with the package manager, Kubuntu had GCC 4.4.5, so perhaps this was a change between 4.4.0 and 4.4.5 or a difference between the GCC build on MinGW vs. Kubuntu. The warning complained that the returned value from gets() was being ignored. To remove the warnings, statements were add to check the return value.

The linker also warned that the gets() function is dangerous and should not be used. This is probably because gets() does not check for a buffer overflow of the allocated buffer given to it. The buffer allocated is plenty big for testing here and this is just the test code anyway, so this warning will be ignored from the time being.

Building on 64-bit Linux

Before attempting the build the project on Linux, the version control repository of the project code was converted from the Concurrent Versioning System (CVS) to git, a more modern version control system. The version installed using Kubuntu's software package manager was 1.7.4.4. However, this version did not have the needed cvs import utility, so the latest version was installed directly from the git repository, though git is required to retrieve it. This was version 1.7.6. Using git will have no impact on this project's software releases except that the CVS versions tags will be removed from the source files.

The intention was to use command line make file distributed with the project to build on Linux. Previously on Windows, NetBeans was used to build the project, though the command line make file (which was initialized generated by an option in NetBeans) was used to test the project before a release. For the moment, NetBeans would not be used (more about this later). Some minor problems needed to be resolved before it would build and work on Linux...

Development Environment Change

Because of a job requirement, I had to install 64-bit Linux on my computer (I chose Kubuntu 10.10, Maverick Meerkat). So development of this project will be moved over to 64-bit Linux, at least for the time being. Since so far, the program only has a simple command line interface (no GUI), this won't cause an immediate problem. The releases will still be tested under Windows XP as I have a Windows XP virtual machine running on Kubuntu. Perhaps when the time comes to start on the GUI, a common tool set will chosen, something like Qt.

For the moment, this will also give the opportunity to present how to set up Windows to be able to build the project since this virtual machine is a new Windows installation and not a clone of my of my previous Windows installation. A lot of things were originally tried and listing the procedure that finally worked in the end was probably impossible.

Also possibly changing will be the actual development tools that will be used. Under consideration is changing the version control system (from cvs to git), the IDE (from NetBeans 6.9.1 to newer version 7.0, which has a plug-in for git, or possibly to the Eclipse CDT IDE), and the build system to build the project (from NetBeans' make system and standalone make file currently distributed with the project to the cmake build system). Each of these will be explained in the posts that follow...

Translator – Colon (Design)

The colon token will work very similar to the end-of-line token and so will have the end statement flag in its table entry. The difference with the end-of-line token is that instead of terminating the translation or a line, the token mode will be set to command for the next statement.

There will be two error conditions. Two colons will not be allowed. While a second colon would not affect anything and is allowed in other BASICs, it will be considered an error here. There is no point to allowing this. Also, a colon will not be allowed at the end of the line with no command after it.

When a colon token is received, the colon sub-code will be set in the command token on top of the command stack. The command token will be appended to the end of the statement when the command has been processed (by the command handler).

However, there is an issue with print statements. The actual print command token may not be appended to the output if there is a semicolon, comma or print function at the end of the print statement. In this case, the print command handler will have to transfer the colon sub-code to the last print code that was appended to the output, which may be a print type (when a semicolon is not needed), comma, semicolon, SPC or TAB.

Colon – Statement Separator

The Colon is used to separate statements in the BASIC language though it is not part of the ANSI BASIC, it is part of the more common BASICs (GW-Basic, QBasic, FreeBasic, etc.). Before moving to the translation of colon tokens, like everything else, the action during run-time must be defined.  However, the Colon does not actually do anything at run-time.

Colons don't actually need to be stored as separately in the program, however, for them to be reproduced, something needs to be put into the internal program. The assumption that there will be a colon at the end of each statement except at the end of the line is not sufficient. Consider this statement:

IF A>B THEN PRINT A ELSE PRINT B:A=B:B=0.0

There is no colon after the first print statement. To reproduce colons properly, there will be a colon sub-code set for a command token that has a colon following the statement. For the statements after the ELSE in the example above, the colon sub-code will be set as shown in this translation:

B PrintDbl'Colon' A<ref> B Assign'Colon' B<ref> 0.0 Assign

When a line is reproduced, the Recreator will add a colon after a statement that has the colon sub-code set.

Translator – A Unary Operator Curiosity

One of the test statements created for testing the unary operator fix was:

A = -B^NOT C% + -D*NOT E%

The intention of this statement was test the NOT unary operator in front of the second operand of both the exponentiation and multiplication operators. The translation of this statement was expected to be (the blue expression being the first operand and the red expression being the second operand of the addition):

A<ref> B C% NOT ^* Neg D Neg E% NOT *%2 + Assign

However, what was produced was this unexpected translation:

A<ref> B C% D Neg E% NOT *%2 +%1 Cvtint NOT ^* Neg Assign

Upon reviewing the precedence of the operators (see Translator – Operator Precedence) and the code, it turns out that this translation was correct. The ADD is higher precedence than the NOT, so the operands of the ADD are C% and the –D*NOT E% expression with MUL (*%2) higher precedence than ADD, its operands are the –D and NOT E%).

So while the exponentiation is highest precedence, with NOT having a low precedence allowed the ADD to bind the rest of the expression to the NOT, which becomes the second operand (yellow above) of the exponentiation, with the first negation being the final operator. In C/C++, the not (!) and negation (-) operators are very high precedence (and there is no exponentiation operator). But here, NOT was given a low precedence just above the other logical operators but below the math operators (see Translator – Operator Precedence for reasoning). Normally the NOT operator would probably not be used in the same expression as exponentiation like above.

Translator – Unary Operator Problem

While testing the negative constant changes, a new problem was discovered with unary operators, specifically this statement:

A = ---B

Which produced a “done stack empty” bug error at the first negation token. The problem occurred because the second negation operator forced the first negation operator from the hold stack because it was greater or equal precedence, and when checking the operand of the first negation operator, there was nothing on the done stack. Here and some additional examples:

A = B*-C
A = B^-C

The first statement translated correctly because negation is higher precedence than multiplication leaving multiplication on the hold stack. However, the second statement failed because negation is lower precedence than exponentiation forcing exponentiation from the hold stack but with only one operand on the done stack generating the done stack empty bug error. A new rule was needed for unary operators.

Basically, unary operators should not force any tokens (unary operators, binary operators, arrays, or functions) from the hold stack regardless of their precedence because not all of their operands have been received yet (the negate and its operand will be their operand and it has not been fully received yet). As currently implemented, other non-unary operators should still force unary operators from the done stack if the unary operator has higher precedence.

The check to force tokens from the hold stack was changed to if the precedence of the operator on the hold stack is higher than the current operator and if the current token is not a unary operator. Unary operators will now not force other tokens from the hold stack, but other tokens will still force unary operators from the hold stack if higher in precedence. While testing this change, a curious result was produced from one of the test statements...

Parser – Negative Constants

Negative constants were previously not considered by the Parser, which interpreted a minus as the subtract operator. The Translator then changed it to a negate operator when it appeared in the operand state. Consider these two examples (along with there current translations):

A = B-1.5      A B 1.5 Sub Assign
A = -1.5+B     A 1.5 Neg B Sub Assign

The reason for the Parser to not look for signs on numerical constants can be seen in the first example. If the Parser produced the four tokens A = B -1.5, the Translator would generate an “expected operator” error at the -1.5 token since a second operand token was received when it was expecting a binary operator. The second example produces an unnecessary negate token after the constant. While perfectly valid, this is not desirable.

In order for the Parser to correctly interpret negative signs on numerical constants, it needs to be aware of whether the Translator is in operand state or not. If in operand state, the Parser can look for a negative sign in front of a number constant, otherwise a minus should be interpreted as an operator.

A new operand state flag was added to the Parser with an access function to set its value (which is initialized to off). The Parser get number routine was modified to have a new sign flag used to determine if a negative sign was found first. This flag will also prevent multiple negative signs. However, it will only check for a negative sign if no digits or a decimal was seen and if the new operand state flag is on.

An access function was added to the Translator to get the current operand state (either operand or operand-or-end state). Before calling the Parser get token routine, the Parser operand state is set from the Translator's current operand state. While testing, a problem was discovered with unary operators...

Parser – Tokens With Parentheses

While correcting issues with define function tokens, it was noticed that it is not necessary to also store the opening parentheses in the string field of the token. This also includes the generic tokens with parentheses. The parentheses is not necessary because there are separate token types to identify tokens with and without parentheses.

It is also advantageous to not store the parentheses so that an array or define function name can be found in the dictionary. For define functions, take the code snippet:

DEF FNHypot(X,Y)
    FNHypot=SQR(X*X+Y*Y)
END DEF
Z = FNHypot(3,4)

Both the FNHypot( and FNHypot tokens appear, which represent the same function. If the parentheses was stored in the dictionary for the function name, it would require complicated string comparisons to figure out that the FNHypot token is the same function. This same issue applies to regular function names.

A similar issue also applies to arrays. When functions and subroutines are implemented, there will be a feature to allow an entire array to be passed to a function or subroutine. Exactly how this will work has not been defined yet, but this code snippet shows how it would look:

DIM Array(10)
CALL subroutine(A)

The changes for removing the parentheses from these tokens were very simple. In the Parser get identifier routine, when creating the string for these tokens, the length provided to the string constructor was changed to be one less than the actual length so the parentheses would not be included. In the Translator when reporting an error at the open parentheses of a define function with parentheses token, the minus one was removed since the length is now one less. Finally, in the token test output routines, an open parentheses was added to the output of these tokens.

Project – Tokens Status Enumeration

Each time a new error (token status) is added, renamed or deleted, two changes were needed. Both the token status enumeration (include file) and the message array (source file) needed to be changed. The correct changes were needed or the two files will be out of sync. To check for problems, code had been added to initialization to check for duplicates and missing entries.

Similar to automatic generation of the code enumeration from the table entries, the token status enumeration will also be generated automatically. Each message array element was a structure containing a token status value and a pointer to a message string. The elements were changed to just the message string with the name of the token status in a comment at the end of the line.

The codes awk script was renamed to enums and was modified to also read the token status message array to generate the token status enumeration automatically be reading the name in the comment after the message string. The name of the output file was changed from codes.h to autoenums.h to be more generic and allow for additional automatic enumerations.

During initialization, in addition to checking for duplicates and missing entries, a translation index array was built to translate from token status value to index. Both the checking and the translation array were removed since they are no longer necessary. The awk script will check for duplicates and the token status is now the same as the index into the message array.

The error type template class is no longer needed for token status errors (but still for the table entry erros). Also, the duplicate and missing were no longer needed and were removed. Some problems were found in this template class for table entries where the range error was not working because the wrong constructor was being called. Instead of storing indexes to the errors, the variables were changed to be the type of the template. This made the range error constructor unique.

Translation – INPUT Command (Release)

The remaining problem was due to the input command handler deleting the token passed in (the token terminating the invalid string prompt expression), however, the caller (the call command handler routine) was also deleting the token since the command handler changed the token to point to the error token. The extra token delete was removed from the input command handler.

The INPUT command is fully working and ibcp_0.1.15-src.zip has been uploaded at Sourceforge IBCP Project along with the binary for the program.  To support the INPUT command, several other changes were needed including making the token codes and table entries indexes one in the same, correcting print function issues, correcting sub-string assignment issues, handling assignment token mode differently, handling define function tokens correctly, implementing an end statement Translator state, and implementing the reference token mode.

Next up, a slight change to the direction of this project to make it a little more interesting. Instead of prodding along with the translation of more commands, work will begin of the other components including the encoder, dictionary, recreator, program (maintaining the internal program and the program editor), and the run-time module.

The goal is to get this BASIC working as there are (almost) enough commands to make a very simple BASIC program (input, assignments and output). Once this is working, more commands can be implemented. But first a couple of minor things will be implemented before proceeding with this new direction.