Interactive BASIC Compiler Project

Switch in Linux Distribution

2011-06-25T10:16:00.000-04:00

The 64-bit Kubuntu 10.10 installation has been problematic. Several comments around the Internet have indicated this to be the case. The reason Kubuntu was chosen was that it contained the KDE desktop, and 10.10 was chosen because of it is the newest on the supported host operating system list for VMWare Workstation (being used for the Windows XP virtual machine). Standard Ubuntu comes with the GNome interface, which I am not a fan of.

The problems with Kubuntu mainly include general sluggishness, a problem that was never resolved after a little over a month of use (something not expected on a Phenom II quad code running at 3.2 GHz with 4GB of memory with a NDIVIA GTS-450). It would completely hose up at times (not lock up or crash) when doing even simple tasks like uncompressing files. Even after all the updates were applied (less the update to 11.04 Natty Narwhal), including KDE and all the fancy desktop effects were turned off, the problems persisted. Mounting a bunch of NTFS partitions (from the previous Windows XP installation) was also suspected to be a cause.

Therefore, the Linux Mint 10 (Julia) with KDE distribution, which is also based on Ubuntu 10.10 (Maverick Meerkat) was chosen for the next installation. Linux Mint is another distribution based on Ubuntu/Debian. All the various commands mentioned in previous posts for Kubuntu also apply to Mint.

However, several issues were discovered in the various required tools that were not mentioned previously. There are some additional packages required for building GCC 4.6.0 and its prerequisite libraries (the GNU autotools) and for building CMake (specifically the ccmake GUI). The Linker Problem on Linux and New Make System posts were updated with new information about these dependencies. After a day of using Linux Mint 10, it is running as expected for a modern operating system on a fairly decent (though not cutting edge) system.

Using CMAKE on Windows

2011-06-23T22:49:00.000-04:00

The latest version of CMAKE (2.8.4) for Windows was installed, which can be obtained from here (the Win32 Installer). The same version used on Kubuntu 10.10. On Windows, there is a CMAKE (cmake-gui) program installed in the CMake 2.8 start menu group and it took some effort to figure out how to use it. Here is the procedure for building the program, starting in the CMAKE (cmake-gui):

Select "Browse Source..." and point to where the source is
Select "Browse Build..." and point to where the build will be (which should not be the source directory)
Optionally there is a "Make New Directory" to create a "build" directory
The next step is to select the "Configure" button, however, if the "Current Generator:" does not say "None" then select File/Delete Cache to make it change to None.

Before doing these steps, CMAKE needs to be able to find the GCC compilers and other tools. The Windows environment variable PATH needs to be modified, which can be done by inserting "C:\MinGW\bin;C:\MinGW\msys\1.0\bin;" at the beginning of PATH (assuming the default directories were used in the MinGW installer). Upon selecting "Configure" in Step 4, CMAKE will pop up a dialog. Change the generator for the project to "MSYS Makefiles" and select the "Finish" button. If all goes well, some lines will be output in the bottom section with no errors (errors are output in red text).

Now select "Generate" and the make file will be created. In an MSYS window, change to the build directory and do a "make" command. The program should now be built. All the generated files will be in the build directory including the executable, the auto-generated header files and the codes.txt file.

Using the CMake System

2011-06-21T23:19:00.001-04:00

There are a few simple steps for building using the CMake system. It's worth noting that when using CMake, all the output files are put into a separate build directory. This prevents the source directory from getting cluttered with files. Assuming that the source files are in the ibcp sub-directory, an ibcp-build directory will be created. These commands are used for building the project:

mkdir ibcp-build
cd ibcp-build
cmake ../ibcp
make

If no errors occur from Step 3 (like the correct version of GCC was not found), then Step 4 will build the program. All auto-generated include files, object files and the program will be located in the ibcp-build directory. To enable debug information in the executable, the ccmake program is used with these steps:

ccmake ../ibcp
Press Enter on the CMAKE_BUILD_TYPE line
Type in "Debug" and press Enter
Press [c] to configure
Press [g] to generate and exit
Now entering make will build with debug information

Setting the string to "Release" in Step 3 will turn off the debugging information. There is also a "make clean" option. If the build is not performed in the source directory (possible but not recommended), the entire contents of the build directory can also be deleted and cmake rerun. Now that CMake is working on Linux, it is now time to make it work under Windows...

The CMake System

2011-06-20T22:19:00.000-04:00

The CMake system consists of a CMakeLists.txt file that describes how to generate the make file that will be used for building a project. The initial CMakeLists.txt file consisted of these major sections:

Check that the GCC version is 4.5 or later.
Find the awk program.
Pass version information to the IBCP source.
Create custom commands for the auto-generated header files.
Define the lists of header and source files.
Add linker definitions for the static library linking.
Define the executable.

Step 1 is needed to make sure the needed version of the GCC compiler is installed and setup correctly. Step 2 checks if the awk program is available and gets it's path - the awk program is needed to create the auto-generated header files. Step 3 is to pass program version and copyright information to the source program. This is accomplished by an ibcp_config.h.in file with definitions set to CMake variables to create an ibcp_config.h file that will be included in the source.

Step 4 contains two custom commands to run awk to create the two auto-generated header files. The awk scripts were modified to take a command line argument for the path of where the output files are to written to since the awk scripts will be run in the build directory, not in the source directory. This path argument was made optional and if not provided, the awk scripts assume the current directory. In the CMakeLists.txt custom commands, the source directory path is passed to the awk scripts. The auto-generated header files are now written into the build directory (CMake calls this the binary directory), not the source directory, so a statement was also needed in CMakeLists.txt to include the binary directory to the list of paths that are searched for header files.

In Step 5, the list of header and source files are put into CMake variables. These variables are used to define what the dependencies are and are used to build the executable. Step 6 adds the ‑static‑libgcc and ‑static‑libstc++ linker options. Finally, Step 7 defines the executable, which depends on the auto-generated header files, the project header files and the source files. The ".exe" is not needed on the executable program name as CMake will automatically add it for Windows, but not for Linux (CMake is aware of which system is being used for building).

New Make System

2011-06-19T21:42:00.003-04:00

The standalone make file was originally generated by a feature in NetBeans, and was heavily modified to make it work, but it never looked very clean. Some consideration was given to rewriting it using the proper make rules (having individual build rules for each individual source file is not the way make is suppose to be utilized).

Continuing to use the make system within NetBeans was not desirable, though it allows the setup to deal with multiple platforms (Linux and Windows). A reason for not using it is that releasing all the associated files is not really an ideal option. And it never really worked right for the auto-generated include files from the awk scripts - warnings were generated for every build and no solution for eliminating these warnings was ever found (perhaps this was a bug in 6.9.1 that may have been fixed in 7.0).

Using the GNU autotools was out of the question - way too complicated and too much time coming up to speed with it. Instead the CMake system (Cross platform make generator) will be used as it is far simpler to setup and use. Plus NetBeans is supposed to understand CMake files. Adding CMake on Kubuntu installed version 2.8.2, however, this version did not contain the ccmake user interface for configuring the build. So the Linux source for version 2.8.4 (the latest) was downloaded (from here) and installed.

There are two prerequisite packages that are required to build the CCMake GUI (more on this later), which are the libncurses5 and libncurses5-dev packages. These can be installed with this command:

sodu apt-get install <package-name>

Upgrade to NetBeans 7.0

2011-06-19T08:20:00.002-04:00

Netbeans 7.0 was recently released (version 6.9.1 was being used for development on Windows), which contains support for git via a plug-in. Getting that to work took some research and experimentation. It appeared that the git plug-in only allowed creating a new git repository, not opening an existing one. However, it was discovered that if a new project is created from existing source and standalone make file in a directory with a git repository, NetBeans will then see the repository.

There was a problem building the project. The compiler complained that it didn't recognize the -static-libstdc++ option. NetBeans was still executing GCC 4.4.5, even though it was correctly pointing to the GCC 4.6.0 directory. Setting the path to include GCC 4.6.0 before executing NetBeans from the command line did not help.

Next an attempt was made to run the debugger on the project. There was no debug option in the standalone make file. Previously, the project was set up under NetBeans and it generated its own complex and convoluted set of files for generating what it needed to build the project. Both a Release and a Debug configuration was created by NetBeans. To release the project, NetBeans was used to create a standalone make file instead of releasing all the NetBeans files, which would have required NetBeans to build.

This time around, NetBeans was given this (heavily modified) make file when the project was created, though there was no Debug configuration. The make file was temporarily modified to turn on debug compiling. The make system needs to be reworked to allow both a release and debug configuration without using the system built into NetBeans.

Linker Problem on Linux

2011-06-18T10:19:00.005-04:00

Now that the program builds and runs on Windows, it was time to make sure it still worked on Linux. However, the linker failed because the compiler/linker did not understand that ‑static‑libstdc++ option. Upon research, it was discovered that this option was only in GCC version 4.5 and later, so it was not available in the GCC 4.4.5 on Kubuntu.

The latest version of GCC (4.6.0) was downloaded and installed on Kubuntu using the source code. This required quite a bit of work to accomplish (for complete details, hit the continue link). Now that the project was building on both Linux and Windows, it was time to introduce it to the NetBeans IDE (which was now at version 7.0, version 6.9.1 was being used for development on Windows).

Before GCC 4.6.0 could be built on Linux, some prerequisite packages and libraries need to be installed first. The packages include m4, autoconf, automake, libtool and libc6-dev-i386. Each of these can be installed with this command:

sudo apt-get install <package>

The libraries include GMP (GNU Multiple Precision arithmetic library), MPFR (Multiple Precision Floating point library with correct Rounding) and MPC (Multiple Precision Complex number library). The sources of the latest version of each of these were downloaded and installed (GMP 5.0.1, MPFR 3.0.1 and MPC 0.9). The procedure used for installing each of these were:

uncompress the library source (tar xjf xxx.tar.bz2 or tar xjf xxx.tar.gz)
change to the new directory created (cd xxx)
./configure
make
sudo make install

All of these libraries were installed at /usr/local/lib and /usr/local/include by default. Now GCC 4.6.0 was successfully built and installed using this procedure:

tar xjf gcc‑4.6.0.tar.bz2 ‑or‑ tar tzf gcc‑4.6.0.tar.gz
mkdir gcc-4.6.0-build
cd gcc-4.6.0-build
../gcc‑4.6.0/configure ‑‑prefix=/usr/gcc‑4.6.0 ‑‑with‑gmp=/usr/local ‑‑with‑mpfr=/usr/local ‑‑with‑mpc=/usr/local
export LD_LIBRARY_PATH=/usr/local/lib
make bootstrap-lean
sudo make install

For a multiple core processor, the ‑jX where X is the number of cores, can be added to the make commands to speed up the build. This will install GCC into the /usr/gcc‑4.6.0 so as not to overwrite the GCC that comes with Kubuntu. However, the PATH and library environment variables need to be set properly so it uses GCC 4.6.0 and not GCC 4.4.5 (or whichever version is installed). To do this, these commands are used:

export PATH=/usr/gcc‑4.6.0/bin:$PATH
export LD_LIBRARY_PATH=/usr/gcc‑4.6.0/lib64:/usr/gcc‑4.6.0/lib:/usr/local/lib

Which can be put into a script file (for example ~/bin/gcc460) and executed using the command ". ~/bin/gcc460" to set the variables. The period tells the current shell to execute the script directly, otherwise the script is executed in a new shell and when done, terminates having no effect in the current shell. The gcc‑4.6.0 (source) and gcc‑4.6.0‑build directories can safely be removed once GCC 4.6.0 is successfully installed, along with the directories used to build the three prerequisite libraries.

Running on Windows

2011-06-17T22:28:00.004-04:00

There were no problems building the modified project back on Windows, however, when running the regression test script on Windows, all tests failed. This occurred because the expected test results files were in Unix (LF only) format, but the freshly generated output files were in DOS (CRLF) format. The files otherwise appeared to be identical.

To verify that the output was the same, the test script was modified where the cmp ‑s command (‑s for silent) was replaced with a diff ‑bq command (‑b for ignore white-space, ‑q for quiet) to have it ignore the differences between CRLF and LF, both considered white-space. Now all tests compared to the expected results. A better solution will need to be found to deal with the differences between Windows and Linux.

One other problem was found when trying to run on Windows, which was discovered when trying to make the program run. Upon the first run, an error was reported that the application failed to start because libstdc++‑6.dll was not found. Once the environment variables were adjusted to include the MinGW binary path, the program ran fine.

This was the same problem as before with libgcc_s_dw2‑1.dll. This problem was resolved by adding a -static-libgcc option to the linker step, which links in the libgcc.a library. It turns out that there is similar option ‑static‑libstdc++ for linking in the libstdc++.a library. The program no ran without requiring any external library. This was tested by moving the program to another computer that did not have MinGW installed.

Setting up and Building on Windows

2011-06-16T22:45:00.001-04:00

Now that the code was building and working on Linux, it was time to setup the Windows XP virtual machine for building. On the previously Windows XP installation, MinGW (with GCC 3.4.6) and MSYS (1.0.11) were installed with a number of packages installed later (like GCC 4.4.0). It was a very hacked together setup in attempt to make things work right.

This time around, the Automated MinGW Installer was used. The current 2011-05-30 version was used, which contains the latest versions of MSYS, MinGW and all the other utilities. The latest version can be obtained here. When installing this package (which actually downloads all the latest versions before installation), both the MSYS and MinGW options were selected. The version of GCC installed was 4.5.2.

Moving the project software back over to Windows, the build succeeded. Turns out the awk scripts still work in Windows. There was also no warning about the gets() function. This may be because the version implemented with MinGW contains more features, which were mentioned back on June 15, 2010 and are not implemented in the Linux version.

Integers on 64-bit Linux

2011-06-15T22:53:00.001-04:00

After a successful compile, the regression test script aborted with errors. The errors occurred because it was trying to execute ibcp but the make file was building ibcp.exe. This worked on Windows because it will execute ibcp.exe when the command is ibcp. The script was modified to run ibcp.exe instead (this issue will be resolved shortly). Now only two of the parser tests and one of the translator test failed.

The first parser test (immediate commands) was not working on the inputs containing large values for line numbers or increments that were suppose to be reported as an error, but now were being accepted and the value output was not the same as the input. A problem in the third parser test (numbers) was a negative integer one beyond the limit of an integer, was now also being accepted as an integer (it should have become a double since it was out of range for an integer). And a problem on the 17th translator test (negative values) was another negative number out of range of an integer, was also being accepted.

These problems were caused by the strtol() function calls used to convert strings to an integer. An overflow was expected for these large numbers. This function returns a long integer and on 32-bit Windows (or any 32-bit OS) this is the same as integer values, which is 32 bits, so an overflow was reported. However, on 64-bit Linux (or any 64-bit OS), long integers are 64 bits, so these large values were now being accepted, but when converting to an integer, were being truncated from 64 bits to 32 bits.

To correct this problem, in additional to the ERANGE (overflow) error occurred condition, a condition was added to check if beyond the maximum integer using the predefined constant INT_MAX (and INT_MIN when the value could also be negative).

The remaining problem in the third parser test was that only two digits were output for floating point exponents on Linux if a third (the hundreds) digit was not required. On Windows, three digits are always output for the exponent. This must be due to slightly different builds or implementation of the standard C library functions. How to deal with this problem for testing will be dealt with a little later.

Resolving 64-bit Linux Build Problems

2011-06-14T22:17:00.003-04:00

There were problems with the awk scripts - they were not generating the auto-generated includes files. This was caused by the RS (Record Separator) and ORS (Output Record Separator) variables being set to CRLF to work with Windows. The files were now all in Unix (LF) format and so the input source files not being read correctly. So the RS and ORS lines were commented.

The git configuration option autocrlf was set to false, causing git upon checkout to change any DOS (CRLF) format files in the repository to be converted to Unix format (LF), or the CRLF was removed when the CVS repository was converted to git. There were previous problems with CRLF on Windows when switching between cvs with MSYS to CVSNT (all the files were changed to DOS format since CVSNT didn't seem to like Unix format files).

Next there were warnings about the gets() function calls in the test routines, which were not present on Windows (using GCC 4.4.0 on MinGW/MSYS). After updating with the package manager, Kubuntu had GCC 4.4.5, so perhaps this was a change between 4.4.0 and 4.4.5 or a difference between the GCC build on MinGW vs. Kubuntu. The warning complained that the returned value from gets() was being ignored. To remove the warnings, statements were add to check the return value.

The linker also warned that the gets() function is dangerous and should not be used. This is probably because gets() does not check for a buffer overflow of the allocated buffer given to it. The buffer allocated is plenty big for testing here and this is just the test code anyway, so this warning will be ignored from the time being.

Building on 64-bit Linux

2011-06-13T23:43:00.000-04:00

Before attempting the build the project on Linux, the version control repository of the project code was converted from the Concurrent Versioning System (CVS) to git, a more modern version control system. The version installed using Kubuntu's software package manager was 1.7.4.4. However, this version did not have the needed cvs import utility, so the latest version was installed directly from the git repository, though git is required to retrieve it. This was version 1.7.6. Using git will have no impact on this project's software releases except that the CVS versions tags will be removed from the source files.

The intention was to use command line make file distributed with the project to build on Linux. Previously on Windows, NetBeans was used to build the project, though the command line make file (which was initialized generated by an option in NetBeans) was used to test the project before a release. For the moment, NetBeans would not be used (more about this later). Some minor problems needed to be resolved before it would build and work on Linux...

Development Environment Change

2011-06-12T21:24:00.000-04:00

Because of a job requirement, I had to install 64-bit Linux on my computer (I chose Kubuntu 10.10, Maverick Meerkat). So development of this project will be moved over to 64-bit Linux, at least for the time being. Since so far, the program only has a simple command line interface (no GUI), this won't cause an immediate problem. The releases will still be tested under Windows XP as I have a Windows XP virtual machine running on Kubuntu. Perhaps when the time comes to start on the GUI, a common tool set will chosen, something like Qt.

For the moment, this will also give the opportunity to present how to set up Windows to be able to build the project since this virtual machine is a new Windows installation and not a clone of my of my previous Windows installation. A lot of things were originally tried and listing the procedure that finally worked in the end was probably impossible.

Also possibly changing will be the actual development tools that will be used. Under consideration is changing the version control system (from cvs to git), the IDE (from NetBeans 6.9.1 to newer version 7.0, which has a plug-in for git, or possibly to the Eclipse CDT IDE), and the build system to build the project (from NetBeans' make system and standalone make file currently distributed with the project to the cmake build system). Each of these will be explained in the posts that follow...

Translator – Colon (Design)

2011-03-30T22:13:00.000-04:00

The colon token will work very similar to the end-of-line token and so will have the end statement flag in its table entry. The difference with the end-of-line token is that instead of terminating the translation or a line, the token mode will be set to command for the next statement.

There will be two error conditions. Two colons will not be allowed. While a second colon would not affect anything and is allowed in other BASICs, it will be considered an error here. There is no point to allowing this. Also, a colon will not be allowed at the end of the line with no command after it.

When a colon token is received, the colon sub-code will be set in the command token on top of the command stack. The command token will be appended to the end of the statement when the command has been processed (by the command handler).

However, there is an issue with print statements. The actual print command token may not be appended to the output if there is a semicolon, comma or print function at the end of the print statement. In this case, the print command handler will have to transfer the colon sub-code to the last print code that was appended to the output, which may be a print type (when a semicolon is not needed), comma, semicolon, SPC or TAB.

Colon – Statement Separator

2011-03-29T21:30:00.000-04:00

The Colon is used to separate statements in the BASIC language though it is not part of the ANSI BASIC, it is part of the more common BASICs (GW-Basic, QBasic, FreeBasic, etc.). Before moving to the translation of colon tokens, like everything else, the action during run-time must be defined. However, the Colon does not actually do anything at run-time.

Colons don't actually need to be stored as separately in the program, however, for them to be reproduced, something needs to be put into the internal program. The assumption that there will be a colon at the end of each statement except at the end of the line is not sufficient. Consider this statement:

IF A>B THEN PRINT A ELSE PRINT B:A=B:B=0.0

There is no colon after the first print statement. To reproduce colons properly, there will be a colon sub-code set for a command token that has a colon following the statement. For the statements after the ELSE in the example above, the colon sub-code will be set as shown in this translation:

B PrintDbl'Colon' A<ref> B Assign'Colon' B<ref> 0.0 Assign

When a line is reproduced, the Recreator will add a colon after a statement that has the colon sub-code set.

Translator – A Unary Operator Curiosity

2011-03-27T19:13:00.000-04:00

One of the test statements created for testing the unary operator fix was:

A = -B^NOT C% + -D*NOT E%

The intention of this statement was test the NOT unary operator in front of the second operand of both the exponentiation and multiplication operators. The translation of this statement was expected to be (the blue expression being the first operand and the red expression being the second operand of the addition):

A<ref> B C% NOT ^* Neg D Neg E% NOT *%2 + Assign

However, what was produced was this unexpected translation:

A<ref> B C% D Neg E% NOT *%2 +%1 Cvtint NOT ^* Neg Assign

Upon reviewing the precedence of the operators (see Translator – Operator Precedence) and the code, it turns out that this translation was correct. The ADD is higher precedence than the NOT, so the operands of the ADD are C% and the –D*NOT E% expression with MUL (*%2) higher precedence than ADD, its operands are the –D and NOT E%).

So while the exponentiation is highest precedence, with NOT having a low precedence allowed the ADD to bind the rest of the expression to the NOT, which becomes the second operand (yellow above) of the exponentiation, with the first negation being the final operator. In C/C++, the not (!) and negation (-) operators are very high precedence (and there is no exponentiation operator). But here, NOT was given a low precedence just above the other logical operators but below the math operators (see Translator – Operator Precedence for reasoning). Normally the NOT operator would probably not be used in the same expression as exponentiation like above.

Translator – Unary Operator Problem

2011-03-27T14:44:00.000-04:00

While testing the negative constant changes, a new problem was discovered with unary operators, specifically this statement:

A = ---B

Which produced a “done stack empty” bug error at the first negation token. The problem occurred because the second negation operator forced the first negation operator from the hold stack because it was greater or equal precedence, and when checking the operand of the first negation operator, there was nothing on the done stack. Here and some additional examples:

A = B*-C
A = B^-C

The first statement translated correctly because negation is higher precedence than multiplication leaving multiplication on the hold stack. However, the second statement failed because negation is lower precedence than exponentiation forcing exponentiation from the hold stack but with only one operand on the done stack generating the done stack empty bug error. A new rule was needed for unary operators.

Basically, unary operators should not force any tokens (unary operators, binary operators, arrays, or functions) from the hold stack regardless of their precedence because not all of their operands have been received yet (the negate and its operand will be their operand and it has not been fully received yet). As currently implemented, other non-unary operators should still force unary operators from the done stack if the unary operator has higher precedence.

The check to force tokens from the hold stack was changed to if the precedence of the operator on the hold stack is higher than the current operator and if the current token is not a unary operator. Unary operators will now not force other tokens from the hold stack, but other tokens will still force unary operators from the hold stack if higher in precedence. While testing this change, a curious result was produced from one of the test statements...

Parser – Negative Constants

2011-03-27T10:55:00.000-04:00

Negative constants were previously not considered by the Parser, which interpreted a minus as the subtract operator. The Translator then changed it to a negate operator when it appeared in the operand state. Consider these two examples (along with there current translations):

A = B-1.5 A B 1.5 Sub Assign
A = -1.5+B A 1.5 Neg B Sub Assign

The reason for the Parser to not look for signs on numerical constants can be seen in the first example. If the Parser produced the four tokens A = B -1.5, the Translator would generate an “expected operator” error at the -1.5 token since a second operand token was received when it was expecting a binary operator. The second example produces an unnecessary negate token after the constant. While perfectly valid, this is not desirable.

In order for the Parser to correctly interpret negative signs on numerical constants, it needs to be aware of whether the Translator is in operand state or not. If in operand state, the Parser can look for a negative sign in front of a number constant, otherwise a minus should be interpreted as an operator.

A new operand state flag was added to the Parser with an access function to set its value (which is initialized to off). The Parser get number routine was modified to have a new sign flag used to determine if a negative sign was found first. This flag will also prevent multiple negative signs. However, it will only check for a negative sign if no digits or a decimal was seen and if the new operand state flag is on.

An access function was added to the Translator to get the current operand state (either operand or operand-or-end state). Before calling the Parser get token routine, the Parser operand state is set from the Translator's current operand state. While testing, a problem was discovered with unary operators...

Parser – Tokens With Parentheses

2011-03-26T20:54:00.000-04:00

While correcting issues with define function tokens, it was noticed that it is not necessary to also store the opening parentheses in the string field of the token. This also includes the generic tokens with parentheses. The parentheses is not necessary because there are separate token types to identify tokens with and without parentheses.

It is also advantageous to not store the parentheses so that an array or define function name can be found in the dictionary. For define functions, take the code snippet:

DEF FNHypot(X,Y)
FNHypot=SQR(X*X+Y*Y)
END DEF
Z = FNHypot(3,4)

Both the FNHypot( and FNHypot tokens appear, which represent the same function. If the parentheses was stored in the dictionary for the function name, it would require complicated string comparisons to figure out that the FNHypot token is the same function. This same issue applies to regular function names.

A similar issue also applies to arrays. When functions and subroutines are implemented, there will be a feature to allow an entire array to be passed to a function or subroutine. Exactly how this will work has not been defined yet, but this code snippet shows how it would look:

DIM Array(10)
CALL subroutine(A)

The changes for removing the parentheses from these tokens were very simple. In the Parser get identifier routine, when creating the string for these tokens, the length provided to the string constructor was changed to be one less than the actual length so the parentheses would not be included. In the Translator when reporting an error at the open parentheses of a define function with parentheses token, the minus one was removed since the length is now one less. Finally, in the token test output routines, an open parentheses was added to the output of these tokens.

Project – Tokens Status Enumeration

2011-03-26T17:24:00.000-04:00

Each time a new error (token status) is added, renamed or deleted, two changes were needed. Both the token status enumeration (include file) and the message array (source file) needed to be changed. The correct changes were needed or the two files will be out of sync. To check for problems, code had been added to initialization to check for duplicates and missing entries.

Similar to automatic generation of the code enumeration from the table entries, the token status enumeration will also be generated automatically. Each message array element was a structure containing a token status value and a pointer to a message string. The elements were changed to just the message string with the name of the token status in a comment at the end of the line.

The codes awk script was renamed to enums and was modified to also read the token status message array to generate the token status enumeration automatically be reading the name in the comment after the message string. The name of the output file was changed from codes.h to autoenums.h to be more generic and allow for additional automatic enumerations.

During initialization, in addition to checking for duplicates and missing entries, a translation index array was built to translate from token status value to index. Both the checking and the translation array were removed since they are no longer necessary. The awk script will check for duplicates and the token status is now the same as the index into the message array.

The error type template class is no longer needed for token status errors (but still for the table entry erros). Also, the duplicate and missing were no longer needed and were removed. Some problems were found in this template class for table entries where the range error was not working because the wrong constructor was being called. Instead of storing indexes to the errors, the variables were changed to be the type of the template. This made the range error constructor unique.

Translation – INPUT Command (Release)

2011-03-25T22:08:00.000-04:00

The remaining problem was due to the input command handler deleting the token passed in (the token terminating the invalid string prompt expression), however, the caller (the call command handler routine) was also deleting the token since the command handler changed the token to point to the error token. The extra token delete was removed from the input command handler.

The INPUT command is fully working and ibcp_0.1.15-src.zip has been uploaded at Sourceforge IBCP Project along with the binary for the program. To support the INPUT command, several other changes were needed including making the token codes and table entries indexes one in the same, correcting print function issues, correcting sub-string assignment issues, handling assignment token mode differently, handling define function tokens correctly, implementing an end statement Translator state, and implementing the reference token mode.

Next up, a slight change to the direction of this project to make it a little more interesting. Instead of prodding along with the translation of more commands, work will begin of the other components including the encoder, dictionary, recreator, program (maintaining the internal program and the program editor), and the run-time module.

The goal is to get this BASIC working as there are (almost) enough commands to make a very simple BASIC program (input, assignments and output). Once this is working, more commands can be implemented. But first a couple of minor things will be implemented before proceeding with this new direction.

ranslation – INPUT Error Debugging

2011-03-24T22:03:00.002-04:00

The main problem with the wrong tokens being reported for errors in the INPUT command was because the token with the error was not put into the command item structure passed to the INPUT command – a requirement of command handlers reporting an error. Once this was added, most of the errors were now pointing to the correct token.

A new “expected operator, semicolon or comma” error was added for when an end statement token (for example the EOL in the incomplete statement INPUT PROMPT A$) is received after a valid string expression because this is a little more accurate than just the “expected semicolon or comma” error.

The two remaining errors were “invalid mode” bug errors that were occurring in the end expression error routine that is called when an end statement token is received during operand state. Support for the reference mode needed to be added to this routine, which needed to return the “expected variable” error.

One problem remains. The error test statement INPUT PROMPT A+B*C is causing extra token deletes, which was detected by the memory leak detection mechanism that was implemented a little while ago. To be continued...

Translation – INPUT PROMPT Debugging

2011-03-23T22:11:00.000-04:00

The first problem found with the INPUT PROMPT command was that reference mode was not being set after the string prompt expression was processed. The next problem was that the Translator was still in binary operator state following the comma or semicolon after the string prompt string expression was processed, so this required setting the state back to operand state, which lead to the discovered of some other minor state issues.

The first operand state is very similar to the operand state except that end expression tokens (like comma, semicolon, and EOL) are also acceptable (normally considered operators). This state was implemented for the PRINT command since these tokens are allowed when an operand is expected (for example the PRINT,,A statement). This state is also set after a command is received, and it is up to the command handler to decide if an immediate end expression is allowed (which it currently is only for the PRINT command).

The equal token handler was found to be incorrectly setting the first operand state in an assignment statement. This did not cause a problem because the end expressions operators were being caught elsewhere when in expressions incorrectly (by their respective token handlers). Anyway, calling it the first operand state was a little confusing and was therefore renamed more appropriately to the operand or end state. The equal token handler was corrected to set only operand state.

The valid INPUT PROMPT test statements are now working, now on to the invalid INPUT statements, which for the most part are not reporting the correct token where an error is detected or just not reporting the correct error including some bug errors...

Translation – INPUT Debugging

2011-03-21T21:58:00.001-04:00

Several minor issues were found and corrected. When the end of the INPUT command occurs, the last input parse code has to be marked with the end sub-code. Support for reference mode also needed to be added to comma token handler.

When the EOL token was received by the INPUT command handler, it reused the token for the input assign code for the variable. Upon return from command handler, the EOL token handler proceeded to delete the EOL token (which was not the EOL token anymore). To prevent this, a check was added that if the token no longer contains an EOL code, it is assumed to have been used by the command handler and will not be deleted.

The InputBegin code was just being appended to the output, but the first variable had already been added to the output, so it was after the variable instead of at the start of the statement. This code could be inserted at the beginning of the output list, however, this would only work if the INPUT command was at the beginning of the line, which may not be the case (multiple statements per line will be supported).

So instead, the element pointer in the command item will be set to the current last item in the output list when a command is pushed to the command stack. Since this pointer can no longer be checked for null to determine if an input begin code has been added, a new input begin command flag was added, which is set once an input begin code has been added to the output.

Another problem found was that none of the input variables had their reference flag set once added to the output. The find code routine was not checking for a reference (not important) but was clearing the reference flag (a problem) since the token being checked (the input assign code) did not have its reference flag set. The reference flag of the input assign token was set before calling the process final operand routine to make the find code routine work as desired, and then cleared upon return.

The valid INPUT test statements are now working, now on to the INPUT PROMPT statements, which are not working...

INPUT Translation – Variable Handling and Ending

2011-03-20T08:01:00.002-04:00

To look up the appropriate input assign code, the process final operand routine is used, which calls the find code routine that checks the token on the done stack to see if the reference flag is set since the input assign codes have the reference flag (this should not be necessary since the INPUT command uses the recently implemented reference mode). The reference variable will be popped from the done stack and the input assign code will be appended to the output before returning.

The InputAssignInt and InputAssignStr are associated codes for the InputAssign code. Once the input assign code has been appended to the output, the appropriate input parse code needs to be inserted after the input begin code or last input parse code. The easiest way to get the appropriate input parse code was to have each (InputParse, InputParseInt, and InputParseStr) be associated codes to the input assign codes. The second associated code will be used for these.

When a final semicolon token is received, the stay on line command flag is set (the same flag used for the PRINT command) and the state is set to end statement. When an end statement token is received, the INPUT command handler check if the stay flag is set and sets the keep sub-code of the INPUT command token, which is then appended to the output.

If an end statement token is received with no semicolon, the INPUT command token is immediately appended to the output without the keep sub-code. The INPUT command handler has now been implemented and the code compiles, so debugging can begin...

Translator – End Statement State

2011-03-19T21:47:00.000-04:00

Once a semicolon is received at the end of an INPUT statement, no more tokens should be received except for an end-of-statement token. If another token is received, then an error should be reported. To accomplish this, a new end statement state similar to the end expression state is needed. While the end expression state only needs to be checked when operators are expected, the end statement state needs to be checked for all tokens.

The end statement state check was added just before the check for operand or first operand state in the main translator add token routine. When in end statement state, if the token does not have the end statement flag set in its table entry, then an “expected end-of-statement” error is reported against the token.

The access function for getting the table entry flags for a token already checks if the token has a table entry, and if there is not table entry, then zero (no flags) is returned. Currently only the EOL code has the end statement flag set, but eventually the Colon, ELSE and ENDIF tokens will also and possibly other codes.

Translation – Define Function Token Issues

2011-03-18T22:13:00.000-04:00

For now in the process operand routine, define function with parentheses tokens will not be allowed in command or assignment mode. This will need to be changed later when the DEF command is implemented. This check was also made in the check assignment list item routine.

However, since a define function without a parentheses token is allowed in assignments, the error was set to point to the open parentheses as an "expected equal or comma for assignment" error. The open parentheses is at the end of the token, so to get the error to point to it, the column of the token was incremented by the length of the token minus one.

Previously in the close parentheses token handler, the reference flag was being set for token with parentheses and define function with parentheses tokens. This check was modified to only set the reference flag for token with parentheses.

The reference flag for define function without parentheses tokens was already being set, so no change was needed for these tokens.

Language – Define Functions

2011-03-17T22:14:00.000-04:00

Before defining what needs to be done with define function tokens in the Translator, a quick review of their syntax is required. The will be two forms of define functions that will be supported, a single line and a multiple line. An example of a single line define function would be:

DEF FNHypot(X,Y)=SQR(X*X+Y*Y)

Notice that this form has the same format as an assignment except for the DEF command at the beginning. The define function token in this statement is FNHypot( - in other words, a define function with parentheses. This implies that a define function with parentheses token could appear in assignment mode (assuming the DEF command sets this mode), but only for the DEF command. An example of the multiple line form of the same function would be:

DEF FNHypot(X,Y)
FNHypot=SQR(X*X+Y*Y)
END DEF

The assignment of the define function name (a define function without a parentheses) returns the value for the function. Here a define function without a parentheses can appear in an assignment statement, but only inside a DEF/END DEF block. Since the Translator is not aware of blocks, it will permit an assignment of define function without a parentheses token. It will be the Encoder's job to verify if the assignment is valid.

Translation – Assignment Token Issues

2011-03-16T21:00:00.000-04:00

Finally, some issues were discovered when making the change from command mode to assignment after processing an operand token. There are two main types of operand tokens, ones with parentheses and ones without parentheses.

The operand tokens with parentheses include internal functions, defined user functions (DEF FN) and generic tokens with parentheses (which can be arrays or user functions). Internal functions were already invalid for command or assignment mode except for sub-string functions. A check was added for defined functions with parentheses.

The operand tokens with no parentheses include internal functions with no arguments (currently only RND), constants, define user functions with no arguments, and generic tokens with no parentheses (which can be variables or user functions with no arguments). Internal functions and constants were already invalid because they didn't have the reference flag set when the comma or equal token looked for the reference flag. That left define function tokens...

Translation – Command Token Issues

2011-03-15T22:44:00.000-04:00

The last problem statements related to unexpected command tokens were:

A PRINT B
MID$(A$ PRINT,4)=""

The first statement gave an “expected operator or end-of-statement” error at the B token. The second statement was actually accepted, but with a strange translation. The problems were caused because when the PRINT command token was received, it was immediately pushed to the command stack because the mode was still set to command.

Both statements start as assignment statements, but assignment mode was not being set (unless preceded by the LET keyword). When an equal token is received expression mode is set, or when a comma token is received assignment list mode is set. The “unexpected command” error was only occurring when the mode was not set to command, which didn't occur with the statements above. Also, this message again does fit with the “expected ...” type of message.

Once a command token is received in command mode, the mode is set according to the token mode in the command's table entry. A change was made to the process operand routine that once an operand token is processed, if in command mode, the operand is assumed the beginning of an assignment statement and so the mode is changed to assignment.

To remove the “unexpected command” error and report a more appropriate error, the command token has to be passed through the rest of the Translator. This will occur when the Translator is not in command mode. So the main add token routine was changed to not report this error if a command token is received and the mode is not command.

Command tokens received in operand state were already being reported correctly since commands are also considered operators, which are not valid operands (unless the operator is a unary operator, which commands are not). Commands are considered operators because some commands can be found where an operator is expected, for example, THEN and ELSE.

Commands tokens received in operator state are only valid if they have a token handler. In the process operator routine, when an operator token does not have a token handler a default operator token handler is called. Before the default operator token handler is called, a check was added to return an appropriate error if the token is a command.

Translation – Other Errors

2011-03-13T21:33:00.000-04:00

After changing the “item cannot be assigned” to “expecting item for assignment” error, there were several other errors that didn't fit the “expecting ...” type of message. It turned out that most of these were not actually being used, so there were removed.

Another remaining message was the “missing open parentheses” error that occurs when there is a parentheses with no open parentheses, function or array. After some consideration of possibly leaving this message as is, it was decided to change this to an “expected operator or end-of-expression” error since the problem could also be a missing function or array, or even that the open parentheses was just a mistake.

Again assuming that everything is correct up to the problem, this change seemed appropriate, and “...expression” was used instead of “...statement” because the next token could be a comma or semicolon in a PRINT statement or a THEN in an IF statement.

The last message was the “unexpected command” error that occurs when there is a command token when not in command mode. However, there were a number of additional problems with command tokens received when not expected...

Translation – Parentheses Issue

2011-03-13T17:16:00.000-04:00

The next problem statement was:

MID$((A$),4)=""

This was reported as an “item cannot be assigned” error and then crashed. Again, this error didn't fit the “expected ...” messages. This error is also returned for statements like 3=A and 1,A=B and was renamed to the “expecting item for assignment” error. For the statement above, the “expecting string variable” error should be returned.

The crash occurred because the open parentheses token was returned for the error with its range extended to the closing parentheses to report the entire (A$). The caller deleted the error token since it was an open parentheses to prevent a memory leak. However, in this case, the A$ token was still on top of the done stack with the open parentheses attached as the first operand. When the Translator clean up routine (called upon an error) was emptying the done stack, it deletes each item's first and last operand – the open parentheses was getting deleted twice causing the crash.

Initially to fix this problem, when an error occurs and the first through last operand is returned, the first operand pointer for the item on the done stack was set to null to prevent it from being deleted a second time. While this fix was sufficient for the statement above, this statement still were not being reported correctly:

MID$(-A$,4)=""

The error was “expected numeric expression” pointing to the A$. Both the open parentheses and the minus are initially processed in the process unary operator routine. So a check was added to this routine to return an error if there is a sub-string function on top of the hold stack with its reference flag set (sub-string assignment) and it is as the first operand. Both of theses statements then correctly reported “expected string variable” at the open parentheses and minus. The initial fix was not necessary since the error was being caught sooner.

Translation – Another Print Function Issue

2011-03-12T22:59:00.003-05:00

While the new reference mode was being implemented, some problems were discovered in the code that carefully constructed statements would exploit giving incorrect results. The first problem statement was:

TAB(10)=A

This caused a “command stack empty for expression” bug error. This should have been “expecting command” error. The check was if the command stack was empty or there was a PRINT command on top of the command stack or if the top of the hold stack was not empty (except for the null token).

The error occurred when trying to get the type of expression because the hold stack was empty (first item to figure out the expression type) so it when to the command stack, which was also empty. This check should not have caught this because there was a check after this with for catching all internal functions in command mode. The check was modified to if the command stack is not empty and if there is a PRINT on top of the command stack or the hold stack is not empty.

Translator – Reference Mode

2011-03-10T22:00:00.002-05:00

A new token mode is needed for the INPUT commands. The current token modes are command, assignment, assignment list and expression. Consider these invalid INPUT statements:

INPUT A+B*C
INPUT A+B*C$

The INPUT commands must have variables, not expressions. If expression mode was used, at each comma and the end of the statement, the INPUT command handler would need to check the token on top of the done stack to see if its reference flag is set. The first statement above, the INPUT command handler will see the multiplication token on top of the stack. Its first token will be A and last token will be C. An “expecting variable” error would be reported pointing to the whole A+B*C expression. This would be acceptable.

However, in the second statement, an error occurs before the INPUT command handler gets a chance to report an error. When the multiplication token checks its second operand, it will report an “expecting numeric expression” error pointing at the C$ token. This would make no sense. The proper error for both of these statements would be an “expecting semicolon, comma or end-of-statement” error pointing at the add token.

The new reference mode will only accept variables and array elements, specifically tokens with no parentheses and tokens with parentheses. If these tokens turn out to be user functions, which the Translator cannot determine, the error will be reported by the Encoder. Reference mode will be set when the INPUT command is received and after the comma or semicolon of the prompt string expression of the INPUT PROMPT command. It will also be used later for the READ command.

In reference mode, internal functions, define functions, and unary operators will be reported as invalid (“expecting variable” error). After the variable or array element token is pushed to the done stack, the state will be set to end expression so that only end expression tokens are valid. Binary operators will then be correctly be reported as invalid.

While reference mode was being implemented, some more problems were found in the Translator code...

Translation – PRINT Function Problem

2011-03-06T09:48:00.000-05:00

Due to the problems found with the print codes, some additional error tests were added to translator test 10 (PRINT statement tests) including these statements:

PRINT (TAB(10))
PRINT INT(TAB(10))
PRINT A(TAB(10))
PRINT A+TAB(10)
PRINT TAB(10)+A

The first three test the situation of a print function inside a parentheses, internal function and an array or user function, which were caught by adding count stack is not empty check when a print function token is received.

In the fourth statement, the error can be caught by checking if the hold stack is not empty (an empty hold stack has only a null token). In this case, an operator will be on top of the hold stack. In fact, this check can replace the count stack is not empty check because the open parentheses, internal function and array/user function will be on top of the hold stack.

The last statement was more difficult to check for - the expression should end after the print function. When the operator was received and checked its operand, a bug occurred because the done stack is empty since the print function didn't get pushed to the done stack. The error should be “expected semicolon, comma or end-of-statement” and point to the operator.

To catch this error, a new end expression state was added. The closing parentheses is received when in binary operator state. After the closing parentheses, the state was left at binary operator since another binary operator is expected (an end of expression token is acceptable as a binary operator). For the end expression state, only operators with the end expression flag are acceptable, which currently include the semicolon, comma and end-of-line tokens – other operators will generate the error.

In resolving these issues, the “invalid used of print function” error did not match the other “expecting...” errors (remember the goal is to help the user by suggesting what is expected at the location of an error). Therefore, this error was changed to be one of the “expecting xxx expression” depending on the current expression type (xxx would be blank, numeric or string).

Translation – PRINT Code Issues

2011-03-05T18:01:00.002-05:00

The process final operand currently doesn't push print codes to the done stack. This needs to be expanded to include the input begin prompt codes. It would be inconvenient not to keep expanding this test to include additional codes that don't need to be pushed to the done stack. The check could be changed to see if the code does not have a return data type (it is set to none). This would include the print codes, the input begin codes, the input parse type codes, the input assign type codes, and probably many more.

However, when this change was made, it did not work for the TAB and SPC print functions because the token data type for these had been incorrectly set to double. This occurred in the set default data type function when the token was received. This function was fixed to not set the data type for internal functions (these are set from their table entry data type).

Now when the token's data type is none, it won't be pushed to the done stack. The check if the command on top of the command stack is the PRINT command only applies to print functions, but is not necessary here as this check was already made when the print function was first received, so the check was removed. The check if the token's has the print flag remains (to set the stay and print function command flags, which is used by the print command handler to determine if the final print code should be appended to the output).

For the print type codes, when process final operand routine is called by the add print code routine, the second token passed is a null, not a closing parentheses. The process final operand was deleting the second token for print functions assuming that it was a closing parentheses (which only applies to TAB and SPC). For some reason, doing a delete with a null argument was not causing a problem. None the less, a check was added to only delete the second token if it is not a null.

Another problem was discovered for print functions. The situation if a TAB or SPC was contained within parentheses, internal function, or and array/user function, was not caught since it was only checking if there was a print command. Therefore, an additional check was added to make sure the count stack is also empty.

INPUT Translation – Some Issues

2011-03-03T22:31:00.000-05:00

As the INPUT command handler was being implemented, some issues were found. When the new element pointer was added to the command item, it was noticed that there was a code member. This member is no longer needed because the token now contains the code (replacing the index member, through the code is now an index), so the code member was removed from the command item.

There was no table entry for the two word INPUT PROMPT command, so one was added along with the three input begin codes. It was noticed that some table entries still had the string flag set. This is not necessary because the string flag is now set automatically during table initialization it there are any string arguments, so these string flags were removed.

To look up which input begin code to append for the INPUT PROMPT command, the process final operand routine will be called, which in turn will call the find code routine that will pick the correct code based on the type (string or temporary string) that is on the done stack. The input begin codes will not push anything to the done stack (accomplished by setting the done push flag to false). Some additional issues were found in these routines...

INPUT Translation – Variables

2011-03-02T21:48:00.000-05:00

Processing variables is a bit more complicated. With the PRINT statement, the appropriate print value type code was simply appended to the output. However, with the INPUT statement, an input parse code needs to be inserted after the begin code or after last parse code (before all of the variables), and an input assign code needs to be appended to the end of the output (after the variable).

There needs to be a way to point to the location where the input parse codes are to be inserted. This will be accomplished with a new output list element pointer member to the command stack item. This pointer will be initialized to null when a new command token is pushed to the command stack.

When an input begin code is appended to the output, this new element pointer will be set to the input begin code element. This pointer can also be used to indicate if any input variables have been received yet (when it does not contain a null).

To insert a input parse code, the input parse token will be appended to the list at (after) this element pointer. The element pointer will then be set to the output list element of the input parse token just inserted so that the next input parse token will be inserted after this input parse token.

INPUT Translation – Beginning

2011-02-28T21:24:00.002-05:00

Execution of the INPUT command has been defined and therefore the form of the translation, the actual process of the translation can now be defined. The translation begins with one of the InputBegin codes, which will be triggered by a comma or semicolon token. The INPUT or INPUT PROMPT token will already be on top of the command stack.

For the INPUT command, when the first comma, a semicolon or an end-of-line token is received, an InputBegin token will be appended to the output. The token received can be converted to the InputBegin token (more efficient to change the token than to delete the token not needed and then create a new one).

For the INPUT PROMPT command, when a comma or semicolon token is received, there must be a string on top of the done stack (the prompt string expression). Depending on whether this string is temporary or not will determine whether an InputBeginStr or InputBeginTmp will be appended to the output. For InputBeginStr, the string will be attached since the translator will not know if it is a variable, array or a user function. The token received can be converted to this token. An end-of-line token at this point would produce an “expected comma or semicolon” error.

Automatic Code Enumeration Generation

2011-02-27T16:28:00.001-05:00

To have the Code enumeration generated automatically from the table entries, the table source was structured so the awk scripts can read it. Since the Code enumeration value will now be the same as the table entry index, the code member of the table entry is not necessary and was removed. The code name initializers in the table entries was moved to a comment on the line of the entries' open brace.

The awk scripts were rewritten to read the table source file instead of the main include file. The awk script were also changed to read the table source file directly and write the output files directly. This eliminates the requirement to redirect the input and output to the correct files when running the awk scripts. Logic was also added to the codes awk script to check for duplicate code names.

The code to index conversion array that was initialized in the table class constructor, along with the check for duplicate and missing codes, was removed. The code and index access functions in the table class were also removed. The token class index member was replaced with a code member. All the code was updated to use the code enumeration value instead of the index, though the code will be used as an index.

One problem with using an enumeration value instead of an integer index is that normal math functions cannot be used, like the add and increment operators. These operators are needed, so operator functions were created for the code enumeration, which includes the add, prefix increment and postfix increment operators. These functions type cast to integer to add and then type cast back to the code enumeration value.

The null code entry at the end of the table was moved to the beginning so that that null code enumeration value (index) would be zero. The table search function for searching for an immediate command previously assumed that the immediate commands were at the beginning of the table entry array. Moving the null code to be the beginning of the array complicated this. Therefore, immediate command bracketing codes were put around these entries for this search function.

Due to the these table entry changes, the parser test output files were updated since all the code indexes changed. This would be a good time to make another pre-release, but since there have been no download activity for recent pre-releases, there will not be a pre-release at this time. Now the translation of the INPUT command can begin...

Code Enumeration vs. Table Entry Indexes

2011-02-26T16:35:00.000-05:00

While designing the error handling mechanism for the INPUT command, a thought occurred related to program codes. For efficient program execution at run-time, the index of the table entry will be stored in the internal program code, not the code enumeration value.

If the code enumeration value was used, the index for the table entry (needed to get the run-time handler function pointer) would first need to be converted to an index by going through the code to index array setup during table initialization. The intention all along has been to use table entry indexes in the internal program code.

For the INPUT command's error recovery, when it is backing up execution and checking for input parse codes, it will need to convert the table entry indexes to a code before it can check if it is an input parse code. This would not efficient during program execution. Even though for the INPUT command, execution time is not critical since it is about to stop and wait for user input. A few extra program cycles won't matter much. But this problem could occur for other more critical commands.

Therefore, it is desirable if the code enumeration values were the same as the table entry indexes. One simple solution is to make sure the code enumeration values matched the table entries. Unfortunately this relies on the programmer to keep the two in sync, is very error prone and is just a general pain to begin with.

There is a better way where the code enumeration is generated automatically from the table entries using an awk script. This method would be similar to how the test_codes.h file (used by the test_ibcp.cpp source file) is generated automatically by scanning for codes in the ibcp.h file.

INPUT Execution – Error Handling

2011-02-26T08:45:00.000-05:00

Errors can occur while parsing the input – in one of the input parse codes. When an error occurs, after it is reported, the already parsed values in the temporary input values stack need to be thrown away and execution needs to resume at the beginning of the INPUT statement, except instead of issuing the prompt again, the cursor will be positioned at the beginning of the input, which will contain the previously erroneous input to allow the user to correct the input instead of reentering it (the traditional “redo from start”).

The temporary input values stack can't simply be reset (setting the internal index to -1, the empty stack indicator) because elements may contain allocated string values, which need to be deleted to prevent memory leaks. Like the evaluation stack, the elements in the temporary input values stack won't have any indicator what data type they are. Consider the basic format of the INPUT statement (only up to the parsing codes is shown):

InputBegin InputParseType1 InputParseType2 InputParseType3'End' ...

Say an error occurs on the second parse code (the program execution pointer will be pointing at next code, the InputParseType3, in other words, the pointer is incremented after reading each program code word, then the code read is executed by calling its run-time handler). Execution needs to be backed up until the InputBegin is reached (reading the program codes in reverse).

As each code is passed, one element will be popped from the temporary input values stack. If the code passed is an InputParseStr code, then the element popped is a string that needs to be deleted. The beginning is reached when a non input parse code is reached (it could be InputBegin, InputBeginStr or InputBeginTmp). The stack will now be empty.

The INPUT begin code will be executed again and the begin code will call the get input routine. The get input routine will normally allocates the temporary input values stack. For error recovery, it will see that the stack is already allocated, so instead of outputting the prompt and saving the cursor position to the beginning of the input, it will restore the saved cursor position and get the input starting with the previously entered erroneous input. Execution will then resume with the corrected input.

INPUT Execution – Temporary Values

2011-02-25T21:51:00.006-05:00

The values parsed from the entered input to be assigned to the input variables need to be stored somewhere other than the evaluation stack. The logical place is another temporary input values stack. This stack will be allocated and initialized in the get input routine and will be removed by the final INPUT command code.

After a value is parsed by one of the input parse codes, it will be saved (pushed) to a temporary input values stack. At the last input parse code (with the 'End' sub-code), an index to be used to access this stack will be set to zero. As each value is assigned by an input assign code, this index will be incremented. In other words, this stack is being used as First-In-First-Out out list instead of a Last-In-First-Out standard stack.

The SimpleStack (to be renamed to just Stack since there is no other class named Stack), does not currently have this mechanism. This will be added when the run-time code implemented, but it is not needed currently for translating the INPUT command, so this addition will wait.

INPUT Execution Codes – Ending

2011-02-24T22:18:00.004-05:00

There will be the final INPUT command code at the end of the input statement. Besides cleaning up, the only action that needs to be performed is to advance to the next line if the 'Keep' sub-code is not set (this sub-code is set when there is a semicolon at the end of the INPUT statement). In summary, the “INPUT I%,A(I%)” statement will be translated as:

InputBegin InputParseInt InputParseDbl'End' I%<ref> InputAssignInt I% A(<ref> InputAssignDbl Input

As usual for RPN format, the command is at the end of the translation. What remains to be designed is where the temporary input values will be stored and how errors will be handled. An error can occur in the parsing codes. When an error occurs, execution must go back to the InputBegin, however, the prompt does not need to be output again, the cursor only needs to be positioned back where is was after the prompt was output (after the error is reported).

INPUT Execution Codes - Assigning

2011-02-23T21:15:00.002-05:00

The assigning of the input values must be done separately from the parsing of the values entered due to the two input rules. For the example, this is steps 7 through 11. Some of these steps are standard expression codes: push a reference to an integer variable (step 7), push a value to an integer variable (step 9), and calculating a reference to an array element by popping a integer subscript value and pushing the reference to the element (step 11).

There will be a code to assign an input value to an input variable for each data type: InputAssignInt, InputAssignDbl and InputAssignStr. For InputAssignStr, the InputParseStr will created a string from the input value. This string will be assigned to the string variable replacing the previous string value, which will be deleted. Therefore, there will be no need to deal with temporary strings.

The values being assigned will need to stored temporarily somewhere other than the evaluation stack.

INPUT Execution Codes - Parsing

2011-02-22T21:59:00.001-05:00

The parsing of the entered input values must be done separately from the assignment of the values entered to the input variables. This a departure from the design previously described and is due to the two input rules. For the example, this is steps 3 through 6. Notice that after a value is parsed (steps 3 and 5), a check is made for the next character. This character must be a comma after each value except for the last value where an end-of-line (no character) is expected.

There will be a code to parse an input value for each data type: InputParseInt, InputParseDbl and InputParseStr. An 'End' sub-code will be set on the last parse code. If this sub-code is not set, then the next character must be a comma, otherwise an end-of-line (no character) is expected.

The values that are parsed need to be put somewhere. If input values were pushed on to the evaluation stack, then when the references to the input variables are pushed, the input values would be down the stack and would not be easily accessible. Therefore, the evaluation stack can't be used.

INPUT Execution Codes - Prompting

2011-02-21T22:07:00.010-05:00

Execution and translation of the INPUT command was previously described mostly in posts on June 24, 2010, June 25, 2010 and June 27, 2010 . This now needs to be revised since the execution broke the two rules listed at the end on February 16, 2011. Steps 1 and 2 handle issuing the prompt and getting input from the user. This is almost the same as previously defined, which are these codes:

InputBegin – output default prompt and get input
InputBeginStr – output string prompt and get input
InputBeginTmp – output string prompt, get input and delete temporary string

During execution, the run-time handlers for each of these codes will call a common routine for getting input. This common get input routine can also handle outputting the prompt and can simply use the string on top of the evaluation stack. There will be an argument for whether to output the prompt string on top of the stack and InputBegin will set this argument to false.

There will be another argument for whether to output the default prompt where InputBegin will set this to true and the other two will set to true if the 'Question' sub-code is set (if the prompt string expression was followed by a comma instead of a semicolon).

For InputBeginTmp, the temporary string can be deleted upon returning from the get input routine since it will no longer be needed with the improvement in execution described in last Saturday's posts.

INPUT Command – Execution Procedure

2011-02-19T15:38:00.001-05:00

To determine how the INPUT command will be encoded into internal memory, the internal codes need to be arranged according to how the INPUT command will be executed including handling errors. Using the same “INPUT I%,A(I%)” example statement from before, here is the procedure (for the moment without taking into account error handling):

Issue prompt (for this statement, the default “? ” prompt)
Get the input from the user (allowing for editing like backspace, cursor left/right, insert/overwrite, delete, and terminated by enter)
Parse an integer value from the input for I% and save it (I% can't actually be assigned yet)
Check if the next character in the input is a comma
Parse a double value from the input for A(I%) and save it (A(I%) can't be assigned yet since I% hasn't been assigned)
Check if there are no more characters
Push reference of I% to evaluation stack
Assign saved integer value to reference on top of stack – I%
Push value of I% to evaluation stack (I% has now been assigned)
Calculate reference for array A by popping index from evaluation stack, push calculated reference, A(I%), to stack
Assign saved double value to reference on top of stack – A(%)
End of INPUT command, advance to next line of output

Codes need to be defined for these steps. Some of these steps are already standard token types, step 7 is I%<ref>, step 9 is I%, and step 10 is A(<ref>. The rest of the steps will be INPUT command related codes.

INPUT Command – Execution Improvement

2011-02-19T07:51:00.002-05:00

An improvement can be made to the execution of the INPUT command that will simplify the execution and will work nicer from the user standpoint. The traditional implementation of the INPUT command was designed to work with Teletypes. This has no place on modern computers. This has to do with what happens when the input entered by the user is invalid.

After surveying several BASIC implementations, all do a form of “Redo from start” on a new line and then reissuing the prompt on another new line for the input again and forcing the user to start their input from the beginning. Many of the implementations don't even check the presence for all the values requested or even accept string values and then simply set the non-entered or invalid values to zero. This is very sloppy programming and forces the programmer to do more work validating input.

A better alternative is to properly parse and validate the input entered. If something isn't valid, then output a temporary error message, point to where the error is (so the user doesn't have to guess what was wrong) and then allow the user to edit their input. Using extra output lines is unnecessary. The error message will be removed from the screen when INPUT is done. This also simplifies run-time in that the prompt string does not need to be saved until the end of the INPUT statement (deleting it if is a temporary string) since it now only needs to be output once.

There are other enhancements that can be made to the INPUT command, like having a fixed length input field with an optional template and accepting special exit keys (function keys, escape, page up/down, arrow up/down, etc) that the programmer can check for, but this will be later once the project is up and running. For now, the translation of current INPUT command needs to be implemented. As always, before the translation can be implemented, some idea how the INPUT command will work at run-time is needed to determine what tokens need to be put into the RPN output list.

Translator – INPUT Command – New Problem

2011-02-16T21:13:00.000-05:00

A few new ideas for the INPUT command have been developed that may simplify the execution of the INPUT command and work better than traditional implementation (more on this in a bit). Upon reviewing the previous posts on the INPUT command and reading the ANSI Standard document for the INPUT command again, a problem was discovered with the previous design. The problem can be shown with this INPUT statement:

INPUT I%,A(I%)

This is probably bad programming - if the value entered is outside the range of the array, an exception occurs. In any case, it is allowed. Both GW-Basic and QBASIC act as expected where the first value entered becomes the index of the element that is assigned the second value. However, the design laid before won't act as expected. Consider the planned translation for this statement:

InputGet I%<ref> InputInt I% A( InputDbl Input

This will not work, consider the items pushed to the evaluation stack upon reaching the Input code at the end:

Input information (prompt is specified, question flag, and location)
I%<ref>
Parsed integer value from input
Pointer to the integer assign routine
A(I%)<ref> (after I% pushed, popped and reference to A(I%) calculated)
Parsed double value from input
Pointer to the double assign routine

Notice in item 5 that the value of I% used to calculated the reference to A(I%) would not be the value of I% just parsed from the input because I% has not been assigned yet, which would occur after all of the input has been parsed and found to be valid. Making the implementation difficult are two rules that must be followed:

Can't put references to the evaluation stack until previous values have been assigned.
Can't assign any variables to the input values until the entire input has been parsed and validated.

Translator – Restructure Pre-Release

2011-02-14T20:53:00.002-05:00

Since the restructuring of the code was significant, it is probably a good time to make a pre-release of the code before commencing with the INPUT command. The file ibcp_0.1.15-pre-1-src.zip has been uploaded at Sourceforge IBCP Project along with the binary for the program. When uploading these files, it was discovered that the ibcp_0.1.14-src.zip was missing the complete set of test files, and so was uploaded again. The shell script used to automatically generate a release was using the wrong list to generate the source zip file. Now on to the INPUT command (finally)...

Translator – More Code Restructuring

2011-02-13T20:08:00.001-05:00

It turns out there were really no more issues, just some minor bugs that needed to be corrected. However, in looking at the error messages, the “expected statement” just didn't seem to be as clear as it could be and so was changed to “expected command” along with the name of the corresponding token status.

The size of the add token routine has been getting out of hand for some time. It's not really a good idea to let a function get so big because it becomes much harder to understand and maintain. So it is time to break it up into smaller functions. Generally, no variables need to be passed between these functions except for a reference to the token pointer and token status (usually the return value). The add token function was broken up into these functions:

process operand – handles operands when in operand status and token is not an operator

end expression error – gets the error when an expression is ended prematurely (an end expression token is received in operand state) and is only called when the state is not first operand

process unary operator – handles the checking if an operator token is a unary operator received in operand state including processing open parentheses tokens

process binary operator – handles tokens when in binary operator state

process_operator – empties higher precedence operators from hold stack adding them to the output list (functionality merged with the add operator function, which was removed since it only contains a few lines), then calls the token handler for the operator token

Now the code will be a bit more manageable as the INPUT and other commands are implemented.

Translator – PRINT and Assign Restructuring

2011-02-12T09:25:00.002-05:00

Only the PRINT and Assign commands commands are implemented so far, which were restructured where the related code for these commands contained in the comma and semicolon tokens was relocated into the command handlers. These token handlers will now call the command handlers just like the end-of-line token handler does. A switch was added to the command handlers on the code of the token passed in. Previously this token was only the end-of-line token.

The code from the end-of-line token handler that called a command handler was moved to a new call command handler routine that does as described in the last post except at the beginning, a check was added if the command stack is empty. This condition can occur in an assignment statement before a comma or equal token has been received. A null token status will be returned for the caller to report the appropriate error.

For the comma token handler, a null token status can only occur in expression only test mode because this mode does not put a command on the command stack. For the semicolon token handler, a null token status can occur if the semicolon is at the beginning of the line (where a command is expected) or after a variable (where a comma or equal is expected). The same conditions don't occur at a comma because there are specific checks depending on the current mode (command, assignment list or expression), where as the semicolon doesn't check the mode.

For the PRINT command handler, the code related code was moved moved from these token handlers into the appropriate cases of the switch. Since no other tokens are currently expected, the default case was set to return an unexpected token bug error.

The Assign Command handler only expects an end-of-line token, so for the default case, the assignment list mode returns a equal or comma expected error and expression mode returns an operator or end of statement expected error when expecting a binary operator or a numeric or string expression expected error when expecting an operand. Before returning the error, the command item token needed to be set to the token passed in to point the error to the token causing the error.

A few other issues were discovered after the code was restructured and tested...

Translator – Token/Command Handler Restructuring

2011-02-09T21:18:00.002-05:00

While looking over the design notes for the INPUT and INPUT PROMPT commands, besides implementing a new INPUT command handler, parts would also be implemented in the Comma and Semicolon token handlers just like the PRINT command. As further commands are implemented, these token handlers will increase in size. The code to handle a command will be located in several different routines. A better design would be if all of the command is processed in a single routine – the command handler.

The EOL token handler already calls the command handler, where each currently assumes that it is called for an EOL token. The code that would be in the comma and semicolon token handlers could be moved to the command handler, which would have a switch on the code of the token passed in to perform the appropriate action. Then the comma and semicolon token handlers would also call a command handler. If the command doesn't support the passed token, then it would return an error.

There is a sequence of code to call a command handler: get the command item from on top of the command stack, get the command token's command handler from the table, return an error if there is no handler in the table, call the command handler, if it returns an error then check if the command handler changed the token (for the error) and if is has delete the original token passed, and set the error token to return. This sequence will be put into a new call command handler routine so the code is not repeated in each token handler.

Translator – Expression Type Release

2011-02-08T20:50:00.001-05:00

The changes in getting the expression type code working correctly are completed. The plan was that the next official release would include support for INPUT, but that was before the issue of checking expression types came up when considering the reporting of errors with the string operand of the INPUT PROMPT command.

This is also the first release since development was moved from VIDE2 (using Insight, a GUI for gdb, for debugging) to NetBeans (which includes a GUI front end to gdb). A make file was developed so that NetBeans is not required to build the program for the project. The program is build by simply issuing the make command in the main IBCP directory.

All of the test programs were also modified to work using the same make file. The VIDE2 project files were removed. The test programs are built using the make tests command. A specific test can be built, for example the stack test program is built with the make test_stack command. The parser and table test programs were removed as there functionality is now built into the main IBCP program.

The ibcp_0.1.14-src.zip file has been uploaded at Sourceforge IBCP Project along with the binary for the program. This release contains the complete current source including the test programs and a single make file for building the main program and optionally the test programs. It is recommended that the source be extracted into a clean directory. Now the INPUT command can be implemented...

Translator – Assignments and INSTR

2011-02-07T21:27:00.000-05:00

While debugging assignments of temporary strings, specifically the assignment of a string that needs to be attached to the assignment operator, done stack empty errors were occurring. This was caused because it was trying to attach two strings, only one of which was on the done stack. Somehow the number of strings member in the table entry was set to two – it appeared to be counting both operands even though a check was put in not to count the first operand of an assignment operator (and was working).

The problem was finally determined to be caused by the INSTR table entries. The assignment operator table entries and the INSTR table entries were sharing the same expression information structure instance. When the assignment operator entry was initialized, it properly set the number of strings to one. But when it processed the INSTR table entries, it changed the number of strings to two, since it was the same instance. This was corrected by giving the assignment operator entries their own instances of the expression information structure.

This corrected the assignment operator problem, but the INSTR functions were giving an “expected operator or close parentheses” error at the second comma of the three argument form of INSTR. This occurred because the associated INSTR codes (Instr2T1, Instr2T2 and Instr2TT) did not have the multiple flag set. This was corrected by setting the multiple flag in the T1, T2 and TT forms of the Instr2 code entries, and the Instr3 associated code entries were placed after each of their corresponding Instr2 codes.

Translator – Assignment Table Entries

2011-02-06T21:06:00.000-05:00

The assignment operator table entries previously only had one operand, which was used for both the variable(s) being assigned and for the expression being assigned, since there both the same data type. This did not work for temporary strings because the variable operand needed to a string and operand being assigned needed to be a temporary string. So, the assignment operators were given two operands, the first for the variable(s) being assigned and the second for the operand of the assignment.

This change caused a problem with the main assign and assign list operators, which contained the list of all the related associated assignment operator codes (one for each data type). When find code is processing the assignment operand (the second operand) for double assignment, it does not need search for associated codes. To stop this, the index to the second set of associated codes, needed to be set to the number of associated codes, but this caused the Table initialization check to fail. So this initialization was modified to accept this condition.

The Table initialization was also modified to not count the first string operand for assignment operators (the entry's reference flag is set) because the string variable(s) being assigned are not attached to the assign string operator.

One strange problem remained with assignments, which involved the INSTR table entries...

Translator – Assignments of Temporary Strings

2011-02-06T18:12:00.002-05:00

A problem with assignments of temporary strings was that the assignment command handler was not performing all the actions it needed to. These additional tasks were found be comparing to the process final operand routine, which it turned out could be used with some special allowances for assignment operators.

The code in the assignment command handler was replaced with a call to the process final operand routine except for the check if the done stack is empty (needed otherwise the find code routine would flag it as a bug with its empty done stack check) and the clearing of the reference flag of the assignment operator token.

In the process final operand routine, a check was added if the token's table entry has the reference flag set (set only for assignment operators), then the assignment operator token will not be put onto the done stack, though it will still get operands attached (strings, but not temporary strings), and any parentheses in the first and last operands of the token being assigned are deleted.

There were additional problems with the Table entries for the assignment operators...

Translator – Temporary String Support - Debugging

2011-02-05T19:09:00.000-05:00

The Parser tests expected results needed to be updated because the EOL code had changed (due to extra table entries being added). Several of the Translator tests expected results needed to be updated for the new temporary string codes. For example, the convention for the operators are CatStr: +$, CatStrT1: +$1, CatStrT2: +$2, and CatStrTT: +$T. The convention for functions is a T added to the function name, for example Len: LEN( and LenTmp: LENT(. The INSTR functions with two string arguments use the convention T1, T2 and TT.

Some minor table entries problems were fixed, but one significant problem that needed correcting was in the number of strings member in the expression information structure in each table entry. This member is automatically calculated during table initialization, but was counting both string and temporary string operands. This field is used to determine how many strings need to be popped from the done stack and attached. Temporary strings are not left on the done stack, so the number of strings should have only included the number of strings only, not temporary strings.

There was one other issue with constant strings. Technically these are already known to be strings, unlike tokens with and without parentheses that may be variables/arrays or user functions. Therefore, constants don't need to be attached to operands. But there is no allowance in the current implementation for this, so to keep things simple, they will continue to be attached. The proper code is already set (for a string operand), and the Encoder will see that they are constants and leave the code as is. Normally, if the Encoder determines that an operand is user function, meaning it produces a temporary string, it will change the code appropriately to one that has a temporary string operand.

There is still a problem with assignments...

Translator – Full Temporary String Support

2011-02-05T09:28:00.000-05:00

The main work for adding full support for temporary string includes adding all the new codes for temporary strings and adding all the table entries for these new codes. Part of this was adding new operand arrays that contain the temporary string type(s), adding new associated arrays and increasing the maximum size of the associated array (the assign and assign list associated arrays contain five associated codes).

For example, previously there was the CatStr code (both operands are strings). This was expanded to the CatStrT1 (first operand is a temporary string), CatStrT2 (second operand is a temporary string), and CatStrTT (both operands are strings) codes. The is similarly for functions. For example, previously there was the Len code (argument is a string). This was expanded to the LenTmp (argument is a temporary string). The INSTR function that takes two arguments, the T1/T2/TT format was used.

The sub-string functions don't need to be expanded, because their operands are transferred to the next operator or function. If their operand is a string, then the next operator or function will be one that takes a string operand. If their operand is a temporary string, then the next operator or function will be one that takes a temporary string (that would need to be deleted at run-time when it is no longer needed). Therefore, there does not need to be separate codes (one that takes a string and a temporary string).

Translator – No Array Assignments

2011-02-03T21:23:00.000-05:00

Tokens with parentheses can be either an array or a user function, but the Translator can't determine which since it doesn't have access to the Dictionary (the repository for all elements of the program including variables, arrays, defined functions, user functions, constants, etc. with information about each). The Dictionary will be developed along side the Encoder. For now, the Translator will simply translate from the input code entered to reverse polish notation.

It was thought that since a token with parentheses on the left side of an assignment can only be an array, that the Translator could assume that it was an array and handle the subscripts accordingly. Currently the Translator just attaches all the operands (subscripts) to these tokens. The Encoder (by looking up the identifier in the Dictionary to determine what it is) will handle arrays and user functions. For arrays, all the subscripts need to be numerical (doubles would get a CvtInt added) and for functions, the arguments must match the function definition.

There is one additional issue with arrays through that the Encoder will handle, which is to check if the number of subscripts is correct for the array. This information will be in the Dictionary, which the Translator does not have access to. If arrays were assumed, the current attachment of subscripts, including the number of subscripts attached, would not include. The Encoder would then not know how many subscripts were found (which needs to be checked) unless this number was somehow passed through. Therefore, the current mechanism will be left in place. The only thing left to the expression type processing is to add full support for temporary strings...

Translator – Array Assignments

2011-02-02T21:23:00.002-05:00

There's been a change of plan for the Translator handling arrays on the left side of assignment statements, which will be explained in the next post. What was needed to handle array subscripts was a stripped down routine similar to the code routine, to handle integers and doubles (for which a hidden CvtInt code would be added). As the customized new function was being implemented, some unnecessary code was found.

In the find code routine, when the token with an error was not a reference, there was a check if the last token added to the output was different from the token on the done stack, and if it was, it was assumed that the last token was a sub-string function (which was not put on the done stack because its operand was left on the done stack). So this sub-string token was returned for the error instead of the token on top of the done stack.

In addition, if the last token was not the expected sub-string function, a bug error was returned since it was assumed to be invalid condition. However, this is a valid condition if a dummy close parentheses token was added to the output, which would then be the last token added to the output.

With the first and last operand implementation, these checks are no longer necessary, since the for sub-string functions, the first operand left on the done stack acquires the sub-string function token as its first operand and this first operand token is returned as the location of the error. Therefore, these extra checks were removed.

Translator – Correcting More Token Leaks

2011-02-01T20:02:00.003-05:00

There were a few more token leaks that needed to corrected, one with the first and last operand changes and rest with the PRINT command. Details of the rest of the token leaks found and corrected are after the continue. All tokens leaks have now been corrected, time to handle array assignments...

When processing the final operand for operators and internal functions, the setting of the first and last operands was handled for non-sub-string internal functions and operators. For non-reference sub-string functions, the sub-string function doesn't get pushed to the done stack, but the first operand on the done stack (the operand of the sub-string function) is set to the sub-string function and the last operand is set to the closing parentheses.

However, reference sub-string functions are pushed to the done stack, but the first and last operands were never set and so were left set to the last argument's first and last operands. The first operand should have been cleared (to be the sub-string function like other internal functions) and the last operand should have been set to the closing parentheses token passed in. Also, no operands should be attached to the sub-string function (it is already known to be a string variable).

In the print command handler, at the end of the statement, the print token is only added to the output to cause the cursor to be moved to the next line (the statement does not end with a semicolon, comma or print function, which keep the cursor on the same line). In this case, the print token was not used but it was not being deleted.

When processing the final operand for print functions (SPC and TAB), which don't get pushed to the done stack, the closing parentheses token of the function was not being deleted.

In the semicolon token handler, when the semicolon sub-code was set, the semicolon token was not being deleted.

Translator – Correcting Token Leaks

2011-01-30T19:37:00.002-05:00

The crash on Translator test 12 was caused because the token leaks were only being output after a successful translation but not when an error was reported. After a number of errors, the token leaks built up and when they were finally output, their information didn't line up with the current input line causing a crash when accessing outside of a buffer. Once that problem was fixed, the token memory leaks were fixed one by one.

For one of the leaks, an incorrect fix was applied causing multiple token deletes. For some reason these tokens were not output. A segmentation fault occurred when attempting to debug this problem before it even got to the output code when it was attempting to delete the string in the token a second time. Setting the string pointer to NULL didn't help – apparently the memory was already being used for something else.

Apparently gdb (under NetBeans) catches these segmentation faults where the program running by itself does not. In any case, at least the code was at least detecting the extra delete condition. Details of the memory leaks found and corrected so far after the continue. A common theme is that the token leaks occur when something is popped from the stack and not put anywhere because any tokens in any of the stacks, the pending parentheses token or the output list get deleted by the Translator's cleanup routine.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
In the comma token handler, the comma token was not deleted when an argument to an internal function was the wrong expression type.

In the closing parentheses token handler, the top token was popped from the done stack, but if error occurs, the error is pointing to the close parentheses token and the top token was forgotten. The code was changed to only pop the top token when no error occurs.

In the closing parentheses token handler, the closing parentheses token was not deleted when an argument to an internal function was the wrong expression type.

In the assignment command handler, if the operand being assigned was the wrong expression type, the first and last operands from the item on top of the done stack are used to report the error. However, it didn't call the delete open and close parentheses routines for the first and last operands if there was no expression type error – the first and last operands were no longer needed.

When checking the item on top of the done stack in an assignment list, if the item was an array or sub-string, it had a closing parentheses set as its last operand, which was not getting deleted.

Before a new operator is processed, any higher precedence operators on the hold stack are popped first to added to the output list. If an error occurs (like the operand for the higher precedence operator is the wrong expression type), both the operator from the hold stack and the new operator tokens were not deleted - the token returned was the operand with the error.

When a range of tokens is reported for an error, the first operand token is returned with its length modified to include all the tokens through the last operand token. However, the last operand was not deleted if it was a closing parentheses when the operand from the done stack was popped.

When processing the first operand of an operator, if the first operand was the wrong expression type, then the first operand of the operator is returned for the error, but the original operator token was not deleted.

When an error occurs, the caller of the Translator (the test code) was only deleting the token passed to the Translator if it was the original token passed, otherwise the token was assumed to be a stack or output list and would be deleted by the Translator clean up routine. An open parentheses token was an exception because these are temporarily used as first operand tokens, so if this was the token with the error, the open parentheses token was not deleted. The delete open parentheses routine was used since it has all the proper checks to delete this token (this function was made public to be accessible).

In the end-of-line token handler, the command token was popped from the command stack before the check if the command handler called returned an error. When an error occurred, the command token was not deleted. The command token pop was moved to after the error check so that the command token will still be on the command when the clean up routine is called.

In the end-of-line token handler, when the command handler returned an error, if the token returned is different from the end-of-line token, the end-of-line token is not deleted. A check was added that if the token returned is a different toke, the end-of-line token is deleted.

Token Leaks – Implementation

2011-01-29T23:18:00.000-05:00

To implement token leak detection, a token pointer list was added to the Token class as a static member and a token list element pointer was added as a regular member. The new and delete operators were overloaded in the Token class to be able to maintain this token list.

The overloaded new operator function first allocates an array of bytes for the size argument, which is type cast to a Token pointer. The token pointer is appended to the static token pointer list and the element pointer returned from the list append function is put into the element pointer member of the token. The value of the token pointer is returned.

The overloaded delete operator function type casts the void pointer to a token pointer. If the element pointer is NULL, then this token has already been deleted. Otherwise the element pointer in the token is used to remove the token pointer from the token pointer list. The element pointer of the token is set to NULL to indicate that it has been deleted (to detect multiple deletes of the same token). The memory used by the token is then deleted.

This additional error that can occur when a token being deleted more than once. To also catch these errors, when a token is being deleted, if its element pointer is NULL, then a copy of the token will be added to a new delete list, which will be a list of tokens. A copy of the token needs to be made because the memory has already been deleted and could be reused by another allocation. It's also possible that the memory may have already been reused and the token copy will be garbage, but at least it will be known there is an extra delete somewhere.

Finally, the test code was modified to output any tokens that have not been deleted once the RPN output list has been output and cleaned up (all of its tokens deleted). The tokens in the delete list are also output. Once the code was working and running, many token leaks were found (in six of the Translator tests), and Translator test 12 (more error tests) was crashing. Time to fix some problems...

List Class Update

2011-01-29T17:38:00.000-05:00

The List class is currently keeping track of the beginning and end of the linked list with a master element. The master element is created when the list is created. A list element contains a previous element and next element link pointers along with the value of the element. The amount of memory allocated for the master element is less the element value using a strange syntax for subtracting the size of the value from the total size of the element – not ideal.

When a list is created, the links of the master element are set to point to the master element itself. The list is empty when the master element's next and previous pointers are the same as the master element pointer. When linking through the list, the first element is the master element's next pointer. The end of the list is detected when the master element is reached when linking to an element's next pointer.

In considering a design for the token linked list, it was realized that the master element is not really necessary. All that is needed is a pointer to the first element in the list. Or, for a first-in-last-out list, a pointer to the last element. The last pointer method will be used and the variable will be named tail. The list is linked circularly meaning that the last element's next pointer points to the first element in the list and the first element's previous pointer points to the last element. If the list is empty, then tail will be set to NULL.

Using the single tail pointer, the first element in the list is now obtained using tail->next assuming the list is not empty, otherwise NULL will be returned. The last element is now obtained simply by using tail. When appending items to end of the list, the new element is linked into the list and tail is set to the new element. When removing the last element from the list, a check is needed to see if the list is empty after removing the item.

Sourceforge Uploads Now Working

2011-01-29T08:55:00.000-05:00

Sourceforge is accepting uploads again and the files for the latest pre-release are now there.

Token Leaks – Initial Design Thoughts

2011-01-28T23:59:00.000-05:00

When considering a design to keep track of tokens created, there needed to be an easy way to remove the token once it has been deleted. Tokens left in the list would be memory leaks. The first idea was a simple linked list where each element consisted of a previous link, a next link, a pointer to the token and some type of identifier of where in the code that the Token was allocated. A pointer to the list element would be added to the Token class.

When a token is created, it would be added to the list and a pointer to the list element would be put into the token. When a token is deleted, it would use the element pointer to delete the item from the list. For tokens still in the list at the end, the identifier would indicate where the token was created, though the contents of the token itself would probably be sufficient in determining why it was not deleted.

At first, a special token list class was considered. It would only need a pointer to the first element in the list. Some code on how this would work (adding and deleted elements) was sketched out. But wait, there was already a List class, and then it was realized that the List class doesn't need a master element to keep track of the first and last elements in the list. Only a pointer to the first (or last) element is needed...

Memory Leak Detection

2011-01-27T22:21:00.000-05:00

Memory leaks have been a continual problem. Frequently another one is discovered like the recent pending parentheses issue. Some short of memory leak detection is needed. In C++, the new and delete operators can be overloaded so that a custom memory management can be used. While this is a possible solution, a memory manager with leak detection can be very involved. Something simpler is desired.

Analyzing the current source code, looking for all the uses of new operators currently used, here's what was found:

Tokens – allocated in a number of places; these have been where most of the memory leaks have been found

Strings – for now these are allocated when creating tokens, so if the tokens get deleted these will get deleted (in the Token destructor)

Stacks – the arrays in the stacks upon first creation and when expanded; the arrays will be deleted when the stacks go out of scope (as part of the destructors of the classes that use them)

Lists – the elements of the list, which are deleted when the lists go out of scope (as part of the destructors of the classes that use them)

Translator Output list – this is a one time allocation in the Translator when a new translation is initiated; it is handed over to the caller when getting the translated results; so it's up to the caller to clean this up (currently handled in the test code)

RpnItems – everyone of these that are allocated is appended to the Translator's output list, so these will be handled when this list is cleaned up

Table – one time allocation in the main function, which is not important as there is only one, but a delete was added at the end of the main function

Error Lists – these are allocated as part of the Token and Table initialization when an error is detected, but were never deleted, which is not important as these were only created when there was an error (and the program exits), but deletes were added

So it looks like Tokens are the only specific area where there is currently a problem with memory leaks. Next, a simple solution for detecting token leaks...

Sourceforge Not Accepting New Uploads

2011-01-27T05:50:00.001-05:00

Sourceforge does not appear to be accepting new uploads at the moment. Last night it just seemed to be taking a long time to be distributed to the mirrors, but this morning the same message appeared. This may explain the problem. For some reason it did except the new release notes. For now the previously posted link for the latest pre-release does not work.

Translator – First/Last Operand – Pre-Release

2011-01-26T21:56:00.008-05:00

The first and last operand changes are now working and all tests are working correctly except for three statements in Translator test 12 (more error tests). The expected results for these statements were changed in anticipation of having the Translator assume that tokens with parentheses on the left side of an assignment are arrays since functions with arguments will never be there. However, one little issue remains. Consider these statements:

Z$ = A$ + (B + C)
Z = A + (B$ + C$)

Because the open and close parentheses are now attached to operator tokens, the second + in the two statements above, when the error is detected (at this +), the whole parenthetical expression including the parentheses are reported instead of the expression within the parentheses. This new behavior is reasonable so it will be left as is for now.

This is a good time to make another pre-release before continuing. The file ibcp_0.1.14-pre-4-src.zip has been uploaded at Sourceforge IBCP Project along with the binary for the program. Now, before continuing on, it's time to do something about detecting memory leaks...

Translator – First/Last Operand – More on Parentheses

2011-01-25T22:13:00.000-05:00

An old undiscovered bug was found where the pending parentheses token was not deleted when the previous token added to the output list's parentheses sub-code was set when it was determined that the parentheses was not necessary (the parentheses sub-code allows the parentheses to be reproduced). This again emphasizes the need for memory leak detection (more on this soon).

There was a conflict between a closing parentheses token being used as a pending parentheses and as a last operand. It's possible that a pending parentheses will become a token added to the output list (occurs when there are two sets of unnecessary parentheses), so when this token is no longer needed as a last operand, it can't be deleted. The solution was for two new sub-codes, one where the closing parentheses token is being used for a pending parentheses (Used) and one where it is being used for a last operand (Last).

When the pending parentheses is set, the token's Used sub-code is set. When a closing parentheses token is set as a last operand, the token's Last sub-code is set. When the pending parentheses token is no longer needed, if its Last sub-code is set, the Used sub-code will be cleared, otherwise the token will be deleted. If the pending parentheses is added to the output, its Used sub-code will be left set to prevent it from getting deleted. Similarly, when a closing parentheses token is no longer needed as a last operand, if its Used sub-code is set, its Last sub-code will be cleared, otherwise the token will be deleted.

In the closing parentheses handler, when a closing parentheses is processed for a parenthetical expression, the opening parentheses (previously deleted) will now be set as the first operand and the closing parentheses (also set to the pending parentheses) will not be set as the last operand of the token on top of the done stack. The closing parentheses token's Last and Used sub-codes will be set.

Translator – First/Last Operand – Parentheses

2011-01-22T22:11:00.000-05:00

Previously it was thought that parentheses could be ignored for the first and last operands because parentheses are always valid in a numeric expression or a string expression, though parentheses would be strange in string expression since there is currently only one string operator (concatenate), but still valid. However, consider this statement:

Z$ = A$ + -(B+C)

The entire -(B+C) sub-expression should be reported as an “expecting string expression” error. However, only the B+C is being reported before the unary operator fix and -(B+C after this fix. In both cases, the C was the last operand of the negate (where the expression type was invalid). The parentheses were removed. Had the statement been:

Z$ = A$ + INT(B+C)

The error would have been correctly reported since the closing parentheses is being attached to the last operand of the function. Therefore, both the open and closing parentheses also need to be carried through the operands.

Currently, when a closing parentheses is processed, the open parentheses token is deleted and the closing parentheses is temporarily saved in the pending parentheses variable, which is checked later to see if it was necessary or not (the parentheses sub-code of the previous operand is set if the parentheses were unnecessary). To be continued...

Translator – First/Last Operand – Unary Operators

2011-01-22T08:12:00.000-05:00

Expression type errors involving unary operators were not pointing to the correct tokens. This was caused because no operands were being attached to unary operators pushed to the done stack. Consider this statement:

Z$ = A$ + -B

The -B tokens should be reported as an “expecting string expression” error. However, only the B token was being reported because the first and last operand of the negate operator was being set to the first and last operand of its operand from the done stack, which was the B token.

The solution was to only attach the last operand from the operand popped from the done stack and leaving the first operand set to nothing of the unary operator token, which will cause the code to use the negate token itself as the first operand.

Translator – First/Last Operand – Initial Debugging

2011-01-21T21:37:00.002-05:00

Initially, all the Translator tests were failing. The source of the crash was the new delete close parentheses routine, which also needed to check if the token type has a table entry. The crash occurred because when token didn't have a table entry as it was trying to access the table entry to check for a close parentheses code.

Next, a very old bug that went undiscovered was occurring in the expression only test mode used for the first four Translator tests. The problem was that the final result was left on the done stack and the Null token was left on the hold stack. The crash occurred on the first error test when the cleanup routine was called and it tried checking for a close parentheses in the last operand – the token if was checking no longer existed. The End-of-line handler was modified to cleanup the stacks in expression mode and to check to make sure that there is only the result item on the done stack.

A series of problems were then discovered relating to unary operators and parenthetical expressions. Also, a realization was made that the first and last operands need to be saved for operands of arrays and used functions so that the Encoder can report errors correctly for array subscripts and function arguments. More on these problems to follow...

Translator – First/Last Operand Implementation

2011-01-16T10:13:00.000-05:00

As the hold stack was being changed from holding just a token pointer to a structure containing a token, a first operand token and last operand token pointers, it was noticed that the end of the Operator and Equal (for the equality operator) token handlers contained almost identical code, both needing modified for the hold stack item change. The code was moved and replaced with a call to a new process first operand routine.

A second token pointer argument was add to the process final operand routine, which is called to add a token to the output list by the add operator routine (to add an operator from the hold stack), add print code routine (to add a hidden print code for a specific data type), and the close parentheses token handler (to add a function or array from the hold stack). This second token pointer will contain the first operand token (from the add operator routine), nothing (from the add print code routine), or the close parentheses token (from the close parentheses token handler).

The find code routine was modified to return the first and last operand token pointers from the operand on the done stack that is being checked. These arguments are pointers to token pointers with a null pointer default value – a caller only passes these when it needs them. The find code routine contains local pointer variables if the caller doesn't pass pointers since first and last operand tokens will be needed if an error occurs.

The line in the close parentheses token handler that deletes the close parentheses token was removed (since this token is now being passed to the process final operand routine to be stored temporarily as the last token pointer for arrays and functions). Several locations need to delete this token when it is no longer needed. This procedure first needs to check if the last operand token pointer is set, if the token is a close parentheses token, and then needs to delete the token. A new delete close parentheses inline function was implemented that performs this procedure.

Finally, a through inline function was implemented in the Token class that takes a second token argument and calculates the total length for a token that includes up through to this second token as described here and returns a pointer to the first token so that it can be conveniently assigned to the token being returned. Now for debugging and testing...

Translator – Sub-String Functions

2011-01-15T08:32:00.000-05:00

Sub-string functions add a wrinkle to storing the first and last token because these are not actually stored on the done stack (their string operand is kept on the stack). Consider this statement:

Z$= LEFT$(A$, 1) + B$ > C$

When the +$ token is processed, the operand on the stack will be A$ left there by the LEFT$( function, which will be considered the first operand of the +$ token. This would then be passed on as the first operand of the >$ token. When the error is reported, the range will incorrectly be from A$ to C$ instead of LEFT$( to C$.

To resolve this, when a sub-string function is processed (popped from the hold stack and added to the output list, but not put on the done stack where its string operand remains), the sub-string token will be put into its operand's first token on the done stack. If this token already has a first token, it will be overwritten.

In the statement above, the first operand of A$ will be set to the LEFT$( token. The +$ token will then inherit the LEFT$( as its first token, followed by the >$ token. When the error is finally reported, the range of tokens will correctly be from LEFT$( to C$. Now on to the implementation of the first and last operand...

Translator – Functions and Arrays

2011-01-14T22:47:00.007-05:00

Reporting range of tokens (sub-expressions) gets a little more involved when arrays and functions (internal or user) are in the expression. Consider this statements:

Z$ = A$(1) + B$(2) > C$(3)

The range of tokens would be A$( to C$( and reporting only up to C$( would not be correct since the 3 and ) tokens should be included. Therefore the closing parentheses of an array or function should be assigned the last token for an array or function token. This implies that closing parentheses tokens cannot not be immediately deleted.

When the array or function token is popped from the done stack, if the last token is a closing parentheses, then it will be deleted, unless it is being assigned as the last operand to an operator token as would happen in the statement above, the closing parentheses to C$( would be assigned to the >$ token as its last token. When the error gets reported, the range would be from the A$( token and the final ) token of the C$(.

There is no issue with regular parentheses expression as these are valid in any expression, therefore, if the statement above included parentheses around the expression, the error would be reported as (in other words, the parentheses are valid, but not the expression inside):

Z$ = (A$(1) + B$(2) > C$(3))
^^^^^^^^^^^^^^^^^^^^^-- expected string expression

This attachment of closing parentheses to tokens will take place in the closing parentheses token handler and will only apply to arrays and functions, not simple parentheses expressions. Also worth noting is that tokens (arrays and functions) will only have a last token assigned, the first token will be blank. In this case the token itself will be considered the first token. Finally, sub-string functions add another wrinkle to this scheme...

Translator – First and Last Operand

2011-01-13T22:07:00.004-05:00

When an error occurs, the inclusive range between the first and last tokens of the offending expression will be reported. Take the previous example statement Z$=A$+B$>C$. The >$ token will be on top of the done stack when the expression type error is detected (the >$ operator returns an integer and cannot be assigned to a string variable). The entire expression from A$ to C$ should be highlighted.

This will be accomplished by saving the first and last operand for each operator as the expression is processed. When an expression type error is detected, these operands will be available. When assigning the first operand to an operator, if the operand on done stack has a first operand assigned to it, then that operand will be used, otherwise the operand token on the done stack will be assigned. The same for the last operand.

In the statement above, the +$ operator will have A$ for its first operand (since A$ does not have a first operand assigned to it) and B$ for its last operand (B$ does not have a last operand). At the >$ operator, its first operand will be assigned to A$ (which is the +$ operand's first operand) and C$ for its last operand (C$ does not have a last operand).

The first operand is no longer saved on the done stack except for string operands (a specific type operator has already been selected), the first operand needs to be attached to the token on the hold stack where the operator is residing until put into the output list. Therefore, the hold stack needs to hold structures containing a token pointer and a first operand token pointer instead of just a token pointer – a new hold stack item will be defined to hold these.

The done stack currently holds output list element pointers (the output list consists of RPN items each containing a token pointer, number of operands and attached operand array, if any). The done stack will also need to hold the first and last operand token pointers. Therefore, the done stack needs to hold structures containing an output list element item pointer along with the first and last token pointers – a new done stack item will be defined to hold these.

Parser – Token Length

2011-01-12T21:30:00.007-05:00

In researching for the last post, some issues were found in the Parser with respect to the token length. The token class contained a length member, but it was contained in the union along with the double value and integer values members used for constants. The thinking was that length would never be used with number constants; the length would be obtained from the string of the constant (which is being kept so that the constant could be reproduced exactly as entered).

The test routine contained code for determining the length of string constants. This required adding two for the surrounding double quotes (which are not stored) and adding one for each double quote contained in the constant (which is entered with two double quotes). It turned out that the length of the original string constant, including the surrounding double quotes was already being stored in the length member of the token. The code in the test routine was redundant and unnecessary (and was removed).

Generally the length member was being used for the length of the token, but it wasn't being set for identifiers (variables, arrays, and user functions) and number constants where the length could be obtained from the string member.

It will be easier if the length member always contained the length of the token so it will not required to obtain the length from the string member for certain token types. The length member was moved out of the union so it can also be used for number constants, and in the Parser, the length member was set for identifiers and number constants so now all token types are covered.

Translator – Errors with a Range of Tokens

2011-01-11T22:54:00.001-05:00

Currently when the Translator detects an error, the error code is returned along with the token where the error was detected. The caller uses the column and length values in the token to point to the token. (Later the token will be highlighted in a different color.)

Now it is desired that a range of tokens be highlighted to show an entire sub-expression that contains an error. While it is possible to modify all the functions within the Translator for a second return token, there is an easier solution.

The column and length members of the token are used to output the error – where the token starts and how long it is. The Translator, upon detecting an error with a range of tokens, only needs to modify the length token where the beginning of the error is detected, to include all the tokens. The Translator can do this by using the column of the first token, the column of the last token and the length of the last token. The equation would be:

first->length = last->column – first->column + last->length

So when an error is detected and a range of tokens are involved, the length of the token being returned will be set using the equation above.

Translator – First Operand (Updated Design)

2011-01-10T21:20:00.001-05:00

Upon considering the first operand design (see posts starting on July 8, 2010), There still be some confusion with the reported error message and the token at which it is pointing. Consider this simple statement (and the error produced currently):

Z$ = A$ + B$ > C$
^-- expected string expression

This does not make sense, because first, the integer expression is valid. The error occurs because this integer expression cannot be assigned to a string variable. And it could be confusing because a string expression cannot be placed where the greater than operator is. Alternatively it could indicate that a string operator is expected at the greater than (later why this was ruled out). The previously proposed first operand design would produce this error:

Z$ = A$ + B$ > C$
^^-- expected string expression

The error is now pointing to the beginning of the integer expression. But, one could say wait a minute, A$+B$ is a string expression, why is this an error? A better solution would produce this error:

Z$ = A$ + B$ > C$
^^^^^^^^^^^^-- expected string expression

This is much better as it highlights the entire expression, an integer expression, and says that a string expression is expected here. This can be accomplished by, in addition to keeping track of the first operand (the A$), also keeping track of the last operand (the C$). The error would point to all tokens between the first and the last of the invalid expression. Now, how can these two tokens be passed back where currently only one token is being returned when an error occurs? Next, the solution to this...

Translator – Expression Type Pre-Release

2011-01-09T16:55:00.004-05:00

There are two main problems with the Twelfth (more error statements) test, namely the handling of the first operand of operators first mentioned on July 8, 2010, and recognizing tokens with parentheses before assignment operators as arrays only (implying the operands are numeric subscripts).

All of the other tests are succeeding, so it's probably a good idea to kick another pre-release first. This is the first release since development was switched to NetBeans, so there are some changes in the released files. NetBeans is not required for building the project. There is now full Makefile for building the program. NetBeans was used to initially generate this file, but it was modified to simplify it. The VIDE2 project file ibcp.vpj will no longer be part of the source files.

Also no longer part of the source files is codes.txt and test_codes.h. Both of these files are now be generated by the new Makefile using the codes.awk and test_codes.awk awk scripts. The MinGW and MSYS packages both need to be installed in order to use the Makefile. The program build has only been tested with GCC 4.4.0 version for MinGW. Also, the awk program is needed for the awk scripts (I don't remember if this is part of MSYS or whether it needs to be installed separately).

The Makefile is currently configured for building the release program with no symbols or debugging information. Debugging can be enabled by changing BASICOPTS from "-O -s" to "-g" in the Makefile. The file ibcp_0.1.14-pre-3-src.zip has been uploaded at Sourceforge IBCP Project along with the binary for the program. Now on to implementing the first operand design...

Translator – More Expression Type Debugging

2011-01-08T18:22:00.001-05:00

The next problem was occurring when there was an unexpected operand when an operator, comma or closing parentheses was expected (for example: Z$=MID$(A$,B C). This was only occurring for functions with multiple arguments (ASC, MID$ and INSTR) when at the last argument of the shorter form. The “expected operator or closing parentheses” error was produced, but a comma could follow for the optional argument.

A checked was added to the parentheses status routine when at the last argument. If the function has the Multiple flag set, indicating there is another form with more arguments, then the “expected operator, comma or closing parentheses” error is produced instead.

One problem remained, the statement Z$=MID$(A$,B, produced an “expected string expression” at the end of the line where it should have been an “expected numeric expression” error. The problem also occurred on the last optional argument of the INSTR and ASC functions. The problem was caused because the top of the count stack contained the index to the first form of these functions instead of to the second form with the optional argument.

When the code went to get the argument's data type, it ran of the end off the operand data type array incorrectly picking up a string data type. The coding error was on the line that incremented the index of the token on the hold stack to the next form of the function and assigned it to the index on top of the count stack. A post-increment C operator was used instead of the pre-increment operator and the index on the count stack was not actually changed.

Now all the test statements in the Eleventh Translator test are working (and on expected error needed to be corrected on the Third test because of the last fixed – it was missed previously). On to the Twelfth (more error statements) test...

Translator – More Expression Type Debugging

2011-01-08T10:46:00.001-05:00

Now that the data type was moved back from the expression information structure back to the main table entry structure, so that it is available for commands as well as operators and functions, and the code was working again the way it was, it was back to debugging the Eleventh (error tests) Translator test (after committing the working code to CVS) to continue fine tuning the error messages produced.

The next problem to tackle was the generic “expected expression” error being produced when a binary operator occurred in an expression when only a unary operator was expected. This error needed to be changed to the more specific “expected <numeric/string> expression” errors. This was accomplish by calling the new get expression data type routine to get the current data type and producing the appropriate error.

Next there were several error test statements that contained an unexpected comma in an expression, which were producing “expected operator or end-of-statement” errors. This was the correct for some statements (for example: Z=A+B, and Z=A,B=1 and Z=A,), but, not for others (for example: Z=, and Z,Y=, and Z$=,) where the correct error should have been “expected <numeric/string> expression” errors.

These errors are being handled in the comma token handler. These conditions can be determined by looking at the done stack. If the done stack is not empty, the expression up to the comma is valid, so expecting another operator or end-of-statement is appropriate. If the done stack is empty, then there is no valid expression yet for the assignment, so expecting an expression is appropriate. Again, to determine the type of expression, the new get expression data type routine is used.

Translator – Command Expression Type

2011-01-06T21:39:00.003-05:00

There was a problem getting the data type for the token on top of the command stack. There is no issue with assignment tokens as they contain a data type (they are just assignment operators put on the command stack).

A while back, a new field was added to the Table entries for the expression type with possible values of None, Numeric, String and Any to be used with command entries. The idea at the time was that this would be used for the expression type checking (for the generation of the correct error messages). However, this field was not being used (the original expression type code added at the time was abandoned because it had too many problems – it is currently commented).

This first attempt at the current problem was to use this expression type and convert it into a data type (None/Any to None, Numeric to Double, and String to String). The only place this was used was in the new get expression data type routine for commands. But this wasn't working for assignment commands (which were operators and their entries didn't have the expression type set – the data type in the expression information structure had the return value).

So it was concluded that the expression type field should be removed and the data type moved back from the expression information structure into the main entry structure (where it was originally). When the Parser finds and creates a command token, it sets the data type of the token to data type from the table entry. Currently for all tokens, the Parser was getting the data type from the expression information structure or none if there was no structure (as is the case for commands).

Translator – Expression Type Implementation

2011-01-05T21:26:00.001-05:00

A new routine was implemented to handle getting the current data type of the expression processed so far. This routine is called when the unexpected end of expression occurs to get the data type so that the correct “expected ...” error can be reported, and is also called before pushing an opening parentheses token to the hold stack to set its data type (so that it can be carried through the rest of the expression).

Once the routine was working, it was simplified. First, the token on top of the hold stack is checked for an operator before looking for the other conditions (parentheses, internal functions, and tokens with parentheses). This reduced the number of tests overall. An empty hold stack causes a bug error since at least the Null token should be on the stack.

If the top hold stack token is the Null token, then the data type is obtained from the token on top of the command stack. This caused another problem with commands (more on this in the next post). If the top token is an opening parentheses, then its data type is used (which could be none). Otherwise the data type of the operator's operand is used (first operand for a unary operator, second operand for binary operator).

If the top token on the hold stack is not an operator, then the top of the count stack is used to determine where to get the data type. If the count stack is empty, a bug error is returned since this situation should not happen. If the top is an internal function, then the data type of the current operand of the function is used. If the top is a parentheses, a bug error is returned since this situation should also not happen (parentheses token should have been on top of the hold stack). Otherwise the top contains a token with parentheses (an array or non-internal function) where the data type cannot be determined, so is set to None.

Translator - Expression Type With Parentheses

2011-01-01T19:15:00.001-05:00

Determining the expression type to report the appropriate error within a parenthetical expression without another operator on the hold stack involves a little more work. To accomplish this, the expression type needs to carried from before the opening parentheses.

To prevent having to scan backwards into the expression, when the opening parentheses is pushed on to the hold stack, the opening parentheses token will be assigned a data type, if possible, otherwise the data type will be left at the default data type of none (which will cause the generic “expected expression” error if an error occurs).

Here are several possibility of tokens that will be on the hold stack when an opening parentheses token is about to be pushed onto the hold stack and how to determine the data type to put into the parentheses token:

Operator – get the expected operand data type of operator
Internal function – get the expected argument operand data type of function
Token with parentheses – can not determine data type
Opening parentheses – inherit this token's data type
Null – Look at data type for item on top of command stack

The first four possibilities are the same as before, so a common function will probably be implemented to handle determining the data type for the opening parentheses token. The last possibilities will require looking for the expected data type for the command on the command stack. Here are some examples of commands and expected data types for expressions:

LET A = ( - assign token on command stack will have data type of variable
PRINT ( - PRINT can take any type of expression, therefore none
INPUT PROMT ( - INPUT PROMPT expects string expression
IF ( - IF expects numerical expression

The IF statement has not been defined as of yet, but most likely will expect an integer expression (the result of all logical operators). However, a double expression will be accepted either with a different IF code expecting a double expression or the more likely, a hidden integer conversion code will be inserted. The IF will be defined later. Now on to the implementation of this...

Translator - Expression Type Determination

2011-01-01T11:17:00.001-05:00

In the Eleventh (error statements) tests, the are many statements for testing the correct error message in a line that contains something unexpected, like either an unexpected comma or end-of-line. One of these was just corrected (see last post) – the situation when the Translator is not in a parenthetical expression including inside array subscripts or function arguments. This was detected when the count stack is empty.

The situation when in an internal function has also been handled when the operand processing was implemented. The code simply checks for the expected operand type for the current operand. An internal function is detected when the number of expected operands for the item on top of the count stack is non-zero. The current operand number is in the number of operands in top item on the count stack.

For the remaining situation, the code was simply returning an “expected expression” error, since it was assumed that the type of expression could not be determined. Turns out, this is not entirely true. Two sub-conditions to this situation are inside a parentheses and inside an array or non-internal function (a token with a parentheses). Each of these can be further sub-divided into whether this is an operator or not. Here are examples of each of these four conditions (terminated with an unexpected end-of-line):

Z = ( in parentheses, no operator
Z = (A+ in Parentheses, with operator
Z = A( in token with parentheses, no operator
Z = A(B+ in token with parentheses, with operator

In the third case, the expected expression type can't be determined because the Translator does not know if A is an array (expression must be numeric for a subscript) or a function (expression can be any type), therefore the “expected expression” error is appropriate. In the other cases, the expression type can be determined and in these examples, this would mean an “expected numeric expression” error.

For the two cases with an operator, the data type of the operand or the operator on top of the hold stack can be used to determine the expected type of expression to follow (just like for a non-parenthetical expression just corrected). For the first case, with just a parentheses, determining the expression type is a little more involved...

Translator – More Operand Processing Debugging

2010-12-30T21:22:00.000-05:00

Continuing with debugging, the next difference was a PRINT command that contained sub-string functions where the expected results needed to be updated (sub-string functions don't get attached tokens, tokens are attached to next token).

Finally in the Tenth (PRINT statements) test, there were three “debug #1” bug errors, which should have been an “expected numeric expression” errors. These test lines were incomplete statements (e.g. PRINT A+). This bug error was put in to fix later because the existing code was not correct. The code was modified to return the expected type error for the operand of the operand token on top of the hold stack (taking into account whether it is the first operand of a unary operator or second operand for a binary operator). It was previously set to return the variable type for the command on top of the command stack (which was wrong – the PRINT command has no data type).

Now on the the Eleventh (error statements) tests, which has quite of few incorrect error messages...

Windows, NetBeans and CVS (CVSNT)

2010-12-29T23:26:00.002-05:00

This post is not strictly related to the IBCP project, but will be helpful explaining how NetBeans (6.9.1) on Windows (XP) was made to work with CVS since it took nearly a day or searching and experimenting to accomplish (and many others were having similar problems).

The initial hope was that NetBeans for Windows would play nice with the CVS installed with MSYS (Minimum SYStem), but it did not. Searching indicated that NetBeans requires a CVS server even for a local repository and apparently, the CVS that comes with MSYS is not sufficient, though when doing a cvs ‑v command, it does respond … 1.11 (client/server). Further searching uncovered The CvsGui project and the WinCVS 2.1.1.1 package (includes WinCVS and CVSNT) was downloaded from http://sourceforge.net/projects/cvsgui/files/. Initially only the CVSNT package was installed (and this may have been sufficient).

Not realizing that there was a Service control panel under the Start menu (under CVSNT) that is used to configure CVSNT, the WinCVS package was installed. Normally WinCVS will also install the required CVSNT (the CVS server), but can be skipped if CVSNT is already installed. WinCVS is a very nice GUI for looking at checked out software (including nice version and branch graphs), however, it does not contain any configuration of CVSNT. And exactly how to check out a module from the repository was not discovered. This was not important as either the command line or NetBeans can be used to check out a module.

Once CVSNT is installed (indicates version 2.0.51d), it requests that Windows be restarted. This is necessary to install and start the CVS services, which occurred automatically upon reboot. Next the CVS server needs to be configured by using the Service control panel. The first tab reported that the services (CVS Service and CVS Lock service) were both running (a good sign).

The CVS repository needs to be identified using the Repositories tab by selecting Add. Using the browse button on the next dialog (the ... button), the repository that had been previously placed at C:/msys/1.0/home/cvsroot by using the MSYS CVS command was selected and entered in the Location field. The Name field was set to /msys/1.0/home/cvsroot.

After selecting Checkout... in NetBeans, the CVS Root was set to :local:/msys/1.0/home/cvsroot using the Edit... button, setting the Access Method to local and setting the Repository path to /msys/1.0/home/cvsroot (the same name as entered in the Name field in the CVSNT Service control panel Repositories tab). If all goes well, selecting Next> will immediately advance to the Module to Checkout page where the Module, Branch and Location Folder fields can be set. If this doesn't appear immediately, NetBeans either responds with an error, or spins its wheels looking for something (don't bother waiting – it's not going to work).

Several environment variables were set in the attempt to get it work and may or may not be necessary. The environment variables can be accessed by right-clicking on My Computer and selecting Properties, then going to the Advanced tab and selecting Environment Variables. Under System variables the following was entered:

PATH – Edited with C:\Program Files\cvsnt; at the beginning
CVSROOT – New set to C:\msys\1.0\home\cvsroot
CVS_EXE – New set to C:\Program Files\cvsnt\cvs.exe
CVS_RSH – New set to C:\Program Files\cvsnt\cvs.exe

One last warning about using CVSNT on Windows. CVSNT expects text files in the repository to be in DOS format (lines are terminated by CR/LF characters). If the text files are in Unix format (terminated by only LF characters), upon checkout, CVSNT will mess up the files by putting extra CR characters on each line. This is probably due to being a Windows program and expecting that text files to be in the expected Windows (DOS) format.

NetBeans and Debugging – Conclusions

2010-12-29T17:51:00.003-05:00

Enough time has now been spent debugging with NetBeans to make some conclusions. After some adjusting, using the GDB debugger under NetBeans appears to be easier then using Insight, the GUI front-end to GDB. The two are definitely different and initially it appeared that is was more difficult looking inside classes with NetBeans, but this was not the case. NetBeans puts the active local variables for each code line executed in the Variables window automatically and it's very easy to step into them to look at the contents.

With Insight, the variables first need to be added to the Watch window. Insight is never aware of allocated arrays for pointer members – it treats them as simply pointers. If you want to look at the contents of a particular array element, another variable needs to be added to the Watch window. And if too many variables are added to the Watch window, Insight becomes unstable (crashes). NetBeans usually appears to be aware of arrays and lets you descend into which ever element you want to look at.

In Insight there is no way to edit a Watch variable name, say to look at a different array element – it must be deleted and a new one entered (a lot of typing). In NetBeans, the watch variables can be edited (using Customize). Though disconcerting, with the jVi plugin installed, NetBeans requires the use of Vi commands in the New Watch and it starts in command mode – this will take a little getting used to. One last thing about the jVi plugin – sometimes it just stops working (NetBeans returns to its default editor). Restarting NetBeans corrects the problem.

In conclusion, NetBeans nicely integrates everything, therefore, development will be transferred to the NetBeans IDE with CVSNT (see next post on how this was made to work). The next release of the source will include the necessary files to build with NetBeans (development with VIDE2 will now be abandoned).

Translator – Operand Processing Debugging

2010-12-29T14:27:00.002-05:00

The Ninth (LET command) test contained multiple equal assignments, so the expected results needed to be updated since the multiple equal assignments feature was removed. Once the expected results were corrected, including removing the comma sub-code that was only needed to tell the difference between multiple equal and comma assignments, this test was successful. There was no necessity to used debugging here, but on to the Tenth (PRINT command) test that is crashing...

The Tenth test was crashing because of an invalid status code was being returned. This was tracked down to the add print code routine that was calling the process final_operand routine incorrectly (passed status variable as an argument instead of setting it to the return value).

Now all the PRINT commands were generating a “done stack not empty” bug error. This was caused by the hidden print codes (PrintDbl, PrintInt and PrintStr) not being popped from the done stack. These should not have been pushed on to the done stack and this was caused because these codes did not have the Print flag set in the table, which would have prevented them from being pushed on to the done stack.

Next the PRINT commands were generating a “invalid use of print function” error at the PRINT command. This was caused during the processing of the last hidden print code on the statement when the top of the command stack was checked for the PRINT command (the stack was empty, so the check was not performed). The PRINT command had been popped by the End-of-line command handler that called the PRINT command handler, which handles the last print code by calling the add print code routine. The add print code calls the process final operand routine, which checked for the print command. The solution was for the End-of-line command handle to leave the command on top of the stack when the command's handler is called and then pop the command off upon returning.

NetBeans and Debugging

2010-12-28T22:29:00.000-05:00

Now with the CVS/CR problem resolved (the NetBeans editor will show modifications from the CVS repository as changes are made) and the program builds and runs, it was time to start the debugger.

First off, NetBeans gave a warning that GDB 6.3 (installed with MinGW) was not supported. A later version (7.2) was found at the MingGW Sourceforge page http://sourceforge.net/projects/mingw/files/MinGW/BaseSystem/GDB/GDB‑7.2/ and the gdb‑7.2‑1‑mingw32‑bin.tar.lzma file was downloaded. This file consisted of three files, gdb.exe, gdbserver.exe and gdb‑python27.exe. The current files at \MinGw\bin were renamed to gdb‑6.3.exe and gdbserver-6.3.exe just in case they were needed and the three new files were copied in. Insight appears to still run, but I'm not sure if it using GDB 6.3 or GDB 7.2. NetBeans now seemed to be happy.

The next thing to figure out was how to specify command line arguments for the program being debugged. With Insight, this was done with the Console window using the “set args” command. The first thing attempted was to enable the GDB console in NetBeans, but there is not need to explain how to do that because the console isn't available until the program is started and by then it is too late for the program to see the arguments set because it had already read them. The solution was discovered under Project Properties in the Run section – there is an Arguments setting. Now with the arguments set to “‑t 9” for Ninth Translator test, debugging can begin...

NetBeans – Building the Program

2010-12-28T16:26:00.000-05:00

Before debugging, the program needed to be built. Within NetBeans, the IBCP project was created and the IBCP source code was checked out of the CVS repository. The .h include files were added in the project under Header files and the .cpp source files were added under Source files. If NetBeans proves satisfactory, the necessary NetBeans files will be included with the source files.

First, some options were set in NetBeans to simply the default directory structure, which assumes a multiple platform product - only Windows is being used here. By default, programs built are put into the directory “dist\Debug\MinGW-Windows” (for the debug configuration, “Release” for the release configuration). This is unnecessary and inconvenient. To make it build ibcp.exe in the root IBCP directory, the option in Project Properties, Linker, Output was changed to just ibcp from the more involved default ${CND_DISTDIR}/${CND_CONF}/${CND_PLATFORM}/ibcp.

Selecting build project immediately generated errors. The first file was opened and it looked as if it was double spaced (back-slash terminated line continuations caused the errors since the blank line prematurely terminated the line). Somehow in most of the source files, a control-M got added to the end of each line. This is a Unix vs. DOS test file format issue – Unix puts a single newline (control-J or linefeed) at the end of each line and DOS (Windows) puts a CRLF (control-M/control-J) at the end of each line. Vim deals with this invisibly and the GCC compilers ignore it.

I first noticed a week ago that these CRLF were in some of the source files when I did a “cvs diff” command it showed the whole file had changed. Outputting the diff into a file and looking at it with Vim showed the problem was all the lines from one file had “^M” characters at the end of each line, therefore causing no lines to compare. This was causing no problems with either Vim or the compiler.

After some playing around, the problem was discovered to be caused by CVSNT needed by NetBeans (more on this later), which seems to get confused by Unix format files in the repository. The first attempt was to convert all the files to Unix format, but this did not help. After all the files were converted to DOS format, the problem appeared to be solved. Further checkouts were OK. The program then built with no issues. Tests ran as last released. Now on to debugging...

NetBeans and Vim (jVi plugin)

2010-12-28T09:50:00.001-05:00

The version of NetBeans installed is the latest 6.9.1 and can be obtained for free at http://netbeans.org/. The C/C++ pack was installed. JDK 6 (Java Development Kit) also needs to be installed, but that was already installed on the computer for something else. The jVi (vi editor clone) plugin can be downloaded at http://sourceforge.net/projects/jvi/. The downloaded nbvi‑1.3.0.x1.zip file was unzipped, which contains a directory that was put into the Program Files\NetBeans 6.9.1 directory. To install this plugin package, once NetBeans was installed and running, go to Tools/Plugins and the Downloaded tab. Select “Add Plugins...” and find the nbvi-1.3.0.x1 directory and select the two .nbm files. Once both are selected, select the Install button.

Before going into what was needed to make CVS work with NetBeans (which is not necessary for building the IBCP program from the source, but I want to give details on how to do it since it may help someone else in a similar situation as it took a bit of searching to figure it out), I wanted to see what was needed to make debugging work within NetBeans (it works very nice under Linux). This can be done by actually debugging starting with the Ninth Translator (Commands) test...

New Development Environment

2010-12-27T23:47:00.003-05:00

I decided to try out a new IDE I recently learned about – NetBeans, which has integration with CVS, an integrated debugger (front end to GDB), and there is an available plugin to get vim editor capability. Getting the Windows version to work properly was going to be a challenge through.

Currently for development, MSYS (Minimum SYStem) with MinGW (Minimum GNU for Windows) is being used with CVS for MSYS installed for version control. MinGW was updated to version 4.4.0 of the GNU Compiler Suite. For controlling (building) the project, VIDE2 (a simple IDE) is being used, but it contains a very primitive editor (gVim is being used instead for editing). For debugging, Insight, a GUI front end for GDB (version 6.3) is installed. Insight is not exactly stable (crashes a lot), and the latest version 6.8.1 could not be made to work with MSYS.

I'm not sure I will be able to get NetBeans working satisfactory. So far, after a lot of struggle, NetBeans is working with CVS, though not the CVS that was installed with MSYS. Complete details will be in a future post. The IBCP was successful checked out of the CVS repository with NetBeans, the IBCP project was created NetBeans and was built successfully. Next will be to get the debugger working.

Translator – Sub-String Pre-Release

2010-12-27T10:09:00.003-05:00

Now the first eight Translator tests succeed. There are more problems to correct with tests 9 through 12 (test 10 crashes). But, since it has been half a year since code has been released, it's time to commit and tag the changes to CVS and make another pre-release.

I was distracted during the last two months with a work project and didn't get much time to work on this project. I made the mistake of not committing the last good working code to CVS (mainly because I wanted to fix just one more bug). It took almost a week to recover, figure out what I was in the middle of, make the code compile again, and get the most recent changes working (almost). With that accomplished, a pre-release can be made. The file ibcp_0.1.14-pre-2-src.zip has been uploaded at Sourceforge IBCP Project along with the binary for the program. Now on to more debugging to get the rest of the tests working again...

Translator – Sub-String Debugging

2010-12-25T23:10:00.000-05:00

To implement the sub-string change, not pushing the sub-string function to the done stack leaving its string operand on the stack (to carry to the next token), is accomplished by using the same mechanism that prevents print function from being pushed on the done stack (set the done push flag to false). With the number of operands being left set to zero, the sub-string function's string operand will be left on the done stack.

As debugging continued, nothing was getting attached to some AssignStr tokens. The assign command handler was only attaching tokens of the String data type, not the TmpStr data type returned (by the concatenate operator and the non-sub-string functions). In these cases, the AssignStr code will be changed to the AssignStrTmp code, but none of the associated codes for TmpStr arguments have been implemented yet (that will come next). Therefore, this will be left as is for now (TmpStr tokens will not be attached to AssignStr).

The new sub-string implementation (not pushing them on the done stack), made sub-string assignments stop working (produced “done stack empty” errors). The variable being assigned was popped from the done stack because it was not needed since it doesn't need to be attached to a token. When the assignment operator (equal) went to look for what was being assigned to determine which assignment code was needed, the done stack was empty. In this case, the sub-string function needs to be on the done stack. Therefore, a check was added to still push the sub-string function token if its reference flag is set.

The sub-string change also affected error reporting - the “expected numeric expression” error for a statement like Z=B+MID$(A$,2,3) was now incorrectly pointing to the A$ instead of the MID$ token (because the A$ carried forward and the MID$ token was not on top of the done stack).

To correct this problem, the code at the end of the find code routine that returns the appropriate error when an operand of the wrong data type was modified. If the last token appended to the output list doesn't match the top token on the done stack and the last token is a sub-string function, then the token returned is set to the sub-string token instead of the top token.

Translator – Sub-String Implementation

2010-12-24T20:30:00.002-05:00

Implementation of the transfer of operands of sub-strings will mostly take place in the process final operand function. But first, some information needs to be added to the Table, namely the number of string arguments each code has. This value could be calculated on the fly as needed, but it will make for a simpler design to just put these values in the table. The table initialization code (Table constructor) was modified to calculate these values automatically.

In the process final operand routine, the internal function and operator section, when the number of operands passed in is zero, needs to set the number of operands to the number of string arguments that are on the done stack. These operands will be popped from the done stack and attached to the internal function or operand token. Only string operands were left on the done stack (other numeric operands were popped by the find code routine as they do not need to be attached to any token).

However, for sub-string functions, which have only one string operand, the string operand will be left on the done stack, to be attached to the next token. If the next token is another sub-string function, then the string operand argument will continue to be carried forward. Therefore, the number of (string) operands to pop from the done stack will only be set to the code's number of string arguments if the code is not a sub-string. For sub-string function, the number of operands will be left set to zero, so nothing will be popped and attached to the sub-string function token.