Parser Errors – New Test

There are seven parser errors that can occur, but only two major types - an unrecognizable character error and some type of error in a numerical constant.  To test each of these seven errors, a new translator test (#14) was created for all the possible places during translation each of these errors can occur.  If no instances were missed, this is 16 places for each of the seven errors for a total of 112 test inputs.

The translator was temporarily modified to output the string "PARSER:" in front of the parser errors so they can easily be seen.  This does affect any of the current expected test results since none of the current translator tests have parser errors (an obvious oversight).

For now, the expected results for this new test contains the current output, but will be updated as these are corrected to the desired "expected such-and-such" error messages.  The numeric errors are appropriate if the translator was expecting a numerical constant (in other words, an operand), so the existing parser errors will be changed to the "expected such-and-such" format with the possible exception of the "constant is out of range" error.

Note that because of the recent changes to the regression test scripts (which now automatically detect tests), no modifications were needed to add this new test.

[commit  aa9271b8c9]

Translator Class Usage

Just like the Parser class, a single instance of the Translator class is created and a reference to it is passed around the program.  Unlike the Parser class, the Translator class has many more member variables, many of which are complex types (mainly stacks).  So that all these members do not need to be initialized for each new instance, the single instance design will remain for now.

There is a loop that performs the translation of an input line - the test translate input routine - which also prints the results of the translation.  This translation loop should really be handled within the Translator class, which should either return the translated output or an error.  The test routine should call this and then handle printing the results.

Therefore, a new setInput() function was implemented in the Translator class (the function name chosen to mirror the Parser class).  The functionality of the start() and getOperateState() functions was moved to this new function, which also instances its own parser.  A boolean success/fail flag is returned.

For the caller, upon success, it can proceed to obtain the output using the Translator output() function, renamed from getResults().  If an error is found, the Translator will clean up its internal variables and save the token at which the error was found with an error message.  The caller can access these through the new errorToken() and errorMessage() access functions.

The Translator will handle releasing the memory used by the error token.  There is a new internal function to set the error token.  If the error token pointer is already set, the old error token is freed.  The error token is also freed in the Translator's destructor if the pointer is set.  There is one problem with this scheme - Parser errors are returned directly, and these are not in the form "expected such-and-such" so errors could be confusing to the user.  This will be tackled next.

One other minor change was made, there was a setDefaultDataType() function in the Translator for setting the default data type of a token.  This was the remaining in-line access function still in the Translator header file and worked by examining the various members of the token.  No Translator variables were used, so it is more appropriate for this to be a Token class function, so was moved to the Token class as a new setDataType() function (taking no arguments, overloading the current function that takes a data type argument).

[commit 0dc86d1a4] [commit a88ed2f2bb]

Parser Class Usage

Currently, an instance of the Parser class is created for the program and a reference to it is passed around between the various routines.  The Parser class only has a few member variables and except for the reference to the table instance, these members are initialized for each new line that is parsed.  Therefore, it is not necessary to have a single parser instance for the program - a parser instance can be created as needed and destroyed when no longer needed.

The Parser class can be though of as a function call, though a complex one.  The parser is given an input line and returns one token at a time until either the end of the line is reached or an error is found.  A new function could be implemented to return a list of tokens for the line being parsed.  There are two locations where the parser is currently being used: the test parse input and test translate input routines.

Both of these callers work slightly different.  The test parse input routine just gets tokens until the line end or an error is found.  However, the test translate routine sets the Parser operands state from its own operand state (whether looking for an operand or not) before getting each token.  This is done so that the Parser can determine when it should be looking for a negative sign on a constant (operand state) or a minus operator (not operand state).

This implies a intimate use of the parser while a line is being translated.  So a function was not implemented to return a list of tokens.  The code was modified to instance a Parser when needed - in the two test routines - instead of passing a single instance reference of the Parser around.  These leads to the usage of the Translator class...

New Change Comment Philosophy

Going through the code and for the most part removing all the change comments (with a date), I've had a change in philosophy one how code should be commented.  Perhaps for a released program, it makes sense to add a comment with a date when a bug is fixed, but during development and debugging it's expected that there will be a lot of changes.  Making a dated change comment for every single change is just a bit excessive - and this certainly could be seen here.

The comments at the beginning of the files probably should be removed also or at least cleaned up, but these were left alone for the time being.  Perhaps some of the major highlights will be left in.  In any case, going forward, the git commit comments will be relied on to comment the changes made to the source file.  Functionality comments will still be added as needed, but without all the change comments, the code should be easier to read.

The latest commit contains more of this comment clean up.  All of the project is now using Qt functions; all the standard C library functions (and associated include statements) have been removed.  Most if not all the variables and functions have been renamed to the new naming scheme.  This is probably a good place to add another tag, and so the current code was tagged as v0.2-3.

[commit ebe943379d]

Qt Transition – Translator Class

The variables and functions of the Translator class were renamed to the Qt style naming.  Nothing much to report here except that the RpnItem structure used by the translator was changed to a class with private members with access functions added.  Through the translator source file along with the new token and command handler source files, all of the dated change comments were mostly removed (some were appropriate comments so only the date was removed).  These were just cluttering up the code (for now the header comments were left alone).

[commit  85daa1fc7e]

Single Class Instance (Singleton Class)

There is only going to be one instance of the Table class in the program.  I recalled from when learning C++ that there was a way to allow only one instance of a class to be created.  After a little research, it was found that this is known as a singleton class.  There are several ways to implement this.  The solution used for the Table class is described below.  The goals were (click Continue... for complete details on the implementation):

1. Allow only one instance of the table to be created
2. Detect table entry errors when the instance is created
3. Return a list of any error messages of errors found in the table entries
4. Get a reference to the table instance created

Qt Transition – Table Class

The Table class contains two internal structures, one for a table entry and one for expression information within the table entry.  Only the Table class should have access to these, so their definitions were moved from the table header file to the table source file along with two constants.  Only a forward reference to the table entry was left behind.  Other code has no need to access these.

As a side effect, the code for all the table access functions also had to be moved to the table source file since the structure definitions were no long available in the header file.  This would appear to cause an inefficiency since these functions are no longer in-line functions since they are not defined directly in the class definition.  However, the GCC compiler (and probably other C++ compilers) as part of their optimization, will still in-line these small functions.

All of the table variables and functions were renamed to Qt style naming.  The remaining C character definitions were changed to QString and the remaining standard library calls (and their include statements) were replaced with Qt equivalents.  All the various initialized arrays used for the table entries were made static since they are only used in the table source file.

[commit e8ddccc029]

Qt Transition – Token Class

The Token class, which previously was defined as a structure (though the only difference being that members are public by default instead of private), was changed to be a full class complete with access functions for all the member variables.  Some additional convenience access functions were created (like checking if the code in the token is equal to a particular code).

[commit fc32a47915]

Code (Class) Reorganization

One convention used in C++ programs are to place each major class into their own header and source file.  While the source code for major classes for this project are already in separate files (table, parser and translator), all the class definitions were in a single header file and this header file was getting unwieldy.

So in preparation for all the new classes that will be added for the GUI, the major classes were separated into their own header files including the Token class with its own source file.  The translator source file was also getting large, so the token handler and command handler functions were moved into their own source files with associated header files.

As part of this reorganization, many of the dated change comments were removed as these were just cluttering up the code.  None of the change comments in the header file were changed, but probably should as they are taking up quite a few lines.  The awk scripts also were modified since the sources of some of the enumerations were relocated and the CMake file was updated accordingly for the new files.

Several static access functions were added to the Translator class so that the token and command handler functions have access to the static values in the translator source file, which were previously accessible since these functions were in the same source file.  These handlers are also accessing many other Translator internal data members (the reason they are defined as friend functions).  This also should be changed, which will be done when the Translator class is transitioned.

[commit 1254b8048b]

Qt Transition – Parser (Final)

Support for immediate command parsing was implemented in anticipation of the temporary console mode interface, essentially the way GW-Basic works.  However, now that the project is being transitioned to Qt for a GUI, these immediate commands are no longer needed.  Another factor for removing this was that a lot of modifications to the command parsing functions, which accounted for about a third of the Parser code, would need to be converted to using Qt functions.  A waste of time since this would eventually be removed.  Therefore, immediate commands support was removed along with parser test 1.

The only code currently using the String class was the Token class.  The Parser class generated tokens and therefore the strings inside the tokens.  The String class has been removed from the Parser and Token classes, which now uses Qt functions for parsing and no longer uses any of the standard C library functions (the goal for the entire program, but one step at a time).

Since the String class is no longer used, the header and source files for this class were also removed along with the string test program and expected results file.  The other three test programs were also removed because these no longer apply, which were for testing conversion of number strings (Qt functions now used), exceptions thrown from constructors (no longer used), and testing operator processing on a stack (Stack class was already removed).

Upon testing on Windows, the range error checking problem reported on October 14 returned.  This was due to how numbers are converted in the Qt libraries.  Originally on Linux exponents -308 and below caused a range error, but on Windows exponents did cause a range error until -324.  Now with Qt, on Linux exponents -324 and below cause a range error and on Windows it takes exponents -509 and below.  Therefore, the test value was changed to 1.234e-509.

[commit 6b4afaa538] [commit 348339c7a7] [commit 819fac5185]

String to Number Conversions

As the modified Parser code was being tested, a weird memory issues was reported by valgrind.  The problems occurred with the toInt() and toDouble() functions of QString.  The problem was duplicated with a very simple program:

#include <QString>
int main(void) {
    qDebug("%d", QString("123").toInt();
}

The same issue occurs if the program above is changed to double with toDouble().  No reason for this error could be found.  However, when this program is turned into a Qt console application, the error no long occurred.  But this same thing applied to the ibcp program did not eliminate the error.  Click Continue... for how to build and run this program from the command line to demonstrate the memory issue (requires Linux with Qt and valgrind installed).

No solution was found for this problem.  When the program was changed from QString to QByteArray, which also contains these same two functions, no memory issue was reported.  Therefore, as a temporary solution, the string to convert is converted to a QByteArray.  A QByteArray was declared and the QString to convert was appended to it.

Minor Change – Vector vs. Map

One of the changes made to the test code was to put the names of each test mode into a QMap where the enumeration name for the test mode is associated with the name of the test mode (as a QString).  This was the original code (the array is then indexed by the enumeration value) and the names were declared separate so there could be referenced directly (though the name[] could have been used to get the names):

char parser_name[] = "parser";
char expression_name[] = "expression";
char translator_name[] = "translator";
char *name[] = {
    parser_name, expression_name, translator_name
};

The problem with this code is that the programmer must insure that the correct names are placed in the array in the correct order matching the enumeration values, otherwise the wrong name will be used.  This is also basically why the Code and TokenStatus enumerations are automatically generated - to eliminated possible coding errors.  Using QMap was a way to eliminate this possibility.  This was the resulting code (the map is still indexed by the enumeration value) and the name map was used to access the names:

QMap<testModeEnum, QString> name;
name[testParser] = "parser";
name[testExpression] = "expression";
name[testTranslator] = "translator";

However, the enumeration declaration had to moved outside the function or it would not compile (apparently, local enumerations can be used).  In later considering this code, a better method would be to use a pre-sized QVector since a QMap has more overhead that is really not needed here and the enumeration values are in order.  The code was changed to the nearly identical:

QVector<QString> name(sizeofTestMode);
name[testParser] = "parser";
name[testExpression] = "expression";
name[testTranslator] = "translator";

Where the sizeofTestMode value was added to the end of the enumeration (which was moved back into the function) so that the vector could be allocated ahead of time.  Using QMap would be needed if the indexes being associated were not in numerical order.

[commit  ad08201092]

Minor Build And Test Issues

A build issue was discovered where CMake does not create a release string if the git command is present but the git repository is not (for instance when building from a downloaded archive).  The git describe command was returning an error and no release string.  CMake now detects this situation and sets the release string the same as if no git command is found.

A test issue was discovered on Windows when building from a downloaded archive, which contains Unix format files (newline only) and not DOS format files (CRLF).  When the program is run from the regression test script (the program builds fine), the output files are in DOS format, but the compares fail because the expected output files are in Unix format.  The cmp command was changed to the diff command, which has an option to ignore the difference in the line separators (the ‑w ignore white space option).

All these changes have been pushed to GitHub and because of the build and test issues found, new tag v0.2‑2 was added.  The changes to the Parser were complete (with the attempt to compile next) before the text stream detour and these other minor issues.

[commit 0e85c83d56] [commit 2be4bd2f91]

Qt Transition – Strings (File Input)

There was an issue in the way the test files were being read.  When the test code was modified to use Qt, the QFile class was used to read the file, where the function used to read a line is actually inherited from the QIODevice class, which QFile is based on.  This function returns the line into a QByteArray, which was easily converted to a character array currently used by the Parser.

However, the Parser is being converted to use the QString class.   The QString class actually contains QChar characters, which supports 16-bit Unicode.  Reading the file as QByteArray would need to be converted to a QString and would not support Unicode text files.  After doing some research, it was found that files can be read as Unicode text using the QTextStream class (into a QString).

Therefore, the file reading code was modified to use a QTextStream (where a pointer to the QFile instance is given in the constructor).  This will also be used for the standard (console) input for the interactive test modes.  The file is opened the same way, but the at end of file check and read line routines from QTextStream are used instead.  This read line routine also strips the line separator from the line (ether newline on Linux or CRLF on Windows).

Temporarily, the QString line is converted to a QByteArray, a null character is added to the end and then converted to character array (constant character pointer) to pass to the Parser.  So that a type cast to a char * was not needed, the argument and variables in the Parser were changed to const char *, which is fine since the Parser does not modify the input line.

[commit 993aa66765]

Qt Transition – Strings (Begin)

The last class to replace is the String class.  The Qt equivalent class is QString and related QByteArray.  As with the List class, the transition will be done in steps.  The Token class would be first.  The first primary user of the Token class is the Parser class, which is also very dense with string operations.  The QString class has many useful functions, and these should help simplify the Parser functions.

Before tackling the Parser, the test code will need to be able to handle the change from String to QString.  The are two other major items in the test code that also needed to be transitioned to Qt, namely file handling, console input and console output.

The Qt QFile class handles file handling and is much simpler to use then the c file handling that was being used.  The QFile class can also handle reading input from the console.  There is also the QFileInfo class that contains many useful functions, but the ones used here were for parsing file names (including the program name).  There are single functions for extracting the path, file name, base file name without extension.

The QTextStream class handles output and is very similar to C++ stream output.  In the main source file, the standard output is opened as a QFile and is attached to a QTextStream named cout.  This text stream is then passed to all the functions that need to do output.

As described in the previous post, the opportunity was taken to rename many of the variables and functions to the Qt naming convention.  Now that the callers to the Parser (and Translator) have been transitioned to Qt, it's time to work on the Parser (and Token) classes.

[commit d94cd3c632]

Qt Transition – Qt Naming Convention

Part of the Qt transition is will be use the Qt naming convention for classes, functions and variables.  Specifically, Qt uses camel casing where the first letter of words in a name are upper case, where the first letter of the name is lower case (except for class names where the first letter is upper case.  There is also a convention for naming memory variables along with there access functions.  This is best shown with this example class:

class MyClass {
    int count;
    int some_value;
public:
    int get_count(void) {
        return count;
    }
    void set_count(int _count) {
        count = _count;
    }
    int get_some_value(void) {
        return some_value;
    }
    int set_some_value(int _some_value) {
        some_value = _some_value;
    }
    bool empty(void);
    bool data(void;
    void process_some_data(int data, int more_data);
}

Note the underline character that is used for separating the words of the variable and function names and the names of the access function.  The arguments on the set functions were also prefixed with an underline to make the name unique from the memory variable.  Using the Qt naming convention, this class definition will look list this:

class MyClass {
    int m_count;
    int m_someValue;
public:
    int count(void) {
        return m_count;
    }
    void setCount(int count) {
        m_count = count;
    }
    int someValue(void) {
        return m_someValue;
    }
    int setSomeValue(int someValue) {
        m_someValue = someValue;
    }
    bool isEmpty(void);
    bool hasData(void);
    void processSomeData(int data, int moreData);
}

Note the "m_" prefix on each member variable, but not on member functions.  Also note that getter functions do not begin with get.  And other than the "m_" prefix, underlines do not appear on any names.  Also note that boolean member functions that return status are prefixed with is or has.  As the Qt transition continues, this naming convention will be used (and this has already started with some of the changes made so far).