Parser – Operator Tokens

The get operator routine was modified to create a new token upon returning when a valid token is found.  If the first character is not the start of an operator, a default token pointer is returned.  The existing table new token function is used to create the new token upon return.  The flow of the function was cleaned up by checking for an invalid operator first, a remark operator next and finally for a two-character operator.

In the main function operator routine, the call to get string was changed like the other get function calls with the member token initialization was finally removed along with the token member.

[branch parser commit 632ce89f80]

Parser – Constant String Tokens

The get string routine was modified to create a new token upon returning when a valid token is found.  If the first character is not the start of a string constant (a double quote), a default token pointer is returned.  A token constructor was added to support creating a string constant token, which in addition to the column and length takes the string constant without the surrounding double quotes.

This routine was changed from setting characters into the token string (by a length index counter that was not otherwise used) to simply appending the characters to a local string (since the token is not created until the return statement).  The former will not work with standard strings.  This local string is moved to the constructor, though this has no effect with a QString (copies if class doesn't support move), but will with standard strings.

The set string character token access function was removed since this routine was the only caller.  In the main function operator routine, the call to get string was changed like the other get function calls with the member token initialization moved below this.

[branch parser commit 87e4ed4fb8]