Parser – Create Token As Needed

The parser will be modified to create a token only when a valid token is found in the input string and is returned directly.  This means that the token will be created on a return statement, which is automatically moved to the caller since the created token (in a shared pointer) is temporary and going out of scope.

Once all the get routines in the parser are changed, it will no longer be necessary to have a member variable to hold the token and there will be no worry of a token being left allocated for an error.  Right now the returned token will be an shared pointer, though is not necessary.  The return pointer will be changed to a unique pointer, which can be assigned to a shared pointer.

The get identifier routine was the first to modified.  Most of the token creation lines were replaced with returns statements creating a token in a shared pointer:

return std::make_shared<Token>(pos, len, type, dataType, m_input);

To support this, two new constructors were added to the token class.  One that in addition to the column, length, type and data types values takes the input string (from which a string is created using the column and length values) as shown above.  The other constructor taking a code and optional string, which is used by a new token function added to the table class that uses the table to set the type and data type values of the new token.

Once all the locations where in the get identifier routine were replaced, it could be seen that the code was repetitive, so the whole function was reorganized and reduced.  If no valid identifier token is found, a default token pointer is returned, which the caller can check as a boolean.

The main function operator routine was modified to support this partial transition.  When the end-of-line is reached, a new token is created and returned (using the new token table function).  The get identifier routine is called in an if statement by itself receiving the return value in a if-scoped variable, which is returned if set:

if (TokenPtr token = getIdentifier()) {
    return token;
}

For now, the current creation of a new token was moved to after the statements above.  It will continue being moved as each get routine is changed until all have been changed at which time it will be removed along with token member pointer.

[branch parser commit e34fbccacc]