Geyacc: Symbols, Terminal and Nonterminal

Symbols, Terminal and Nonterminal

Symbols in geyacc grammars represent the grammatical classifications of the language.

A terminal symbol (also known as a token type) represents a class of syntactically equivalent tokens. You use the symbol in grammar rules to mean that a token in that class is allowed. The symbol is represented in the geyacc parser by a numeric code, and the read_token routine returns a token type code last_token to indicate what kind of token has been read. You don't need to know what the code value is; you can use the symbol to stand for it.

A nonterminal symbol stands for a class of syntactically equivalent groupings. The symbol name is used in writing grammar rules. By convention, it should be in lower case.

Symbol names are case-insensitive words beginning with a letter and followed by zero or more letters, digits, or underscores.

There are three ways of writing terminal symbols in the grammar:

A named token type is written with an identifier, like an identifier in Eiffel. By convention, it should be all upper case. Each such name must be defined with a geyacc declaration such as %token.

A character token type (or literal character token) is written in the grammar using the same syntax used in C for character constants; for example, '+' is a character token type. A character token type doesn't need to be declared unless you need to specify its associativity, or precedence.

By convention, a character token type is used only to represent a token that consists of that particular character. Thus, the token type '+' is used to represent the character + as a token. Nothing enforces this convention, but if you depart from it, your program will confuse other readers.

All the usual escape sequences used in character literals in C can be used in geyacc as well, but you must not use the null character as a character literal because its ASCII code, zero, is the code read_token returns for end-of-input.

Note that Unicode characters are supported. Just make sure that the input file is using the UTF-8 encoding and starts with the BOM character. Alternatively this escaped forms '\u03B2' or '\u{03B2}' can be used.

A literal string token is written like a C or Eiffel string constant; for example "<=" is a literal string token. A literal string token needs to be associated with a symbolic name as an alias, using the %token declaration. That way you can also eventually specify its semantic value type and its associativity or precedence.

By convention, a literal string token is used only to represent a token that consists of that particular string. Thus, you should use the token type "<=" to represent the string <= as a token. Geyacc does not enforce this convention, but if you depart from it, people who read your program will be confused.

Because literal string tokens have been introduced to make the grammar description more readable, no escape sequences are allowed. But they can contain Unicode characters. Just make sure that the input file is using the UTF-8 encoding and starts with the BOM character. Furthermore, a literal string token must contain at least two characters; for a token containing just one character, use a character token.

How you choose to write a terminal symbol has no effect on its grammatical meaning. That depends only on where it appears in rules.

The value returned by read_token is always one of the terminal symbols (or 0 for end-of-input). Whichever way you write the token type in the grammar rules, you write it the same way in the definition of read_token. The numeric code for a character token type is simply the code for the character, so read_token can use the identical character constant to generate the requisite code. Each named token type becomes an integer constant feature in the parser class, so read_token can use the name to stand for the code. For a literal string token, read_token has to use the named token type associated with it.

If read_token is defined in a separate class, you need to arrange for the token-type integer constants definitions to be available there. Use the -t option when running geyacc, so that it will write these constants definitions into a separate class from which you can inherit in the classes that need it.

The symbol error is a terminal symbol reserved for error recovery; it shouldn't be used for any other purpose. In particular, read_token should never return this value.