Gelex: The Generated Scanner

The Generated Scanner

For each scanner description file given as input, gelex will generate an Eiffel class as output. By default gelex sends its output to the standard output but this can easily be overridden with option -o filename.

The deferred class YY_SCANNER, which is part of Gobo Eiffel Lexical Library, provides an abstraction for scanners. Every scanner class generated by gelex will be a descendant of this class. The main feature of YY_SCANNER is routine read_token which, when called, scans tokens from the input buffer until it either reaches an end-of-file or last_token is given a non-negative value. Each time read_token is called again it continues processing tokens from where it last left off until it either reaches the end of the file or last_token is given a non-negative value again. On the other hand, the routine scan scans the input buffer until it reaches the end-of-file, ignoring successive values of last_token. Class YY_SCANNER is equipped with three procedures which should be used as creation routines in descendant classes. These routines are make which prepares the scanner for scanning from the standard input, make_with_file or make_with_unicode_file for scanning from a given file, and make_with_buffer for scanning from a given input buffer. Before scanning a new input buffer, the scanner should be reinitialized using routine reset. Also available to descendants of YY_SCANNER is a set of features which can be called from the semantic actions.

An implementation for most of these routines described above is provided in class YY_SCANNER_SKELETON. This class has itself three descendants, each of them providing a different flavor of the pattern-matching engine - a Deterministic Finite Automaton (or DFA for short) - used for the implementation of feature read_token. The first of these three classes is YY_COMPRESSED_SCANNER_SKELETON whose DFA is optimized in terms of memory space. Gelex should in that case compress the generated tables by taking advantage of similar transition functions for different states. This is what gelex does by default. The second class is YY_FULL_SCANNER_SKELETON whose DFA is optimized in terms of execution speed but uses big tables. In this case the option full has to be specified for gelex to generate non-compressed tables. The last class is YY_INTERACTIVE_SCANNER_SKELETON whose DFA can deal with interactive input such as input from the keyboard. An interactive scanner is one that only looks ahead to decide what token has been matched if it absolutely has to. It turns out that always looking one extra character ahead, even if the scanner has already seen enough text to disambiguate the current token, is a bit faster than only looking ahead when necessary. But scanners that always look ahead give dreadful interactive performance; for example, when one types a newline, it is not recognized as a newline token until one enters another token, which often means typing in another whole line. YY_INTERACTIVE_SCANNER_SKELETON is an heir of YY_COMPRESSED_SCANNER_SKELETON and only requires from gelex to generated compressed tables. Because of the performance penalty already implied by this more intuitive interactive behavior, this facility is not available in conjunction with full table scanners.

Gelex does not automatically generate the note, class header, formal generics, obsolete, inheritance, creation and invariant clauses. These have to be specified in Eiffel declarations in the first section and in the user code section of the scanner description file. In particular, one has to specify which of the three scanner skeleton classes described above (namely YY_COMPRESSED_SCANNER_SKELETON, YY_FULL_SCANNER_SKELETON or YY_INTERACTIVE_SCANNER_SKELETON) to inherit from as an implementation for its pattern-matching engine. The following example shows a typical scanner description file:

%{
class MY_SCANNER

inherit

    YY_COMPRESSED_SCANNER_SKELETON
        redefine
            make, reset
        end

create

    make
%}

%%

...patterns...   ...actions...

%%

feature {NONE} -- Initialization

    make
            -- Create a new scanner with
            -- standard input as input file.
        do
            Precursor
            create buffer.make (256)
        end

feature -- Initialization

    reset
            -- Reset scanner before scanning next input file.
        do
            Precursor
            buffer.wipe_out
        end

feature -- Access

    buffer: STRING
            -- Scanner's buffer

invariant

    buffer_not_void: buffer /= Void

end

The generated scanner class, named MY_SCANNER, uses a pattern-matching engine optimized in terms of memory space. It redefines features make and reset inherited from YY_COMPRESSED_SCANNER_SKELETON to take into account its internal buffer and preserve the invariant. The last two examples are excerpts from descriptions of a scanner optimized in terms of execution speed and an interactive scanner:

%{
class MY_FAST_SCANNER

inherit

    YY_FULL_SCANNER_SKELETON

create

    make
%}

%option full

%%

...patterns...   ...actions...

%%

end

Please note in particular in the above example the %option full directing gelex to generate non-compressed tables.

%{
class MY_INTERACTIVE_SCANNER

inherit

    YY_INTERACTIVE_SCANNER_SKELETON

create

    make
%}

%%

...patterns...   ...actions...

%%

end