REX: Ruby Lex for Racc
About
Lexical Scanner Generator for Ruby, with Racc.
Usage
rex [options] grammarfile
-o --output-file filename designated output filename.
-s --stub append stub main for debug.
-i --ignorecase ignore char case
-C --check-only syntax check only.
--independent independent mode.
-d --debug print debug information
-h --help print usage.
--version print version.
--copyright print copyright.
Default Output Filename
It destinate from foo.rex to foo.rex.rb.
This name is for a follow description.
require 'foo.rex'
Grammar File Format
A definition is given in order of a header part, a rule part,
and the footer part. One or more sections are included in a rule part.
As for each section, the head of the sentence starts by the keyword.
Summary:
[Header Part]
"class" Foo
["option"
[options] ]
["inner"
[methods] ]
["macro"
[macro-name regular-expression] ]
"rule"
[start-state] pattern [actions]
"end"
[Footer Part]
Grammar File Example
class Foo
macro
BLANK \s+
DIGIT \d+
rule
{BLANK}
{DIGIT} { [:NUMBER, text.to_i] }
. { [text, text] }
end
Header Part ( Optional )
All the contents described before the definition of a rule part are
posted to head of the output file.
Footer Part ( Optional )
All the contents described after the definition of a rule part are
posted to tail of the output file.
Rule Part
Rule part is from the line which begins from the "class" keyword
to the line which begins from the "end" keyword.
The class name outputted after a keyword "class" is specified.
If embellished with a module name, it will become a class in a module.
The class which inherited Racc::Parser is generated.
Rule Header Example
class Foo
class Bar::Foo
Option Section ( Optional )
"option" is start keyword.
"ignorecase" when pattern match, ignore char case.
"stub" append stub main for debug.
"independent" independent mode, for it is not inherited Racc.
Inner Section ( Optional )
"inner" is start keyword.
The contents defined here are defined by the inside of the class
of the generated scanner.
Macro Section ( Optional )
"macro" is start keyword.
One regular expression is named.
A blank character (0x20) can be included by escaping by \ .
Macro Section Example
DIGIT \d+
IDENT [a-zA-Z_][a-zA-Z0-9_]*
BLANK [\ \t]+
REMIN \/\*
REMOUT \*\/
Rule Section
"rule" is start keyword.
[state] pattern [actions]
state: Start State ( Optional )
A start state is expressed with the identifier which prefaces ":".
When the continuing alphabetic character is a capital letter,
it will be in an exclusive start state.
When it is a small letter, it will be in an inclusive start state.
Initial value and default value of a start state is nil.
pattern: String Pattern
The regular expression for specifying a character string.
The macro definition bundled with { } can be used for description
of a regular expression. A macro definition is used in order to use
a regular expression including a blank.
actions: Processing Actions ( Optional )
Action is performed when a pattern is suited.
The processing which creates a suitable token is defined.
The arrangement whose token has the second clause of classification
and a value, or nil.
The following elements can be used in order to create a token.
lineno Line number ( Read Only )
text Matched string ( Read Only )
state Start state ( Read/Write )
Action is bundled with { }. It is the block of Ruby.
Don't use the function to change the flow of control exceeding a block.
( return, exit, next, break, ... )
If action is omitted, the character string which matched will be canceled
and will progress to the next scan.
Rule Part Example
{REMIN} { state = :REM ; [:REM_IN, text] }
:REM {REMOUT} { state = nil ; [:REM_OUT, text] }
:REM (.+)(?={REMOUT}) { [:COMMENT, text] }
{BLANK}
-?{DIGIT} { [:NUMBER, text.to_i] }
{WORD} { [:word, text] }
. { [text, text] }
Comment ( Optional )
From "#" to the end of the line becomes a comment in each line.
Usage for Generated Class
scan_setup()
The event for initializing at the time of the execution start of a scanner.
It is redefined and used.
scan_str( str )
Parse the string described by the defined grammar.
Token is stored in an inside.
scan_file( filename )
Parse the file described by the defined grammar.
Token is stored in an inside.
next_token
One token stored in the inside is taken out.
The last returns nil.
Notice
This specification is provisional and may be changed without a preliminary
announcement.