Package org.antlr.grammar.v2
Class AssignTokenTypesWalker
- java.lang.Object
-
- antlr.TreeParser
-
- org.antlr.grammar.v2.AssignTokenTypesWalker
-
- All Implemented Interfaces:
AssignTokenTypesWalkerTokenTypes
- Direct Known Subclasses:
AssignTokenTypesBehavior
public class AssignTokenTypesWalker extends antlr.TreeParser implements AssignTokenTypesWalkerTokenTypes
[Warning: TJP says that this is probably out of date as of 11/19/2005, but since it's probably still useful, I'll leave in. Don't have energy to update at the moment.] Compute the token types for all literals and rules etc.. There are a few different cases to consider for grammar types and a few situations within. CASE 1 : pure parser grammar a) Any reference to a token gets a token type. b) The tokens section may alias a token name to a string or char CASE 2 : pure lexer grammar a) Import token vocabulary if available. Set token types for any new tokens to values above last imported token type b) token rule definitions get token types if not already defined c) literals do NOT get token types CASE 3 : merged parser / lexer grammar a) Any char or string literal gets a token type in a parser rule b) Any reference to a token gets a token type if not referencing a fragment lexer rule c) The tokens section may alias a token name to a string or char which must add a rule to the lexer d) token rule definitions get token types if not already defined e) token rule definitions may also alias a token name to a literal. E.g., Rule 'FOR : "for";' will alias FOR to "for" in the sense that references to either in the parser grammar will yield the token type What this pass does: 0. Collects basic info about the grammar like grammar name and type; Oh, I have go get the options in case they affect the token types. E.g., tokenVocab option. Imports any token vocab name/type pairs into a local hashtable. 1. Finds a list of all literals and token names. 2. Finds a list of all token name rule definitions; no token rules implies pure parser. 3. Finds a list of all simple token rule defs of form ": ;" and aliases them. 4. Walks token names table and assign types to any unassigned 5. Walks aliases and assign types to referenced literals 6. Walks literals, assigning types if untyped 4. Informs the Grammar object of the type definitions such as: g.defineToken( , ttype); g.defineToken( , ttype); g.defineToken( , ttype); where some of the ttype values will be the same for aliases tokens.
-
-
Field Summary
Fields Modifier and Type Field Description static String[]
_tokenNames
protected static GrammarAST
charAlias
protected static GrammarAST
charAlias2
protected String
currentRuleName
protected Grammar
grammar
protected static GrammarAST
stringAlias
protected static GrammarAST
stringAlias2
-
Fields inherited from class antlr.TreeParser
_retTree, astFactory, ASTNULL, inputState, returnAST, tokenNames, traceDepth
-
Fields inherited from interface org.antlr.grammar.v2.AssignTokenTypesWalkerTokenTypes
ACTION, ACTION_CHAR_LITERAL, ACTION_ESC, ACTION_STRING_LITERAL, ALT, AMPERSAND, ARG, ARG_ACTION, ARGLIST, ASSIGN, BACKTRACK_SEMPRED, BANG, BLOCK, CHAR_LITERAL, CHAR_RANGE, CHARSET, CLOSE_ELEMENT_OPTION, CLOSURE, COLON, COMBINED_GRAMMAR, COMMA, COMMENT, DIGIT, DOC_COMMENT, DOLLAR, DOT, DOUBLE_ANGLE_STRING_LITERAL, DOUBLE_QUOTE_STRING_LITERAL, EOA, EOB, EOF, EOR, EPSILON, ESC, ETC, FORCED_ACTION, FRAGMENT, GATED_SEMPRED, ID, IMPLIES, IMPORT, INITACTION, INT, INTERNAL_RULE_REF, LABEL, LEXER, LEXER_GRAMMAR, LITERAL_catch, LITERAL_finally, LITERAL_grammar, LITERAL_lexer, LITERAL_private, LITERAL_protected, LITERAL_public, LITERAL_returns, LITERAL_throws, LITERAL_tree, LPAREN, ML_COMMENT, NESTED_ACTION, NESTED_ARG_ACTION, NOT, NULL_TREE_LOOKAHEAD, OPEN_ELEMENT_OPTION, OPTIONAL, OPTIONS, OR, PARSER, PARSER_GRAMMAR, PLUS, PLUS_ASSIGN, POSITIVE_CLOSURE, QUESTION, RANGE, RCURLY, RET, REWRITE, ROOT, RPAREN, RULE, RULE_REF, SCOPE, SEMI, SEMPRED, SL_COMMENT, SRC, STAR, STRAY_BRACKET, STRING_LITERAL, SYN_SEMPRED, SYNPRED, TEMPLATE, TOKEN_REF, TOKENS, TREE_BEGIN, TREE_GRAMMAR, WILDCARD, WS, WS_LOOP, WS_OPT, XDIGIT
-
-
Constructor Summary
Constructors Constructor Description AssignTokenTypesWalker()
-
Method Summary
All Methods Instance Methods Concrete Methods Modifier and Type Method Description protected void
alias(GrammarAST t, GrammarAST s)
protected void
aliasTokenIDsAndLiterals(Grammar root)
void
alternative(antlr.collections.AST _t)
protected void
assignStringTypes(Grammar root)
protected void
assignTokenIDTypes(Grammar root)
void
ast_suffix(antlr.collections.AST _t)
void
atom(antlr.collections.AST _t)
void
attrScope(antlr.collections.AST _t)
void
block(antlr.collections.AST _t)
void
charSet(antlr.collections.AST _t)
void
charSetElement(antlr.collections.AST _t)
protected void
defineStringLiteralsFromDelegates()
protected void
defineTokenNamesAndLiteralsInGrammar(Grammar root)
void
defineTokens(Grammar root)
void
delegateGrammars(antlr.collections.AST _t)
void
ebnf(antlr.collections.AST _t)
void
element(antlr.collections.AST _t)
void
exceptionGroup(antlr.collections.AST _t)
void
exceptionHandler(antlr.collections.AST _t)
void
finallyClause(antlr.collections.AST _t)
void
grammar(antlr.collections.AST _t, Grammar g)
void
grammarSpec(antlr.collections.AST _t)
protected void
init(Grammar root)
protected void
initASTPatterns()
void
modifier(antlr.collections.AST _t)
void
option(antlr.collections.AST _t, Map opts)
Map
optionsSpec(antlr.collections.AST _t)
Object
optionValue(antlr.collections.AST _t)
void
reportError(antlr.RecognitionException ex)
void
rewrite(antlr.collections.AST _t)
void
rule(antlr.collections.AST _t)
void
rules(antlr.collections.AST _t)
void
ruleScopeSpec(antlr.collections.AST _t)
void
tokenSpec(antlr.collections.AST _t)
void
tokensSpec(antlr.collections.AST _t)
protected void
trackString(GrammarAST t)
protected void
trackToken(GrammarAST t)
protected void
trackTokenRule(GrammarAST t, GrammarAST modifier, GrammarAST block)
void
tree(antlr.collections.AST _t)
-
-
-
Field Detail
-
grammar
protected Grammar grammar
-
currentRuleName
protected String currentRuleName
-
stringAlias
protected static GrammarAST stringAlias
-
charAlias
protected static GrammarAST charAlias
-
stringAlias2
protected static GrammarAST stringAlias2
-
charAlias2
protected static GrammarAST charAlias2
-
_tokenNames
public static final String[] _tokenNames
-
-
Method Detail
-
reportError
public void reportError(antlr.RecognitionException ex)
- Overrides:
reportError
in classantlr.TreeParser
-
initASTPatterns
protected void initASTPatterns()
-
trackString
protected void trackString(GrammarAST t)
-
trackToken
protected void trackToken(GrammarAST t)
-
trackTokenRule
protected void trackTokenRule(GrammarAST t, GrammarAST modifier, GrammarAST block)
-
alias
protected void alias(GrammarAST t, GrammarAST s)
-
defineTokens
public void defineTokens(Grammar root)
-
defineStringLiteralsFromDelegates
protected void defineStringLiteralsFromDelegates()
-
assignStringTypes
protected void assignStringTypes(Grammar root)
-
aliasTokenIDsAndLiterals
protected void aliasTokenIDsAndLiterals(Grammar root)
-
assignTokenIDTypes
protected void assignTokenIDTypes(Grammar root)
-
defineTokenNamesAndLiteralsInGrammar
protected void defineTokenNamesAndLiteralsInGrammar(Grammar root)
-
init
protected void init(Grammar root)
-
grammar
public final void grammar(antlr.collections.AST _t, Grammar g) throws antlr.RecognitionException
- Throws:
antlr.RecognitionException
-
grammarSpec
public final void grammarSpec(antlr.collections.AST _t) throws antlr.RecognitionException
- Throws:
antlr.RecognitionException
-
optionsSpec
public final Map optionsSpec(antlr.collections.AST _t) throws antlr.RecognitionException
- Throws:
antlr.RecognitionException
-
delegateGrammars
public final void delegateGrammars(antlr.collections.AST _t) throws antlr.RecognitionException
- Throws:
antlr.RecognitionException
-
tokensSpec
public final void tokensSpec(antlr.collections.AST _t) throws antlr.RecognitionException
- Throws:
antlr.RecognitionException
-
attrScope
public final void attrScope(antlr.collections.AST _t) throws antlr.RecognitionException
- Throws:
antlr.RecognitionException
-
rules
public final void rules(antlr.collections.AST _t) throws antlr.RecognitionException
- Throws:
antlr.RecognitionException
-
option
public final void option(antlr.collections.AST _t, Map opts) throws antlr.RecognitionException
- Throws:
antlr.RecognitionException
-
optionValue
public final Object optionValue(antlr.collections.AST _t) throws antlr.RecognitionException
- Throws:
antlr.RecognitionException
-
charSet
public final void charSet(antlr.collections.AST _t) throws antlr.RecognitionException
- Throws:
antlr.RecognitionException
-
charSetElement
public final void charSetElement(antlr.collections.AST _t) throws antlr.RecognitionException
- Throws:
antlr.RecognitionException
-
tokenSpec
public final void tokenSpec(antlr.collections.AST _t) throws antlr.RecognitionException
- Throws:
antlr.RecognitionException
-
rule
public final void rule(antlr.collections.AST _t) throws antlr.RecognitionException
- Throws:
antlr.RecognitionException
-
modifier
public final void modifier(antlr.collections.AST _t) throws antlr.RecognitionException
- Throws:
antlr.RecognitionException
-
ruleScopeSpec
public final void ruleScopeSpec(antlr.collections.AST _t) throws antlr.RecognitionException
- Throws:
antlr.RecognitionException
-
block
public final void block(antlr.collections.AST _t) throws antlr.RecognitionException
- Throws:
antlr.RecognitionException
-
exceptionGroup
public final void exceptionGroup(antlr.collections.AST _t) throws antlr.RecognitionException
- Throws:
antlr.RecognitionException
-
alternative
public final void alternative(antlr.collections.AST _t) throws antlr.RecognitionException
- Throws:
antlr.RecognitionException
-
rewrite
public final void rewrite(antlr.collections.AST _t) throws antlr.RecognitionException
- Throws:
antlr.RecognitionException
-
element
public final void element(antlr.collections.AST _t) throws antlr.RecognitionException
- Throws:
antlr.RecognitionException
-
exceptionHandler
public final void exceptionHandler(antlr.collections.AST _t) throws antlr.RecognitionException
- Throws:
antlr.RecognitionException
-
finallyClause
public final void finallyClause(antlr.collections.AST _t) throws antlr.RecognitionException
- Throws:
antlr.RecognitionException
-
atom
public final void atom(antlr.collections.AST _t) throws antlr.RecognitionException
- Throws:
antlr.RecognitionException
-
ebnf
public final void ebnf(antlr.collections.AST _t) throws antlr.RecognitionException
- Throws:
antlr.RecognitionException
-
tree
public final void tree(antlr.collections.AST _t) throws antlr.RecognitionException
- Throws:
antlr.RecognitionException
-
ast_suffix
public final void ast_suffix(antlr.collections.AST _t) throws antlr.RecognitionException
- Throws:
antlr.RecognitionException
-
-