antlr
Class TokenStreamRewriteEngine
public
class
TokenStreamRewriteEngine
extends Object
implements TokenStream, IASDebugStream
This token stream tracks the *entire* token stream coming from
a lexer, but does not pass on the whitespace (or whatever else
you want to discard) to the parser.
This class can then be asked for the ith token in the input stream.
Useful for dumping out the input stream exactly after doing some
augmentation or other manipulations. Tokens are index from 0..n-1
You can insert stuff, replace, and delete chunks. Note that the
operations are done lazily--only if you convert the buffer to a
String. This is very efficient because you are not moving data around
all the time. As the buffer of tokens is converted to strings, the
toString() method(s) check to see if there is an operation at the
current index. If so, the operation is done and then normal String
rendering continues on the buffer. This is like having multiple Turing
machine instruction streams (programs) operating on a single input tape. :)
Since the operations are done lazily at toString-time, operations do not
screw up the token index values. That is, an insert operation at token
index i does not change the index values for tokens i+1..n-1.
Because operations never actually alter the buffer, you may always get
the original token stream back without undoing anything. Since
the instructions are queued up, you can easily simulate transactions and
roll back any changes if there is an error just by removing instructions.
For example,
TokenStreamRewriteEngine rewriteEngine =
new TokenStreamRewriteEngine(lexer);
JavaRecognizer parser = new JavaRecognizer(rewriteEngine);
...
rewriteEngine.insertAfter("pass1", t, "foobar");}
rewriteEngine.insertAfter("pass2", u, "start");}
System.out.println(rewriteEngine.toString("pass1"));
System.out.println(rewriteEngine.toString("pass2"));
You can also have multiple "instruction streams" and get multiple
rewrites from a single pass over the input. Just name the instruction
streams and use that name again when printing the buffer. This could be
useful for generating a C file and also its header file--all from the
same buffer.
If you don't use named rewrite streams, a "default" stream is used.
Terence Parr, parrt at antlr.org
University of San Francisco
February 2004
Method Summary |
protected void | addToSortedRewriteList(TokenStreamRewriteEngine.RewriteOperation op) If op.index > lastRewriteTokenIndexes, just add to the end.
|
protected void | addToSortedRewriteList(String programName, TokenStreamRewriteEngine.RewriteOperation op) Add an instruction to the rewrite instruction list ordered by
the instruction number (use a binary search for efficiency).
|
void | delete(int index) |
void | delete(int from, int to) |
void | delete(Token indexT) |
void | delete(Token from, Token to) |
void | delete(String programName, int from, int to) |
void | delete(String programName, Token from, Token to) |
void | deleteProgram() |
void | deleteProgram(String programName) Reset the program so that no instructions exist |
void | discard(int ttype) |
String | getEntireText() |
int | getLastRewriteTokenIndex() |
protected int | getLastRewriteTokenIndex(String programName) |
TokenOffsetInfo | getOffsetInfo(Token token) |
protected List | getProgram(String name) |
TokenWithIndex | getToken(int i) |
int | getTokenStreamSize() |
int | index() |
void | insertAfter(Token t, String text) |
void | insertAfter(int index, String text) |
void | insertAfter(String programName, Token t, String text) |
void | insertAfter(String programName, int index, String text) |
void | insertBefore(Token t, String text) |
void | insertBefore(int index, String text) |
void | insertBefore(String programName, Token t, String text) |
void | insertBefore(String programName, int index, String text) |
Token | nextToken() |
void | replace(int index, String text) |
void | replace(int from, int to, String text) |
void | replace(Token indexT, String text) |
void | replace(Token from, Token to, String text) |
void | replace(String programName, int from, int to, String text) |
void | replace(String programName, Token from, Token to, String text) |
void | rollback(int instructionIndex) |
void | rollback(String programName, int instructionIndex) Rollback the instruction stream for a program so that
the indicated instruction (via instructionIndex) is no
longer in the stream. |
protected void | setLastRewriteTokenIndex(String programName, int i) |
int | size() |
String | toDebugString() |
String | toDebugString(int start, int end) |
String | toOriginalString() |
String | toOriginalString(int start, int end) |
String | toString() |
String | toString(String programName) |
String | toString(int start, int end) |
String | toString(String programName, int start, int end) |
Which (whitespace) token(s) to throw out
public static final String DEFAULT_PROGRAM_NAME
protected int index
track index of tokens
protected Map lastRewriteTokenIndexes
Map String (program name) -> Integer index
public static final int MIN_TOKEN_INDEX
protected Map programs
You may have multiple, named streams of rewrite operations.
I'm calling these things "programs."
Maps String (name) -> rewrite (List)
public static final int PROGRAM_INIT_SIZE
Who do we suck tokens from?
protected List tokens
Track the incoming list of tokens
public TokenStreamRewriteEngine(
TokenStream upstream, int initialSize)
If op.index > lastRewriteTokenIndexes, just add to the end.
Otherwise, do linear
Add an instruction to the rewrite instruction list ordered by
the instruction number (use a binary search for efficiency).
The list is ordered so that toString() can be done efficiently.
When there are multiple instructions at the same index, the instructions
must be ordered to ensure proper behavior. For example, a delete at
index i must kill any replace operation at i. Insert-before operations
must come before any replace / delete instructions. If there are
multiple insert instructions for a single index, they are done in
reverse insertion order so that "insert foo" then "insert bar" yields
"foobar" in front rather than "barfoo". This is convenient because
I can insert new InsertOp instructions at the index returned by
the binary search. A ReplaceOp kills any previous replace op. Since
delete is the same as replace with null text, i can check for
ReplaceOp and cover DeleteOp at same time. :)
public void delete(int index)
public void delete(int from, int to)
public void delete(
Token indexT)
public void delete(String programName, int from, int to)
public void delete(String programName,
Token from,
Token to)
public void deleteProgram()
public void deleteProgram(String programName)
Reset the program so that no instructions exist
public void discard(int ttype)
public String getEntireText()
public int getLastRewriteTokenIndex()
protected int getLastRewriteTokenIndex(String programName)
protected List getProgram(String name)
public int getTokenStreamSize()
public int index()
public void insertAfter(
Token t, String text)
public void insertAfter(int index, String text)
public void insertAfter(String programName,
Token t, String text)
public void insertAfter(String programName, int index, String text)
public void insertBefore(
Token t, String text)
public void insertBefore(int index, String text)
public void insertBefore(String programName,
Token t, String text)
public void insertBefore(String programName, int index, String text)
public void replace(int index, String text)
public void replace(int from, int to, String text)
public void replace(
Token indexT, String text)
public void replace(
Token from,
Token to, String text)
public void replace(String programName, int from, int to, String text)
public void replace(String programName,
Token from,
Token to, String text)
public void rollback(int instructionIndex)
public void rollback(String programName, int instructionIndex)
Rollback the instruction stream for a program so that
the indicated instruction (via instructionIndex) is no
longer in the stream. UNTESTED!
protected void setLastRewriteTokenIndex(String programName, int i)
public int size()
public String toDebugString()
public String toDebugString(int start, int end)
public String toOriginalString()
public String toOriginalString(int start, int end)
public String toString()
public String toString(String programName)
public String toString(int start, int end)
public String toString(String programName, int start, int end)