TokenBuffer Class Reference
A Stream of Token objects fed to the parser from a TokenStream that can be rewound via mark()/rewind() methods. More...
#include <TokenBuffer.hpp>
Public Member Functions | |
TokenBuffer (TokenStream &input_) | |
Create a token buffer. | |
void | reset (void) |
Reset the input buffer to empty state. | |
int | LA (int i) |
Get a lookahead token value. | |
RefToken | LT (int i) |
Get a lookahead token. | |
int | mark () |
Return an integer marker that can be used to rewind the buffer to its current state. | |
void | rewind (int mark) |
Rewind the token buffer to a marker. | |
void | consume () |
Mark another token for deferred consumption. | |
Protected Attributes | |
TokenStream & | input |
Private Member Functions | |
void | fill (int amount) |
Ensure that the token buffer is sufficiently full. | |
void | syncConsume () |
Sync up deferred consumption. | |
TokenBuffer (const TokenBuffer &other) | |
const TokenBuffer & | operator= (const TokenBuffer &other) |
Private Attributes | |
int | nMarkers |
int | markerOffset |
int | numToConsume |
CircularQueue< RefToken > | queue |
Detailed Description
A Stream of Token objects fed to the parser from a TokenStream that can be rewound via mark()/rewind() methods.A dynamic array is used to buffer up all the input tokens. Normally, "k" tokens are stored in the buffer. More tokens may be stored during guess mode (testing syntactic predicate), or when LT(i>k) is referenced. Consumption of tokens is deferred. In other words, reading the next token is not done by conume(), but deferred until needed by LA or LT.
- Todo:
- see if we can integrate this one with InputBuffer into one template or so.
- See also:
- antlr.Token
antlr.TokenStream
antlr.TokenQueue
Definition at line 36 of file TokenBuffer.hpp.
Constructor & Destructor Documentation
|
Create a token buffer.
Definition at line 30 of file TokenBuffer.cpp. References TokenBuffer(). Referenced by TokenBuffer(). |
|
|
Member Function Documentation
|
Mark another token for deferred consumption.
Definition at line 67 of file TokenBuffer.hpp. |
|
Ensure that the token buffer is sufficiently full.
Definition at line 36 of file TokenBuffer.cpp. References CircularQueue< RefToken >::append(), CircularQueue< RefToken >::entries(), fill(), input, markerOffset, TokenStream::nextToken(), queue, and syncConsume(). |
|
Get a lookahead token value.
Definition at line 47 of file TokenBuffer.cpp. References CircularQueue< RefToken >::elementAt(), fill(), LA(), markerOffset, and queue. Referenced by LA(). |
|
Get a lookahead token.
Definition at line 54 of file TokenBuffer.cpp. References CircularQueue< RefToken >::elementAt(), fill(), LT(), markerOffset, and queue. Referenced by LT(). |
|
Return an integer marker that can be used to rewind the buffer to its current state.
Definition at line 63 of file TokenBuffer.cpp. References markerOffset, nMarkers, and syncConsume(). |
|
|
|
Reset the input buffer to empty state.
Definition at line 42 of file TokenBuffer.hpp. |
|
Rewind the token buffer to a marker.
Definition at line 73 of file TokenBuffer.cpp. References markerOffset, nMarkers, rewind(), and syncConsume(). Referenced by rewind(). |
|
Sync up deferred consumption.
Definition at line 100 of file TokenBuffer.hpp. References markerOffset, nMarkers, numToConsume, queue, and CircularQueue< RefToken >::removeItems(). |
Member Data Documentation
|
Definition at line 79 of file TokenBuffer.hpp. Referenced by fill(). |
|
Definition at line 86 of file TokenBuffer.hpp. Referenced by fill(), LA(), LT(), mark(), rewind(), and syncConsume(). |
|
Definition at line 83 of file TokenBuffer.hpp. Referenced by mark(), rewind(), and syncConsume(). |
|
Definition at line 89 of file TokenBuffer.hpp. Referenced by syncConsume(). |
|
Definition at line 92 of file TokenBuffer.hpp. Referenced by fill(), LA(), LT(), and syncConsume(). |
The documentation for this class was generated from the following files:
- lib/antlr/antlr/TokenBuffer.hpp
- lib/antlr/src/TokenBuffer.cpp