KDevelop API Documentation

TokenBuffer Class Reference

A Stream of Token objects fed to the parser from a TokenStream that can be rewound via mark()/rewind() methods. More...

#include <TokenBuffer.hpp>

Collaboration diagram for TokenBuffer:

Collaboration graph
[legend]
List of all members.

Public Member Functions

 TokenBuffer (TokenStream &input_)
 Create a token buffer.
void reset (void)
 Reset the input buffer to empty state.
int LA (int i)
 Get a lookahead token value.
RefToken LT (int i)
 Get a lookahead token.
int mark ()
 Return an integer marker that can be used to rewind the buffer to its current state.
void rewind (int mark)
 Rewind the token buffer to a marker.
void consume ()
 Mark another token for deferred consumption.

Protected Attributes

TokenStreaminput

Private Member Functions

void fill (int amount)
 Ensure that the token buffer is sufficiently full.
void syncConsume ()
 Sync up deferred consumption.
 TokenBuffer (const TokenBuffer &other)
const TokenBufferoperator= (const TokenBuffer &other)

Private Attributes

int nMarkers
int markerOffset
int numToConsume
CircularQueue< RefTokenqueue

Detailed Description

A Stream of Token objects fed to the parser from a TokenStream that can be rewound via mark()/rewind() methods.

A dynamic array is used to buffer up all the input tokens. Normally, "k" tokens are stored in the buffer. More tokens may be stored during guess mode (testing syntactic predicate), or when LT(i>k) is referenced. Consumption of tokens is deferred. In other words, reading the next token is not done by conume(), but deferred until needed by LA or LT.

Todo:
see if we can integrate this one with InputBuffer into one template or so.
See also:
antlr.Token

antlr.TokenStream

antlr.TokenQueue

Definition at line 36 of file TokenBuffer.hpp.


Constructor & Destructor Documentation

TokenBuffer::TokenBuffer TokenStream input_  ) 
 

Create a token buffer.

Definition at line 30 of file TokenBuffer.cpp.

TokenBuffer::TokenBuffer const TokenBuffer other  )  [private]
 


Member Function Documentation

void TokenBuffer::consume  )  [inline]
 

Mark another token for deferred consumption.

Definition at line 67 of file TokenBuffer.hpp.

References numToConsume.

void TokenBuffer::fill int  amount  )  [private]
 

Ensure that the token buffer is sufficiently full.

Definition at line 36 of file TokenBuffer.cpp.

References CircularQueue< T >::append(), CircularQueue< T >::entries(), input, TokenStream::nextToken(), queue, and syncConsume().

Referenced by LA(), and LT().

int TokenBuffer::LA int  i  ) 
 

Get a lookahead token value.

Definition at line 47 of file TokenBuffer.cpp.

References CircularQueue< T >::elementAt(), fill(), markerOffset, and queue.

RefToken TokenBuffer::LT int  i  ) 
 

Get a lookahead token.

Definition at line 54 of file TokenBuffer.cpp.

References CircularQueue< T >::elementAt(), fill(), markerOffset, queue, and RefToken.

int TokenBuffer::mark  ) 
 

Return an integer marker that can be used to rewind the buffer to its current state.

Definition at line 63 of file TokenBuffer.cpp.

References nMarkers, and syncConsume().

const TokenBuffer& TokenBuffer::operator= const TokenBuffer other  )  [private]
 

void TokenBuffer::reset void   )  [inline]
 

Reset the input buffer to empty state.

Definition at line 42 of file TokenBuffer.hpp.

References CircularQueue< T >::clear(), markerOffset, nMarkers, numToConsume, and queue.

Referenced by ParserInputState::reset().

void TokenBuffer::rewind int  mark  ) 
 

Rewind the token buffer to a marker.

Parameters:
mark Marker returned previously from mark()

Definition at line 73 of file TokenBuffer.cpp.

References markerOffset, nMarkers, and syncConsume().

void TokenBuffer::syncConsume  )  [inline, private]
 

Sync up deferred consumption.

Definition at line 100 of file TokenBuffer.hpp.

References markerOffset, nMarkers, numToConsume, queue, and CircularQueue< T >::removeItems().

Referenced by fill(), mark(), and rewind().


Member Data Documentation

TokenStream& TokenBuffer::input [protected]
 

Definition at line 79 of file TokenBuffer.hpp.

Referenced by fill().

int TokenBuffer::markerOffset [private]
 

Definition at line 86 of file TokenBuffer.hpp.

Referenced by LA(), LT(), reset(), rewind(), and syncConsume().

int TokenBuffer::nMarkers [private]
 

Definition at line 83 of file TokenBuffer.hpp.

Referenced by mark(), reset(), rewind(), and syncConsume().

int TokenBuffer::numToConsume [private]
 

Definition at line 89 of file TokenBuffer.hpp.

Referenced by consume(), reset(), and syncConsume().

CircularQueue<RefToken> TokenBuffer::queue [private]
 

Definition at line 92 of file TokenBuffer.hpp.

Referenced by fill(), LA(), LT(), reset(), and syncConsume().


The documentation for this class was generated from the following files:
KDE Logo
This file is part of the documentation for KDevelop Version 3.1.2.
Documentation copyright © 1996-2004 the KDE developers.
Generated on Wed Mar 23 00:31:05 2005 by doxygen 1.3.9.1 written by Dimitri van Heesch, © 1997-2003