public class StandardTokenizer extends Tokenizer implements StandardTokenizerConstants
This should be a good tokenizer for most European-language documents:
Many applications have specific tokenizer needs. If this tokenizer does not suit your application, please consider copying this source code directory to your project and maintaining your own grammar-based tokenizer.
Modifier and Type | Field and Description |
---|---|
Token |
jj_nt |
Token |
token |
StandardTokenizerTokenManager |
token_source |
Constructor and Description |
---|
StandardTokenizer(CharStream stream) |
StandardTokenizer(Reader reader)
Constructs a tokenizer for this Reader.
|
StandardTokenizer(StandardTokenizerTokenManager tm) |
Modifier and Type | Method and Description |
---|---|
void |
disable_tracing() |
void |
enable_tracing() |
ParseException |
generateParseException() |
Token |
getNextToken() |
Token |
getToken(int index) |
Token |
next()
Returns the next token in the stream, or null at EOS.
|
void |
ReInit(CharStream stream) |
void |
ReInit(StandardTokenizerTokenManager tm) |
public StandardTokenizerTokenManager token_source
public Token token
public Token jj_nt
public StandardTokenizer(Reader reader)
public StandardTokenizer(CharStream stream)
public StandardTokenizer(StandardTokenizerTokenManager tm)
public final Token next() throws ParseException, IOException
The returned token's type is set to an element of StandardTokenizerConstants.tokenImage
.
next
in class TokenStream
ParseException
IOException
public void ReInit(CharStream stream)
public void ReInit(StandardTokenizerTokenManager tm)
public final Token getNextToken()
public final Token getToken(int index)
public ParseException generateParseException()
public final void enable_tracing()
public final void disable_tracing()
Copyright © 2000-2013 Apache Software Foundation. All Rights Reserved.