public abstract class CharTokenizer extends Tokenizer
Constructor and Description |
---|
CharTokenizer(Reader input) |
Modifier and Type | Method and Description |
---|---|
protected abstract boolean |
isTokenChar(char c)
Returns true iff a character should be included in a token.
|
Token |
next()
Returns the next token in the stream, or null at EOS.
|
protected char |
normalize(char c)
Called on each token character to normalize it before it is added to the
token.
|
public CharTokenizer(Reader input)
protected abstract boolean isTokenChar(char c)
protected char normalize(char c)
public final Token next() throws IOException
next
in class TokenStream
IOException
Copyright © 2000-2013 Apache Software Foundation. All Rights Reserved.