|
||||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | |||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |
java.lang.Objectde.l3s.boilerpipe.util.UnicodeTokenizer
public class UnicodeTokenizer
Tokenizes text according to Unicode word boundaries and strips off non-word characters.
Constructor Summary | |
---|---|
UnicodeTokenizer()
|
Method Summary | |
---|---|
static java.lang.String[] |
tokenize(java.lang.CharSequence text)
Tokenizes the text and returns an array of tokens. |
Methods inherited from class java.lang.Object |
---|
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait |
Constructor Detail |
---|
public UnicodeTokenizer()
Method Detail |
---|
public static java.lang.String[] tokenize(java.lang.CharSequence text)
text
- The text
|
||||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | |||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |