A simple tokenizer used to parse the keywords. More...
#include <ikeywords.h>
Public Member Functions | |
virtual int | operator() (std::vector< const char * > &tkns, char *buf) |
Tokenizer. | |
tokenizer (const char *d=ibis::util::delimiters) | |
Constructor. | |
virtual | ~tokenizer () |
Destructor. |
A simple tokenizer used to parse the keywords.
ibis::keywords::tokenizer::tokenizer | ( | const char * | d = ibis::util::delimiters | ) | [inline] |
Constructor.
It takes a list of delimiters. Any character in the list of delimiters will terminate a token. If no delimiter is given, anything other than alphanumerical characters will terminate a token. By default, the delimiters defined in ibis::util::delimiters are used.
int ibis::keywords::tokenizer::operator() | ( | std::vector< const char * > & | tkns, |
char * | buf | ||
) | [virtual] |
Tokenizer.
Turn the buffer buf into a list of tokens based on the following rules.
This function returns 0 in normal cases.
Implements ibis::text::tokenizer.
![]() |