tokenizer.h File Reference

#include <set>
#include <map>
#include <list>
#include "fries/language.h"
#include "regexp-pcre++.h"
Include dependency graph for tokenizer.h:
This graph shows which files directly or indirectly include this file:

Go to the source code of this file.

Classes

class  tokenizer
 Class tokenizer implements a token splitter, which converts a string into a sequence of word objects, according to a set of tokenization rules read from aconfiguration file. More...
Generated on Tue Jul 27 16:29:30 2010 for FreeLing by  doxygen 1.6.3