#include <set>
#include <map>
#include <list>
#include "fries/language.h"
#include "regexp-pcre++.h"
Go to the source code of this file.
Classes | |
class | tokenizer |
Class tokenizer implements a token splitter, which converts a string into a sequence of word objects, according to a set of tokenization rules read from aconfiguration file. More... |