1
2
3
4
5
6
7
8
9 """
10 Functions for detecting a token's X{features}. Features are stored in
11 a dictionary which maps feature names to feature values.
12
13 (Not yet ported from NLTK: A X{feature encoder} can then be used to
14 translate the feature dictionary into a homogenous representation
15 (such as a sparse boolean list), suitable for use with other
16 processing tasks.)
17 """
18
20 """
21 Return a feature detector that applies the supplied functions
22 to each token.
23
24 @type functions: dictionary of functions
25 @param properties: one or more functions in one string argument to compute
26 the features.
27 """
28
29 return lambda tokens: list((feature,function(tokens)) for (feature, function) in functions.items())
30
31
33 """
34 takes a string
35 returns a list of tuples (feature type, feature value)
36 """
37
38
40 return feature({'text': lambda t:t})
41
43 return feature({'stem': stemmer})
44
45
46
47
48
49
50
51
52
61
62 if __name__ == '__main__': demo()
63