本文整理汇总了Python中nltk.TreebankWordTokenizer方法的典型用法代码示例。如果您正苦于以下问题：Python nltk.TreebankWordTokenizer方法的具体用法？Python nltk.TreebankWordTokenizer怎么用？Python nltk.TreebankWordTokenizer使用的例子？那么恭喜您, 这里精选的方法代码示例或许可以为您提供帮助。您也可以进一步了解该方法所在类nltk的用法示例。
在下文中一共展示了nltk.TreebankWordTokenizer方法的4个代码示例，这些例子默认根据受欢迎程度排序。您可以为喜欢或者感觉有用的代码点赞，您的评价将有助于系统推荐出更棒的Python代码示例。
示例1: __init__
# 需要导入模块: import nltk [as 别名]
# 或者: from nltk import TreebankWordTokenizer [as 别名]
def __init__(self):
self._word_tokenizer = nltk.TreebankWordTokenizer()
if FLAGS.punkt_tokenizer_file is not None:
self._sent_tokenizer = py_utils.load_pickle(FLAGS.punkt_tokenizer_file)
else:
self._sent_tokenizer = nltk.load("tokenizers/punkt/english.pickle") 示例2: _treebank_en
# 需要导入模块: import nltk [as 别名]
# 或者: from nltk import TreebankWordTokenizer [as 别名]
def _treebank_en(self, text):
if self.word_tokenizer is None:
import nltk
self.word_tokenizer = nltk.TreebankWordTokenizer()
return [
token.replace("''", '"').replace("``", '"')
for token in self.word_tokenizer.tokenize(text)
] 示例3: tokenize
# 需要导入模块: import nltk [as 别名]
# 或者: from nltk import TreebankWordTokenizer [as 别名]
def tokenize(self, text):
return TreebankWordTokenizer().tokenize(text) 示例4: __init__
# 需要导入模块: import nltk [as 别名]
# 或者: from nltk import TreebankWordTokenizer [as 别名]
def __init__(self):
self.sent_tokenzier = nltk.load('tokenizers/punkt/english.pickle')
self.word_tokenizer = nltk.TreebankWordTokenizer()