本文整理汇总了Python中CharniakParser.tokenize方法的典型用法代码示例。如果您正苦于以下问题：Python CharniakParser.tokenize方法的具体用法？Python CharniakParser.tokenize怎么用？Python CharniakParser.tokenize使用的例子？那么, 这里精选的方法代码示例或许可以为您提供帮助。您也可以进一步了解该方法所在类CharniakParser的用法示例。
在下文中一共展示了CharniakParser.tokenize方法的3个代码示例，这些例子默认根据受欢迎程度排序。您可以为喜欢或者感觉有用的代码点赞，您的评价将有助于系统推荐出更棒的Python代码示例。
示例1: __init__
# 需要导入模块: import CharniakParser [as 别名]
# 或者: from CharniakParser import tokenize [as 别名]
def __init__(self, text_or_tokens, max_sentence_length=399):
if isinstance(text_or_tokens, Sentence):
self.sentrep = text_or_tokens.sentrep
elif isinstance(text_or_tokens, basestring):
self.sentrep = parser.tokenize('<s> ' + text_or_tokens + ' </s>',
max_sentence_length)
else:
self.sentrep = parser.SentRep(text_or_tokens)示例2: __init__
# 需要导入模块: import CharniakParser [as 别名]
# 或者: from CharniakParser import tokenize [as 别名]
def __init__(self, text_or_tokens, max_sentence_length=399):
if isinstance(text_or_tokens, Sentence):
self.sentrep = text_or_tokens.sentrep
elif isinstance(text_or_tokens, basestring):
self.sentrep = parser.tokenize('<s> ' + text_or_tokens + ' </s>',
max_sentence_length)
else:
# text_or_tokens is a sequence -- need to make sure that each
# element is a string to avoid crashing
text_or_tokens = map(str, text_or_tokens)
self.sentrep = parser.SentRep(text_or_tokens)示例3: __init__
# 需要导入模块: import CharniakParser [as 别名]
# 或者: from CharniakParser import tokenize [as 别名]
def __init__(self, text_or_tokens):
if isinstance(text_or_tokens, Sentence):
self.sentrep = text_or_tokens.sentrep
elif isinstance(text_or_tokens, basestring):
self.sentrep = parser.tokenize('<s> ' + text_or_tokens + ' </s>')
else:
# text_or_tokens is a sequence -- need to make sure that each
# element is a string to avoid crashing
text_or_tokens = [parser.ptbEscape(str(token))
for token in text_or_tokens]
self.sentrep = parser.SentRep(text_or_tokens)