本文整理匯總了Python中nltk.corpus.treebank.tokenized方法的典型用法代碼示例。如果您正苦於以下問題：Python treebank.tokenized方法的具體用法？Python treebank.tokenized怎麽用？Python treebank.tokenized使用的例子？那麽, 這裏精選的方法代碼示例或許可以為您提供幫助。您也可以進一步了解該方法所在類nltk.corpus.treebank的用法示例。
在下文中一共展示了treebank.tokenized方法的1個代碼示例，這些例子默認根據受歡迎程度排序。您可以為喜歡或者感覺有用的代碼點讚，您的評價將有助於係統推薦出更棒的Python代碼示例。
示例1: pcfg_demo
# 需要導入模塊: from nltk.corpus import treebank [as 別名]
# 或者: from nltk.corpus.treebank import tokenized [as 別名]
def pcfg_demo():
"""
A demonstration showing how a ``PCFG`` can be created and used.
"""
from nltk.corpus import treebank
from nltk import treetransforms
from nltk import induce_pcfg
from nltk.parse import pchart
pcfg_prods = toy_pcfg1.productions()
pcfg_prod = pcfg_prods[2]
print('A PCFG production:', repr(pcfg_prod))
print(' pcfg_prod.lhs() =>', repr(pcfg_prod.lhs()))
print(' pcfg_prod.rhs() =>', repr(pcfg_prod.rhs()))
print(' pcfg_prod.prob() =>', repr(pcfg_prod.prob()))
print()
grammar = toy_pcfg2
print('A PCFG grammar:', repr(grammar))
print(' grammar.start() =>', repr(grammar.start()))
print(' grammar.productions() =>', end=' ')
# Use .replace(...) is to line-wrap the output.
print(repr(grammar.productions()).replace(',', ',\n'+' '*26))
print()
# extract productions from three trees and induce the PCFG
print("Induce PCFG grammar from treebank data:")
productions = []
item = treebank._fileids[0]
for tree in treebank.parsed_sents(item)[:3]:
# perform optional tree transformations, e.g.:
tree.collapse_unary(collapsePOS = False)
tree.chomsky_normal_form(horzMarkov = 2)
productions += tree.productions()
S = Nonterminal('S')
grammar = induce_pcfg(S, productions)
print(grammar)
print()
print("Parse sentence using induced grammar:")
parser = pchart.InsideChartParser(grammar)
parser.trace(3)
# doesn't work as tokens are different:
#sent = treebank.tokenized('wsj_0001.mrg')[0]
sent = treebank.parsed_sents(item)[0].leaves()
print(sent)
for parse in parser.parse(sent):
print(parse)