本文整理汇总了Python中torchtext.vocab.freqs方法的典型用法代码示例。如果您正苦于以下问题：Python vocab.freqs方法的具体用法？Python vocab.freqs怎么用？Python vocab.freqs使用的例子？那么, 这里精选的方法代码示例或许可以为您提供帮助。您也可以进一步了解该方法所在类torchtext.vocab的用法示例。
在下文中一共展示了vocab.freqs方法的7个代码示例，这些例子默认根据受欢迎程度排序。您可以为喜欢或者感觉有用的代码点赞，您的评价将有助于系统推荐出更棒的Python代码示例。
示例1: merge_vocabs
# 需要导入模块: from torchtext import vocab [as 别名]
# 或者: from torchtext.vocab import freqs [as 别名]
def merge_vocabs(vocabs, vocab_size=None):
"""
Merge individual vocabularies (assumed to be generated from disjoint
documents) into a larger vocabulary.
Args:
vocabs: `torchtext.vocab.Vocab` vocabularies to be merged
vocab_size: `int` the final vocabulary size. `None` for no limit.
Return:
`torchtext.vocab.Vocab`
"""
merged = sum([vocab.freqs for vocab in vocabs], Counter())
return torchtext.vocab.Vocab(merged,
specials=[UNK_WORD, PAD_WORD,
BOS_WORD, EOS_WORD],
max_size=vocab_size) 示例2: merge_vocabs
# 需要导入模块: from torchtext import vocab [as 别名]
# 或者: from torchtext.vocab import freqs [as 别名]
def merge_vocabs(vocabs, min_freq=0, vocab_size=None):
"""
Merge individual vocabularies (assumed to be generated from disjoint
documents) into a larger vocabulary.
Args:
vocabs: `torchtext.vocab.Vocab` vocabularies to be merged
vocab_size: `int` the final vocabulary size. `None` for no limit.
Return:
`torchtext.vocab.Vocab`
"""
merged = Counter()
for vocab in vocabs:
merged += filter_counter(vocab.freqs, min_freq)
return torchtext.vocab.Vocab(merged,
specials=list(special_token_list),
max_size=vocab_size, min_freq=min_freq) 示例3: merge_vocabs
# 需要导入模块: from torchtext import vocab [as 别名]
# 或者: from torchtext.vocab import freqs [as 别名]
def merge_vocabs(vocabs, vocab_size=None):
"""
Merge individual vocabularies (assumed to be generated from disjoint
documents) into a larger vocabulary.
Args:
vocabs: `torchtext.vocab.Vocab` vocabularies to be merged
vocab_size: `int` the final vocabulary size. `None` for no limit.
Return:
`torchtext.vocab.Vocab`
"""
merged = Counter()
for vocab in vocabs:
merged += vocab.freqs
return torchtext.vocab.Vocab(merged,
specials=list(special_token_list),
max_size=vocab_size) 示例4: merge_vocabs
# 需要导入模块: from torchtext import vocab [as 别名]
# 或者: from torchtext.vocab import freqs [as 别名]
def merge_vocabs(vocabs, vocab_size=None):
"""
Merge individual vocabularies (assumed to be generated from disjoint
documents) into a larger vocabulary.
Args:
vocabs: `torchtext.vocab.Vocab` vocabularies to be merged
vocab_size: `int` the final vocabulary size. `None` for no limit.
Return:
`torchtext.vocab.Vocab`
"""
merged = sum([vocab.freqs for vocab in vocabs], Counter())
return torchtext.vocab.Vocab(merged,
specials=[PAD_WORD, BOS_WORD, EOS_WORD],
max_size=vocab_size) 示例5: merge_vocabs
# 需要导入模块: from torchtext import vocab [as 别名]
# 或者: from torchtext.vocab import freqs [as 别名]
def merge_vocabs(vocabs, vocab_size=None):
"""
Merge individual vocabularies (assumed to be generated from disjoint
documents) into a larger vocabulary.
Args:
vocabs: `torchtext.vocab.Vocab` vocabularies to be merged
vocab_size: `int` the final vocabulary size. `None` for no limit.
Return:
`torchtext.vocab.Vocab`
"""
merged = sum([vocab.freqs for vocab in vocabs], Counter())
return torchtext.vocab.Vocab(merged,
specials=list(special_token_list),
max_size=vocab_size) 示例6: merge_vocabs
# 需要导入模块: from torchtext import vocab [as 别名]
# 或者: from torchtext.vocab import freqs [as 别名]
def merge_vocabs(vocabs, vocab_size=None):
"""
Merge individual vocabularies (assumed to be generated from disjoint
documents) into a larger vocabulary.
Args:
vocabs: `torchtext.vocab.Vocab` vocabularies to be merged
vocab_size: `int` the final vocabulary size. `None` for no limit.
Return:
`torchtext.vocab.Vocab`
"""
merged = Counter(chain(*[vocab.freqs for vocab in vocabs]))
return torchtext.vocab.Vocab(merged,
specials=[PAD_WORD, BOS_WORD, EOS_WORD],
max_size=vocab_size) 示例7: filter_counter
# 需要导入模块: from torchtext import vocab [as 别名]
# 或者: from torchtext.vocab import freqs [as 别名]
def filter_counter(freqs, min_freq):
cnt = Counter()
for k, v in freqs.items():
if (min_freq is None) or (v >= min_freq):
cnt[k] = v
return cnt