本文整理汇总了Python中pytorch_pretrained_bert.tokenization.BasicTokenizer方法的典型用法代码示例。如果您正苦于以下问题：Python tokenization.BasicTokenizer方法的具体用法？Python tokenization.BasicTokenizer怎么用？Python tokenization.BasicTokenizer使用的例子？那么, 这里精选的方法代码示例或许可以为您提供帮助。您也可以进一步了解该方法所在类pytorch_pretrained_bert.tokenization的用法示例。
在下文中一共展示了tokenization.BasicTokenizer方法的1个代码示例，这些例子默认根据受欢迎程度排序。您可以为喜欢或者感觉有用的代码点赞，您的评价将有助于系统推荐出更棒的Python代码示例。
示例1: __init__
# 需要导入模块: from pytorch_pretrained_bert import tokenization [as 别名]
# 或者: from pytorch_pretrained_bert.tokenization import BasicTokenizer [as 别名]
def __init__(self,
entity_candidate_generators: Dict[str, MentionGenerator],
entity_indexers: Dict[str, TokenIndexer],
bert_model_type: str,
do_lower_case: bool,
whitespace_tokenize: bool = True,
max_word_piece_sequence_length: int = 512) -> None:
"""
Note: the fields need to be used with a pre-generated allennlp vocabulary
that contains the entity id namespaces and the bert name space.
entity_indexers = {'wordnet': indexer for wordnet entities,
'wiki': indexer for wiki entities}
"""
# load BertTokenizer from huggingface
self.candidate_generators = entity_candidate_generators
self.bert_tokenizer = BertTokenizer.from_pretrained(
bert_model_type, do_lower_case=do_lower_case
)
self.bert_word_tokenizer = BasicTokenizer(do_lower_case=False)
# Target length should include start and end token
self.max_word_piece_sequence_length = max_word_piece_sequence_length
self._entity_indexers = entity_indexers
# for bert, we'll give an empty token indexer with empty name space
# and do the indexing directly with the bert vocab to bypass
# indexing in the indexer
self._bert_single_id_indexer = {'tokens': SingleIdTokenIndexer('__bert__')}
self.do_lowercase = do_lower_case
self.whitespace_tokenize = whitespace_tokenize
self.dtype = np.float32