本文整理匯總了Python中pytorch_pretrained_bert.tokenization.BasicTokenizer方法的典型用法代碼示例。如果您正苦於以下問題：Python tokenization.BasicTokenizer方法的具體用法？Python tokenization.BasicTokenizer怎麽用？Python tokenization.BasicTokenizer使用的例子？那麽, 這裏精選的方法代碼示例或許可以為您提供幫助。您也可以進一步了解該方法所在類pytorch_pretrained_bert.tokenization的用法示例。
在下文中一共展示了tokenization.BasicTokenizer方法的1個代碼示例，這些例子默認根據受歡迎程度排序。您可以為喜歡或者感覺有用的代碼點讚，您的評價將有助於係統推薦出更棒的Python代碼示例。
示例1: __init__
# 需要導入模塊: from pytorch_pretrained_bert import tokenization [as 別名]
# 或者: from pytorch_pretrained_bert.tokenization import BasicTokenizer [as 別名]
def __init__(self,
entity_candidate_generators: Dict[str, MentionGenerator],
entity_indexers: Dict[str, TokenIndexer],
bert_model_type: str,
do_lower_case: bool,
whitespace_tokenize: bool = True,
max_word_piece_sequence_length: int = 512) -> None:
"""
Note: the fields need to be used with a pre-generated allennlp vocabulary
that contains the entity id namespaces and the bert name space.
entity_indexers = {'wordnet': indexer for wordnet entities,
'wiki': indexer for wiki entities}
"""
# load BertTokenizer from huggingface
self.candidate_generators = entity_candidate_generators
self.bert_tokenizer = BertTokenizer.from_pretrained(
bert_model_type, do_lower_case=do_lower_case
)
self.bert_word_tokenizer = BasicTokenizer(do_lower_case=False)
# Target length should include start and end token
self.max_word_piece_sequence_length = max_word_piece_sequence_length
self._entity_indexers = entity_indexers
# for bert, we'll give an empty token indexer with empty name space
# and do the indexing directly with the bert vocab to bypass
# indexing in the indexer
self._bert_single_id_indexer = {'tokens': SingleIdTokenIndexer('__bert__')}
self.do_lowercase = do_lower_case
self.whitespace_tokenize = whitespace_tokenize
self.dtype = np.float32