Python PyTorch HashChecker用法及代碼示例

本文簡要介紹python語言中 torchdata.datapipes.iter.HashChecker 的用法。

用法:

class torchdata.datapipes.iter.HashChecker(source_datapipe: IterDataPipe[Tuple[str, io.IOBase]], hash_dict: Dict[str, str], hash_type: str = 'sha256', rewind: bool = True)

參數：

• source_datapipe-IterDataPipe 包含文件名和數據/流的元組

• hash_dict-將文件名映射到其相應哈希的字典

• hash_type-要應用的哈希函數的類型

• rewind-在使用流計算哈希後倒回流(這不適用於不可搜索的流，例如 HTTP)

根據文件名和數據/流元組的輸入 DataPipe 計算並檢查每個文件的哈希值(函數名稱： check_hash )。如果哈希值與字典中給定的哈希值匹配，則會生成文件名和數據/流的元組。否則，將會引發錯誤。

示例

>>> from torchdata.datapipes.iter import IterableWrapper, HttpReader
>>> file_url = "https://raw.githubusercontent.com/pytorch/data/main/LICENSE"
>>> expected_MD5_hash = "bb9675028dd39d2dd2bf71002b93e66c"
>>> http_reader_dp = HttpReader(IterableWrapper([file_url]))
>>> # An exception is only raised when the hash doesn't match, otherwise (path, stream) is returned
>>> check_hash_dp = http_reader_dp.check_hash({file_url: expected_MD5_hash}, "md5", rewind=False)
>>> reader_dp = check_hash_dp.readlines()
>>> it = iter(reader_dp)
>>> path, line = next(it)
>>> path
https://raw.githubusercontent.com/pytorch/data/main/LICENSE
>>> line
b'BSD 3-Clause License'