Python PyTorch HashChecker用法及代码示例

本文简要介绍python语言中 torchdata.datapipes.iter.HashChecker 的用法。

用法:

class torchdata.datapipes.iter.HashChecker(source_datapipe: IterDataPipe[Tuple[str, io.IOBase]], hash_dict: Dict[str, str], hash_type: str = 'sha256', rewind: bool = True)

参数：

• source_datapipe-IterDataPipe 包含文件名和数据/流的元组

• hash_dict-将文件名映射到其相应哈希的字典

• hash_type-要应用的哈希函数的类型

• rewind-在使用流计算哈希后倒回流(这不适用于不可搜索的流，例如 HTTP)

根据文件名和数据/流元组的输入 DataPipe 计算并检查每个文件的哈希值(函数名称： check_hash )。如果哈希值与字典中给定的哈希值匹配，则会生成文件名和数据/流的元组。否则，将会引发错误。

示例

>>> from torchdata.datapipes.iter import IterableWrapper, HttpReader
>>> file_url = "https://raw.githubusercontent.com/pytorch/data/main/LICENSE"
>>> expected_MD5_hash = "bb9675028dd39d2dd2bf71002b93e66c"
>>> http_reader_dp = HttpReader(IterableWrapper([file_url]))
>>> # An exception is only raised when the hash doesn't match, otherwise (path, stream) is returned
>>> check_hash_dp = http_reader_dp.check_hash({file_url: expected_MD5_hash}, "md5", rewind=False)
>>> reader_dp = check_hash_dp.readlines()
>>> it = iter(reader_dp)
>>> path, line = next(it)
>>> path
https://raw.githubusercontent.com/pytorch/data/main/LICENSE
>>> line
b'BSD 3-Clause License'