本文整理汇总了Python中boto.s3.connection.Key.get_contents_to_file方法的典型用法代码示例。如果您正苦于以下问题：Python Key.get_contents_to_file方法的具体用法？Python Key.get_contents_to_file怎么用？Python Key.get_contents_to_file使用的例子？那么恭喜您, 这里精选的方法代码示例或许可以为您提供帮助。您也可以进一步了解该方法所在类boto.s3.connection.Key的用法示例。
在下文中一共展示了Key.get_contents_to_file方法的1个代码示例，这些例子默认根据受欢迎程度排序。您可以为喜欢或者感觉有用的代码点赞，您的评价将有助于系统推荐出更棒的Python代码示例。
示例1: mapper
# 需要导入模块: from boto.s3.connection import Key [as 别名]
# 或者: from boto.s3.connection.Key import get_contents_to_file [as 别名]
def mapper(self, _, line):
line = line.rstrip().split('\t', 1)[-1]
filename = line.rsplit('/', 1)[-1]
first_rec = None
f = open(filename, 'w')
for i in xrange(10):
try:
conn = boto.connect_s3(anon=True)
bucket = conn.get_bucket('aws-publicdatasets')
key = Key(bucket, line)
key.get_contents_to_file(f)
f.close()
records = warc.WARCFile(fileobj=gzip.open(filename, 'rb'))
break
except Exception as e:
continue
else:
logger.error('10 attempts to get file {} failed, skipping...'.format(filename))
return
try:
for i, record in enumerate(records):
if record.type != 'response':
_ = record.payload.read()
continue
for key, value in self.process_record(record):
yield key, value
self.increment_counter('commoncrawl', 'processed_records', 1)
except Exception:
logger.error(traceback.format_exc())
self.increment_counter('errors', 'general', 1)
finally:
f.close()
os.unlink(filename)