本文整理匯總了Python中dipper.models.Dataset.Dataset.set_date_issued方法的典型用法代碼示例。如果您正苦於以下問題：Python Dataset.set_date_issued方法的具體用法？Python Dataset.set_date_issued怎麽用？Python Dataset.set_date_issued使用的例子？那麽, 這裏精選的方法代碼示例或許可以為您提供幫助。您也可以進一步了解該方法所在類dipper.models.Dataset.Dataset的用法示例。
在下文中一共展示了Dataset.set_date_issued方法的1個代碼示例，這些例子默認根據受歡迎程度排序。您可以為喜歡或者感覺有用的代碼點讚，您的評價將有助於係統推薦出更棒的Python代碼示例。
示例1: fetch
# 需要導入模塊: from dipper.models.Dataset import Dataset [as 別名]
# 或者: from dipper.models.Dataset.Dataset import set_date_issued [as 別名]
#.........這裏部分代碼省略.........
:param is_dl_forced - boolean
:param files dict - override instance files dict
:return: None
"""
fstat = None
if files is None:
files = self.files
for fname in files:
headers = None
filesource = files[fname]
if 'headers' in filesource:
headers = filesource['headers']
LOG.info("Getting %s", fname)
# if the key 'clean' exists in the sources `files` dict
# expose that instead of the longer url
if 'clean' in filesource and filesource['clean'] is not None:
self.dataset.setFileAccessUrl(filesource['clean'])
else:
self.dataset.setFileAccessUrl(filesource['url'])
LOG.info('Fetching %s', filesource['url'])
self.fetch_from_url(
filesource['url'], '/'.join((self.rawdir, filesource['file'])),
is_dl_forced, headers)
fstat = os.stat('/'.join((self.rawdir, filesource['file'])))
# only keeping the date from the last file
filedate = datetime.utcfromtimestamp(fstat[ST_CTIME]).strftime("%Y-%m-%d")
# FIXME
# change this so the date is attached only to each file, not the entire dataset
self.dataset.set_date_issued(filedate)
def fetch_from_url(
self, remotefile, localfile=None, is_dl_forced=False, headers=None):
"""
Given a remote url and a local filename, attempt to determine
if the remote file is newer; if it is,
fetch the remote file and save it to the specified localfile,
reporting the basic file information once it is downloaded
:param remotefile: URL of remote file to fetch
:param localfile: pathname of file to save locally
:return: None
"""
response = None
if ((is_dl_forced is True) or localfile is None or
(self.checkIfRemoteIsNewer(remotefile, localfile, headers))):
# TODO url verification, etc
if headers is None:
headers = self._get_default_request_headers()
request = urllib.request.Request(remotefile, headers=headers)
response = urllib.request.urlopen(request)
if localfile is not None:
with open(localfile, 'wb') as binwrite:
while True:
chunk = response.read(CHUNK)
if not chunk:
break
binwrite.write(chunk)