本文整理汇总了Python中pybabe.Babe.get_config_with_env方法的典型用法代码示例。如果您正苦于以下问题：Python Babe.get_config_with_env方法的具体用法？Python Babe.get_config_with_env怎么用？Python Babe.get_config_with_env使用的例子？那么恭喜您, 这里精选的方法代码示例或许可以为您提供帮助。您也可以进一步了解该方法所在类pybabe.Babe的用法示例。
在下文中一共展示了Babe.get_config_with_env方法的1个代码示例，这些例子默认根据受欢迎程度排序。您可以为喜欢或者感觉有用的代码点赞，您的评价将有助于系统推荐出更棒的Python代码示例。
示例1: pull_kontagent
# 需要导入模块: from pybabe import Babe [as 别名]
# 或者: from pybabe.Babe import get_config_with_env [as 别名]
def pull_kontagent(nostream, start_time, end_time,
sample_mode=False, discard_names=None,
**kwargs):
"""
Generate streams from kontagent logs.
Generates a stream per hour and per message type. The streams are outputed per hour.
babes = Babe().pull_kontagent_streams(start_time='...', 'end_time='...')
for babe in babes:
babe.push_sql(table='$typename')
start_time : hour of the first stream to getattr
end_time : hour of the first stream to getattr
referent_timezone (optional, default utc): the timezone to use to interpret "day"
KT_USER : user id
KT_APPID : id of the app
KT_PASS : password of the user
KT_FILECACHE : local copy of kontagent files.
Version 1.1
"""
referent_timezone = Babe.get_config_with_env("kontagent", "timezone", kwargs, "utc")
kt_user = Babe.get_config_with_env("kontagent", "KT_USER", kwargs)
kt_pass = Babe.get_config_with_env("kontagent", "KT_PASS", kwargs)
kt_filecache = Babe.get_config_with_env(section='kontagent', key='KT_FILECACHE')
if discard_names:
discard_names = set(discard_names)
else:
discard_names = set()
if not os.path.exists(kt_filecache):
os.makedirs(kt_filecache)
kt_appid = Babe.get_config_with_env("kontagent", "KT_APPID", kwargs)
for hour in enumerate_period_per_hour(start_time, end_time, referent_timezone):
url = get_url(hour, kt_user, kt_pass, kt_appid)
log.info("Kontagent: retrieving list: %s" % url)
s = urllib.urlopen(url).read()
if s == "No files available":
continue
file_urls = json.loads(s)
if sample_mode and len(file_urls) > 0:
# Sample mode: just process the first file.
file_urls = file_urls[:1]
p = Pool(8)
downloaded_files = p.map(lambda url: read_url_with_cache(
url, kt_user, kt_pass, kt_filecache), file_urls)
p.close()
header = kt_msg.replace(partition=[
("date", datetime.date(hour.year, hour.month, hour.day)),
("hour", hour.hour)])
yield header
gzips = [Popen(['gzip', '-d', '-c', f], stdin=PIPE, stdout=PIPE) for f in downloaded_files]
for gzip in gzips:
for row in process_file(hour, gzip.stdout, discard_names):
yield row
gzip.stdin.close()
gzip.wait()
yield StreamFooter()