本文整理汇总了Python中twitter.common.fs.HDFSHelper.copy_to_local方法的典型用法代码示例。如果您正苦于以下问题：Python HDFSHelper.copy_to_local方法的具体用法？Python HDFSHelper.copy_to_local怎么用？Python HDFSHelper.copy_to_local使用的例子？那么恭喜您, 这里精选的方法代码示例或许可以为您提供帮助。您也可以进一步了解该方法所在类twitter.common.fs.HDFSHelper的用法示例。
在下文中一共展示了HDFSHelper.copy_to_local方法的2个代码示例，这些例子默认根据受欢迎程度排序。您可以为喜欢或者感觉有用的代码点赞，您的评价将有助于系统推荐出更棒的Python代码示例。
示例1: test_copy_to_local
# 需要导入模块: from twitter.common.fs import HDFSHelper [as 别名]
# 或者: from twitter.common.fs.HDFSHelper import copy_to_local [as 别名]
def test_copy_to_local(self):
hdfs_helper = HDFSHelper("/etc/hadoop/hadoop-conf-tst-smf1",
command_class=MockCommandUtil)
cmd = hdfs_helper.copy_to_local('text_file','dest')
expected_cmd = "hdfs --config /etc/hadoop/hadoop-conf-tst-smf1 dfs " + \
"-copyToLocal text_file dest"
self.assertEqual(cmd, expected_cmd)示例2: fetch
# 需要导入模块: from twitter.common.fs import HDFSHelper [as 别名]
# 或者: from twitter.common.fs.HDFSHelper import copy_to_local [as 别名]
def fetch(self, uri, directory):
log.info("Fetching %s from HDFS" % uri)
if "JAVA_HOME" in os.environ:
log.info("Using JAVA_HOME '%s' for HDFS commands" % os.environ["JAVA_HOME"])
config = os.environ.get("HADOOP_CONF_DIR", HADOOP_CONF_DIR)
h = HDFSHelper(config, heap_limit=Amount(256, Data.MB))
try:
f = lambda: h.copy_to_local(uri, directory)
deadline(f, timeout=self._timeout, propagate=True, daemon=True)
except HDFSHelper.InternalError as e:
raise self.Error('Unable to fetch HDFS package: %s' % e)
except Timeout as e:
raise self.Error("Failed to fetch package from HDFS within : %s" % (self._timeout, e))