本文整理匯總了Python中airflow.contrib.operators.bigquery_operator.BigQueryOperator方法的典型用法代碼示例。如果您正苦於以下問題：Python bigquery_operator.BigQueryOperator方法的具體用法？Python bigquery_operator.BigQueryOperator怎麽用？Python bigquery_operator.BigQueryOperator使用的例子？那麽, 這裏精選的方法代碼示例或許可以為您提供幫助。您也可以進一步了解該方法所在類airflow.contrib.operators.bigquery_operator的用法示例。
在下文中一共展示了bigquery_operator.BigQueryOperator方法的1個代碼示例，這些例子默認根據受歡迎程度排序。您可以為喜歡或者感覺有用的代碼點讚，您的評價將有助於係統推薦出更棒的Python代碼示例。
示例1: _get_operator_param_name_and_values
# 需要導入模塊: from airflow.contrib.operators import bigquery_operator [as 別名]
# 或者: from airflow.contrib.operators.bigquery_operator import BigQueryOperator [as 別名]
def _get_operator_param_name_and_values(operator_class_name, task_details):
""" Internal helper gets the name of the python parameter for the Airflow operator class. In
some cases, we do not expose the airflow parameter name in its native form, but choose to
expose a name that's more standard for Datalab, or one that's more friendly. For example,
Airflow's BigQueryOperator uses 'bql' for the query string, but we want %%bq users in Datalab
to use 'query'. Hence, a few substitutions that are specific to the Airflow operator need to
be made.
Similarly, we the parameter value could come from the notebook's context. All that happens
here.
Returns:
Dict containing _only_ the keys and values that are required in Airflow operator definition.
This requires a substituting existing keys in the dictionary with their Airflow equivalents (
i.e. by adding new keys, and removing the existing ones).
"""
# We make a clone and then remove 'type' and 'up_stream' since these aren't needed for the
# the operator's parameters.
operator_task_details = task_details.copy()
if 'type' in operator_task_details.keys():
del operator_task_details['type']
if 'up_stream' in operator_task_details.keys():
del operator_task_details['up_stream']
# We special-case certain operators if we do some translation of the parameter names. This is
# usually the case when we use syntactic sugar to expose the functionality.
# TODO(rajivpb): It should be possible to make this a lookup from the modules mapping via
# getattr() or equivalent. Avoid hard-coding these class-names here.
if (operator_class_name == 'BigQueryOperator'):
return PipelineGenerator._get_bq_execute_params(operator_task_details)
if (operator_class_name == 'BigQueryToCloudStorageOperator'):
return PipelineGenerator._get_bq_extract_params(operator_task_details)
if (operator_class_name == 'GoogleCloudStorageToBigQueryOperator'):
return PipelineGenerator._get_bq_load_params(operator_task_details)
return operator_task_details