当前位置: 首页>>代码示例>>Python>>正文


Python python_operator.PythonOperator方法代码示例

本文整理汇总了Python中airflow.operators.python_operator.PythonOperator方法的典型用法代码示例。如果您正苦于以下问题:Python python_operator.PythonOperator方法的具体用法?Python python_operator.PythonOperator怎么用?Python python_operator.PythonOperator使用的例子?那么, 这里精选的方法代码示例或许可以为您提供帮助。您也可以进一步了解该方法所在airflow.operators.python_operator的用法示例。


在下文中一共展示了python_operator.PythonOperator方法的15个代码示例,这些例子默认根据受欢迎程度排序。您可以为喜欢或者感觉有用的代码点赞,您的评价将有助于系统推荐出更棒的Python代码示例。

示例1: get_dated_main_runner_operator

# 需要导入模块: from airflow.operators import python_operator [as 别名]
# 或者: from airflow.operators.python_operator import PythonOperator [as 别名]
def get_dated_main_runner_operator(
        dag,
        main_function,
        execution_timeout,
        day_shift=0,
        task_id='pull_image_data',
):
    args_str = f'{{{{ macros.ds_add(ds, -{day_shift}) }}}}'
    return PythonOperator(
        task_id=task_id,
        python_callable=main_function,
        op_args=[args_str],
        execution_timeout=execution_timeout,
        depends_on_past=False,
        dag=dag
    ) 
开发者ID:creativecommons,项目名称:cccatalog,代码行数:18,代码来源:operator_util.py

示例2: get_action_xcom

# 需要导入模块: from airflow.operators import python_operator [as 别名]
# 或者: from airflow.operators.python_operator import PythonOperator [as 别名]
def get_action_xcom(self, task_id=dn.ACTION_XCOM):
        """Generate the action_xcom step

        Step responsible for getting the action information passed
        by the invocation of the dag, which includes any options.
        """
        def xcom_push(**kwargs):
            """xcom_push function

            Defines a push function to store the content of 'action' that is
            defined via 'dag_run' in XCOM so that it can be used by the
            Operators. Includes action-related information for later steps.
            """

            kwargs['ti'].xcom_push(key='action',
                                   value=kwargs['dag_run'].conf['action'])
            kwargs['ti'].xcom_push(key='action_type',
                                   value=self.action_type)

        return PythonOperator(task_id=task_id,
                              dag=self.dag,
                              python_callable=xcom_push) 
开发者ID:airshipit,项目名称:shipyard,代码行数:24,代码来源:common_step_factory.py

示例3: generate_parallel_tasks

# 需要导入模块: from airflow.operators import python_operator [as 别名]
# 或者: from airflow.operators.python_operator import PythonOperator [as 别名]
def generate_parallel_tasks(name_prefix, num_of_tasks, deps):
    """
    Generate a list of PythonOperator tasks. The generated tasks are set up to
    be dependent on the `deps` argument.
    """
    tasks = []
    for t_id in range(num_of_tasks):
        run_this = PythonOperator(
            task_id=f"{name_prefix}_{t_id}",
            python_callable=print_context,
        )
        run_this << deps
        tasks.append(run_this)
    return tasks 
开发者ID:apache,项目名称:airflow,代码行数:16,代码来源:sql_perf_dag.py

示例4: get_runner_operator

# 需要导入模块: from airflow.operators import python_operator [as 别名]
# 或者: from airflow.operators.python_operator import PythonOperator [as 别名]
def get_runner_operator(dag):
    return PythonOperator(
        task_id='pull_wikimedia_commons_data',
        python_callable=wikimedia_commons.main,
        op_args=['{{ ds }}'],
        depends_on_past=False,
        dag=dag
    ) 
开发者ID:creativecommons,项目名称:cccatalog,代码行数:10,代码来源:wikimedia_workflow.py

示例5: get_runner_operator

# 需要导入模块: from airflow.operators import python_operator [as 别名]
# 或者: from airflow.operators.python_operator import PythonOperator [as 别名]
def get_runner_operator(dag):
    return PythonOperator(
        task_id="pull_rawpixel_data",
        python_callable=raw_pixel.main,
        depends_on_past=False,
        dag=dag
    ) 
开发者ID:creativecommons,项目名称:cccatalog,代码行数:9,代码来源:rawpixel_workflow.py

示例6: get_runner_operator

# 需要导入模块: from airflow.operators import python_operator [as 别名]
# 或者: from airflow.operators.python_operator import PythonOperator [as 别名]
def get_runner_operator(dag):
    return PythonOperator(
        task_id='pull_metropolitan_museum_data',
        python_callable=metropolitan_museum_of_art.main,
        op_args=['{{ ds }}'],
        depends_on_past=False,
        dag=dag
    ) 
开发者ID:creativecommons,项目名称:cccatalog,代码行数:10,代码来源:metropolitan_museum_workflow.py

示例7: get_runner_operator

# 需要导入模块: from airflow.operators import python_operator [as 别名]
# 或者: from airflow.operators.python_operator import PythonOperator [as 别名]
def get_runner_operator(dag):
    return PythonOperator(
        task_id='pull_europeana_data',
        python_callable=europeana.main,
        op_args=['{{ ds }}'],
        depends_on_past=False,
        dag=dag
    ) 
开发者ID:creativecommons,项目名称:cccatalog,代码行数:10,代码来源:europeana_workflow.py

示例8: get_runner_operator

# 需要导入模块: from airflow.operators import python_operator [as 别名]
# 或者: from airflow.operators.python_operator import PythonOperator [as 别名]
def get_runner_operator(dag):
    return PythonOperator(
        task_id="pull_cleveland_data",
        python_callable=cleveland_museum_of_art.main,
        depends_on_past=False,
        dag=dag
    ) 
开发者ID:creativecommons,项目名称:cccatalog,代码行数:9,代码来源:cleveland_museum_workflow.py

示例9: get_table_creator_operator

# 需要导入模块: from airflow.operators import python_operator [as 别名]
# 或者: from airflow.operators.python_operator import PythonOperator [as 别名]
def get_table_creator_operator(
        dag,
        postgres_conn_id,
        identifier=TIMESTAMP_TEMPLATE
):
    return PythonOperator(
        task_id='create_loading_table',
        python_callable=sql.create_loading_table,
        op_args=[postgres_conn_id, identifier],
        dag=dag
    ) 
开发者ID:creativecommons,项目名称:cccatalog,代码行数:13,代码来源:operators.py

示例10: get_load_local_data_operator

# 需要导入模块: from airflow.operators import python_operator [as 别名]
# 或者: from airflow.operators.python_operator import PythonOperator [as 别名]
def get_load_local_data_operator(
        dag,
        output_dir,
        postgres_conn_id,
        identifier=TIMESTAMP_TEMPLATE
):
    return PythonOperator(
        task_id='load_local_data',
        python_callable=loader.load_local_data,
        op_args=[output_dir, postgres_conn_id, identifier],
        trigger_rule=TriggerRule.ALL_SUCCESS,
        dag=dag
    ) 
开发者ID:creativecommons,项目名称:cccatalog,代码行数:15,代码来源:operators.py

示例11: get_copy_to_s3_operator

# 需要导入模块: from airflow.operators import python_operator [as 别名]
# 或者: from airflow.operators.python_operator import PythonOperator [as 别名]
def get_copy_to_s3_operator(
        dag,
        output_dir,
        storage_bucket,
        aws_conn_id,
        identifier=TIMESTAMP_TEMPLATE
):
    return PythonOperator(
        task_id='copy_to_s3',
        python_callable=loader.copy_to_s3,
        op_args=[output_dir, storage_bucket, identifier, aws_conn_id],
        dag=dag
    ) 
开发者ID:creativecommons,项目名称:cccatalog,代码行数:15,代码来源:operators.py

示例12: get_load_s3_data_operator

# 需要导入模块: from airflow.operators import python_operator [as 别名]
# 或者: from airflow.operators.python_operator import PythonOperator [as 别名]
def get_load_s3_data_operator(
        dag,
        bucket,
        aws_conn_id,
        postgres_conn_id,
        identifier=TIMESTAMP_TEMPLATE
):
    return PythonOperator(
        task_id='load_s3_data',
        python_callable=loader.load_s3_data,
        op_args=[bucket, aws_conn_id, postgres_conn_id, identifier],
        dag=dag
    ) 
开发者ID:creativecommons,项目名称:cccatalog,代码行数:15,代码来源:operators.py

示例13: get_file_deletion_operator

# 需要导入模块: from airflow.operators import python_operator [as 别名]
# 或者: from airflow.operators.python_operator import PythonOperator [as 别名]
def get_file_deletion_operator(
        dag,
        output_dir,
        identifier=TIMESTAMP_TEMPLATE
):
    return PythonOperator(
        task_id='delete_staged_file',
        python_callable=paths.delete_staged_file,
        op_args=[output_dir, identifier],
        trigger_rule=TriggerRule.ALL_SUCCESS,
        dag=dag,
    ) 
开发者ID:creativecommons,项目名称:cccatalog,代码行数:14,代码来源:operators.py

示例14: get_failure_moving_operator

# 需要导入模块: from airflow.operators import python_operator [as 别名]
# 或者: from airflow.operators.python_operator import PythonOperator [as 别名]
def get_failure_moving_operator(
        dag,
        output_dir,
        identifier=TIMESTAMP_TEMPLATE
):
    return PythonOperator(
        task_id='move_staged_failures',
        python_callable=paths.move_staged_files_to_failure_directory,
        op_args=[output_dir, identifier],
        trigger_rule=TriggerRule.ONE_SUCCESS,
        dag=dag
    ) 
开发者ID:creativecommons,项目名称:cccatalog,代码行数:14,代码来源:operators.py

示例15: get_flickr_sub_provider_update_operator

# 需要导入模块: from airflow.operators import python_operator [as 别名]
# 或者: from airflow.operators.python_operator import PythonOperator [as 别名]
def get_flickr_sub_provider_update_operator(
        dag,
        postgres_conn_id,
):
    return PythonOperator(
        task_id='update_flickr_sub_providers',
        python_callable=sql.update_flickr_sub_providers,
        op_args=[postgres_conn_id],
        dag=dag
    ) 
开发者ID:creativecommons,项目名称:cccatalog,代码行数:12,代码来源:operators.py


注:本文中的airflow.operators.python_operator.PythonOperator方法示例由纯净天空整理自Github/MSDocs等开源代码及文档管理平台,相关代码片段筛选自各路编程大神贡献的开源项目,源码版权归原作者所有,传播和使用请参考对应项目的License;未经允许,请勿转载。