academic_observatory_workflows.crossref_fundref_telescope.telescope

Classes

DagParams

Parameters for the Crossref Fundref Telescope

Functions

create_dag(→ airflow.DAG)

Construct a CrossrefFundrefTelescope instance.

Module Contents

class academic_observatory_workflows.crossref_fundref_telescope.telescope.DagParams(dag_id: str, cloud_workspace: observatory_platform.airflow.workflow.CloudWorkspace, bq_dataset_id: str = 'crossref_fundref', bq_table_name: str = 'crossref_fundref', api_bq_dataset_id: str = 'dataset_api', schema_folder: str = project_path('crossref_fundref_telescope', 'schema'), dataset_description: str = 'The Crossref Funder Registry dataset: https://www.crossref.org/services/funder-registry/', table_description: str = 'The Crossref Funder Registry dataset: https://www.crossref.org/services/funder-registry/', start_date: pendulum.DateTime = pendulum.datetime(2014, 2, 23), schedule: str = '@weekly', catchup: bool = True, gitlab_pool_name: str = 'gitlab_pool', gitlab_pool_slots: int = 2, gitlab_pool_description: str = 'A pool to limit the connections to Gitlab', retries: int = 3)[source]

Parameters for the Crossref Fundref Telescope

Parameters:

  • dag_id – the id of the DAG.

  • cloud_workspace – the cloud workspace settings.

  • bq_dataset_id – the BigQuery dataset id.

  • bq_table_name – the BigQuery table name.

  • api_bq_dataset_id – the Dataset ID to use when storing releases.

  • schema_folder – the SQL schema path.

  • dataset_description – description for the BigQuery dataset.

  • table_description – description for the BigQuery table.

  • start_date – the start date of the DAG.

  • schedule – the schedule interval of the DAG.

  • catchup – whether to catchup the DAG or not.

  • gitlab_pool_name – name of the Gitlab Pool.

  • gitlab_pool_slots – number of slots for the Gitlab Pool.

  • gitlab_pool_description – description for the Gitlab Pool.

  • retries – the number of times to retry a task.

dag_id[source]
cloud_workspace[source]
bq_dataset_id = 'crossref_fundref'[source]
bq_table_name = 'crossref_fundref'[source]
api_bq_dataset_id = 'dataset_api'[source]
schema_folder[source]
dataset_description = 'The Crossref Funder Registry dataset: https://www.crossref.org/services/funder-registry/'[source]
table_description = 'The Crossref Funder Registry dataset: https://www.crossref.org/services/funder-registry/'[source]
start_date[source]
schedule = '@weekly'[source]
catchup = True[source]
gitlab_pool_name = 'gitlab_pool'[source]
gitlab_pool_slots = 2[source]
gitlab_pool_description = 'A pool to limit the connections to Gitlab'[source]
retries = 3[source]
academic_observatory_workflows.crossref_fundref_telescope.telescope.create_dag(dag_params: DagParams) → airflow.DAG[source]

Construct a CrossrefFundrefTelescope instance.