academic_observatory_workflows.data_quality_workflow.tests.test_data_quality_workflow

Module Contents

Classes

TestDataQualityWorkflow	Tests for the Data Quality Check Workflow
TestDataQualityUtils

Attributes

FIXTURES_FOLDER

academic_observatory_workflows.data_quality_workflow.tests.test_data_quality_workflow.FIXTURES_FOLDER

class academic_observatory_workflows.data_quality_workflow.tests.test_data_quality_workflow.TestDataQualityWorkflow(*args, **kwargs)

Bases: observatory.platform.observatory_environment.ObservatoryTestCase

Tests for the Data Quality Check Workflow

test_dag_load()

Test that the DataQualityCheck DAG can be loaded from a DAG bag.

test_dag_structure()

Test that the DAG has the correct structure.

test_workflow()

Test the Data Quality Check Workflow end to end

Borrowing off of the doi test structure.

class academic_observatory_workflows.data_quality_workflow.tests.test_data_quality_workflow.TestDataQualityUtils(*args, **kwargs)

Bases: observatory.platform.observatory_environment.ObservatoryTestCase

test_create_table_hash_id()

Test if hash can be reliably created.

test_create_dq_record()

Test if a data quality check record can be reliably created.

test_is_in_dqc_table()

Test if a data quality check has already been previously performed by checking the table hash that it creates.

test_bq_count_duplicate_records()

Test if duplicate records can be reliably found in a table.

test_bq_count_nulls()

Test if the number of nulls under a field can be correctly determined.

test_bq_count_distinct_records()

Test that the number of distinct records can be reliably detmerined.

test_table_object()

Test that a table’s shard limit can be set properly.