Integrity Tests

Rate Object Space Completeness

After the Rate Object Space is created, we run the following tests:

• No duplicate ROIDs
• No missing payers
• No missing providers
• No missing networks
• No missing provider types
• No missing bill_types

tip

Once the Rate Object Space is constructed, all we need to check throughout the pipeline is that the expected ROIDs are present. If the expected ROIDs are present and unique, then we can be confident that the data is complete.

Rate Presence

• All payers have at least one rate
• All networks have at least one rate
• All billing code types have at least one rate
• All bill types have at least one rate
• All provider types have at least one rate
• All network types have at least one rate
• At least 50% of providers have at least one rate

Name Presence

• provider_name is not null
• provider_type is not null
• payer_name is not null
• network_name is not null
• network_type is not null
• network_class is not null

Data Types

• Validates prod_clickhouse_abridged schema against methodology/airflow/abridged_schema
  • Assume varchar unless 'cast' is specified in schema
• Checks that npi array does not contain NULLs (no NULL, ARRAY[NULL], or ARRAY['123456789', NULL])

Feature Recommendation Tables

• check whether any of the feature array values are null

Rollup Table Uniqueness Tests

Check that rolllup tables have unique keys:

{
    "prod_rollup_provider": [
        "provider_id",
    ],
    "prod_rollup_payer": [
        "payer_id",
    ],
    "prod_rollup_network": [
        "network_id",
    ],
    "prod_rollup_provider_network": [
        "provider_id",
        "network_id",
        "bill_type"
    ],
    "prod_rollup_code": [
        'bill_type',
        "billing_code",
        "billing_code_type",
    ],
    "prod_rollup_cbsa_code": [
        "cbsa",
        'bill_type',
        "billing_code",
        "billing_code_type",
    ],
}