Data Persistence

This page provides a comprehensive overview of the data persistence strategy across the Link platform. Persistence is categorized into three main types, each serving specific architectural needs.

Persistence Strategy Overview

The Link platform utilizes a polyglot persistence approach to optimize for different data characteristics:

graph TD
    %% Databases
    SQL[(SQL Server)]
    Mongo[(MongoDB)]
    ABS[(Azure Blob Storage)]

    %% Services
    AccountService[[Account Service]]
    AuditService[[Audit Service]]
    CensusService[[Census Service]]
    DAService[[Data Acquisition Service]]
    NormService[[Normalization Service]]
    NotifService[[Notification Service]]
    TenantService[[Tenant Service]]
    MEService[[Measure Evaluation Service]]
    ReportService[[Report Service]]
    SubService[[Submission Service]]
    ValService[[Validation Service]]

    %% Connections to SQL
    AccountService --- SQL
    AuditService --- SQL
    CensusService --- SQL
    DAService --- SQL
    NormService --- SQL
    NotifService --- SQL
    ReportService --- SQL
    TenantService --- SQL
    ValService --- SQL
    SubService --- SQL

    %% Connections to MongoDB
    MEService --- Mongo

    %% Connections to ABS
    MEService --- ABS
    SubService --- ABS
    ValService --- ABS

    classDef database fill:#f9f,stroke:#333,stroke-width:2px;
    class SQL,Mongo,ABS database;

• SQL Server: The primary store for configuration and metadata. Most services use SQL Server to maintain state, tenant settings, and operational logs.
• MongoDB (Azure Cosmos DB): Deployed as Azure Cosmos for MongoDB, this layer is used for clinical and evaluation artifacts. It handles the high-volume, semi-structured nature of FHIR resources and evaluation results.
• Azure Blob Storage: Used for large file persistence (e.g., patient bundles, submission NDJSON files).
  • Platform Agnosticism: Blob storage access is abstracted into interface patterns (e.g., IBlobStorageService) to ensure the platform remains agnostic of the underlying cloud provider.
  • Usage: Used for internal submissions within the reporting pipeline and external submissions to public health agencies.

Service-Specific Persistence

Detailed persistence schemas for each service are available directly on their respective documentation pages under the "Database Schema" section. These schemas are maintained as JSON Schema files (db.schema.json) within each service's directory.

SQL Server (Relational)

The following services utilize SQL Server for structured configuration and metadata:

• Account Service
• Audit Service
• Census Service
• Data Acquisition Service
• Normalization Service
• service:NotificationService
• Report Service
• Submission Service
• Tenant Service
• Validation Service

MongoDB (Document-based)

Used for high-volume clinical and evaluation data:

• Measure Evaluation Service

Azure Blob Storage (Object Storage)

Used for large artifacts and submission bundles:

• Measure Evaluation Service (evaluated reports)
• Submission Service (internal and external bundles)
• Validation Service (validation results)

Reporting Pipeline Data Flow

The following diagram illustrates the flow of data through the reporting pipeline and the specific points where it is persisted to the various database layers:

graph TD
    subgraph Pipeline ["Reporting Pipeline Flow"]
        DAService[[Data Acquisition Service]] -->|Acquires| NormService[[Normalization Service]]
        NormService -->|Normalizes| MEService[[Measure Evaluation Service]]
        MEService -->|Evaluates| ReportService[[Report Service]]
        ReportService -->|Validates| ValService[[Validation Service]]
        ValService -->|Submits| SubService[[Submission Service]]
    end

    %% Persistence
    Mongo[(MongoDB)]
    IntABS[(Internal ABS)]
    ExtABS[(External ABS)]

    %% Connections
    MEService -.->|Persist Normalized Data| Mongo
    ReportService -.->|Persist Evaluated Reports| IntABS
    SubService -.->|Persist Final Submissions| ExtABS

    classDef service fill:#fff,stroke:#333,stroke-width:2px;
    classDef storage fill:#f9f,stroke:#333,stroke-width:2px;
    class DAService,NormService,MEService,ReportService,ValService,SubService service;
    class Mongo,IntABS,ExtABS storage;

Data Persistence Lifecycle

1. Acquisition & Normalization: Data is retrieved from EHRs and normalized to ensure FHIR compliance.
2. Measure Evaluation: The normalized data is passed to the Measure Evaluation Service, where it is persisted in MongoDB for evaluation processing.
3. Evaluation & Reporting: After the CQL engine evaluates the data, the resulting MeasureReports are processed by the Report Service and persisted in Internal Blob Storage.
4. Validation & Submission: The evaluation results are validated against FHIR profiles and quality measures. Once validated, the Submission Service packages the content and persists the final bundles in External Blob Storage for delivery to public health agencies.

Relationships

flowchart LR
ndata_persistence_A306054["Design: Data Persistence"]