Overview

As Link scales from initial deployment to supporting tens of thousands of facilities, it is critical to have a clear performance model. This document outlines the throughput requirements at various milestones and identifies the key points in the reporting pipeline that must be monitored to ensure the system meets its performance goals.

Scale Milestones

The primary driver for performance is the number of facilities and the volume of patient data they produce. Based on an average of 12,000 patients per facility per month, we can derive the required system throughput (patients per minute) for each milestone.

Assuming a month has approximately 44,000 minutes:

Data Volume Scaling

Beyond patient counts, the total volume of FHIR resources processed by the Normalization Service and stored by service:DataAccess scales linearly with patient count.

Based on initial data analysis (averaging 708 resources per patient):

• 100 Facilities: ~850 Million resources / month (~19,300 resources / minute)
• 30,000 Facilities: ~255 Billion resources / month (~5.8 Million resources / minute)

This highlights the necessity of a highly performant, distributed data lake (e.g., Iceberg) for the Data Access domain.

Patient Payload Metrics (First Half)

In the First Half of the pipeline (Acquisition through Evaluation), the system processes individual patient data sets. The following metrics represent the volume of data (in serialized JSON format) associated with a single patient.

These metrics are based on an initial analysis of 10,765 patients and approximately 7,622,175 resources.

Serialization Context

These metrics are derived using BSON.encode, which represents the actual serialized bytes of the data, rather than pretty-printed or indented JSON text. While Kafka events often include pretty-printing for human readability, this model focuses on the raw serialized size as it directly impacts storage requirements for Report Service's database and intermediate Redis caching layers.

Encryption Impact

While encryption-at-rest is handled natively by the database (e.g., Cosmos DB/Mongo), the system must consider the impact of encrypting data in transient storage. Specifically, using AES-128 to encrypt each resource before storing it in cache (e.g., Redis) adds a fixed overhead of approximately 32 bytes per encryption operation.

Per-Resource in-transit encryption to support individual resource caching with an average of 708 resources per patient adds ~22.7 KB of metadata bloats every patient's footprint; increasing the average patient size from 2.44 KB to ~25.14 KB (a ~930% increase); not including processing timme for encrypting each resource.

In this model, where individual resources are small, per-resource encryption leads to significant "metadata bloat," where the encryption overhead (32 bytes) frequently exceeds the size of the resource itself.

Throughput Implications (First Half)

Based on the Average (2.44 KB) and Maximum (40.08 KB) payload sizes, we can project the data throughput requirements for the ingestion pipeline across different scales:

These metrics inform the infrastructure requirements for transient data storage (e.g., Kafka message sizes, cache subscriptions) between the Data Acquisition Service, Normalization Service, and Measure Evaluation Service.

Reporting Pipeline Segmentation

To better analyze and scale the system, the reporting pipeline is divided into two primary segments:

First Half: Data Ingestion & Transformation

This segment covers the movement of data from the source EHR through the evaluation of measure logic. It is characterized by high-frequency, low-latency processing of individual patient payloads.

• Services: Data Acquisition Service, Data Acquisition Worker Service, Normalization Service, Measure Evaluation Service
• Workflow: Acquisition -> Normalization -> Initial Evaluation -> (Optional) Supplemental Acquisition -> (Optional) Supplemental Normalization -> Supplemental Evaluation.

Second Half: Validation & Submission

This segment covers final quality checks and the actual submission to the reporting authority. While processing the same aggregate patient data volume as the first half, this phase typically handles larger bundles and performs high-latency operations like FHIR profile validation.

• Services: Validation Service, Submission Service
• Workflow: Validation -> Submission.

Current Scaling & Capacity

As of version 0.5.0, the system is configured with the following resource allocations. These settings provide a baseline for the 25-100 facility milestones.

Second Half Throughput (Validation)

Based on current performance monitoring (observing ReadyForValidation consumer lag), the Validation Service has the following capacity:

• Avg Throughput: ~1 patient every 2-3 seconds per replica.
• Total Capacity (5 replicas): ~100-150 patients per minute.

This capacity is sufficient for the 100 Facilities milestone (~27.3 patients/minute). However, to meet the 1,000 Facilities milestone (~272.7 patients/minute), the Validation Service will need to scale to approximately 10-15 replicas, assuming linear scaling of the underlying FHIR validation engine.

Key Monitoring Points

Monitoring is divided between the two primary segments of the pipeline to better isolate performance bottlenecks.

First Half: Ingestion & Evaluation

1. Data Acquisition (EHR Interop)

• Monitoring Point: Data Acquisition Service and Data Acquisition Worker Service.
• Metric: Time to retrieve patient data from external EHRs.
• Constraints: Limited to 8 threads per facility to prevent EHR overload.
• Milestone Impact: At 30k facilities, even a small increase in EHR latency can lead to massive backlogs in the acquisition workers, as we are capped on concurrent requests per facility.

2. Normalization Throughput

• Monitoring Point: Normalization Service.
• Metric: Resources normalized per second.
• Observation: Tracked via Resource Normalized processing times.
• Milestone Impact: Normalization is an atomic, high-frequency task. As patient volume grows, this service must scale horizontally to prevent being a bottleneck.

3. Measure Evaluation Latency

• Monitoring Point: Measure Evaluation Service.
• Metric: Time to evaluate CQL for a single patient (Initial and Supplemental).
• Milestone Impact: Evaluation is computationally intensive. Monitoring the InitialEvaluation and SupplementalEvaluation statuses in event:ReportStatusUpdated is critical.

Second Half: Validation & Submission

4. Validation Throughput

• Monitoring Point: Validation Service.
• Metric: Validation time per patient (targeting ~2-3 seconds per patient).
• Observation: Consumer lag on the ReadyForValidation topic.
• Milestone Impact: As reported in Targeted Runtime Validation, full validation at the 30k scale is prohibitive. Monitoring the impact of "Unacceptable-only" validation rules on throughput is essential to maintain the 2-3 second/patient target.

5. Submission Success Rate

• Monitoring Point: Submission Service.
• Metric: Submission success rate and latency.
• Milestone Impact: Final submission is the last mile. Monitoring for transient network errors or submission endpoint saturation is critical as the frequency of submissions increases.

Cross-Pipeline Health

6. Kafka Pipeline Health

• Monitoring Point: Kafka Broker and Consumer Groups.
• Metric: Consumer Lag (especially for ReportStatusUpdated and ResourceNormalized topics).
• Milestone Impact: As throughput increases to thousands of patients per minute, Kafka lag becomes the leading indicator of system saturation.

7. Report State Persistence

• Monitoring Point: Report Service and its underlying database.
• Metric: Database write latency for status updates.
• Milestone Impact: The ReportService must handle thousands of status updates per minute. Database indexing and write-ahead logging (WAL) performance will be key at the 10k+ facility scale.

EHR Acquisition Performance & Constraints

Real-world observations from our first facility deployments indicate that data acquisition from external EHRs is the most variable and constrained part of the pipeline. These metrics may vary from hospital to hospital depending on the underlying EHR infrastructure.

Resource Query Performance

Based on one week of data (25,125 data acquisition logs across 2,991 patients):

• Avg Logs per Patient: ~8.4
• Avg Query Duration: 5.32 seconds
• Estimated Total Acquisition Time per Patient: ~44.7 seconds

Query Performance by Resource Type

The following table summarizes the average latency and data volume for common FHIR resource queries. Some resources, like Observation, are significantly more expensive to retrieve and represent much larger payloads.

Concurrency Constraints

To prevent overloading external EHR infrastructure and impacting hospital operations, concurrency is strictly limited:

• Max Threads per Facility: 8

Throughput Capacity (Facility Level)

Given an average patient acquisition time of ~44.7 seconds and a concurrency limit of 8 threads:

• Max Patients / Minute / Facility: ~10.7 patients

While this capacity is well above the average monthly requirement (~0.27 patients/minute), it limits the system's ability to "catch up" if backlogs occur or if reporting windows are short (e.g., daily submission cycles requiring all data to be pulled in a 2-hour window).

Reporting Cadence & Burst Load Model

The current state of this Performance Model design represents the "typical" daily workflow. However, at the end of each reporting period, the system must burst to a peak load to process all configured facilities' currently admitted patients.

An important variable in this model is the average number of patients in the hospital on any given day, since that determines how many patients the system will attempt to process at the end of each reporting period. This average should be refined over time as more data flows through the system.

Reporting Cadence Assumptions

• 100% of facilities report monthly.
• Roughly 40% of facilities also report daily.
• For monthly reporting, at the start of each month, each facility's currently admitted patients are queried, normalized, and evaluated.
• For daily reporting, at the start of every day, each facility's currently admitted patients are queried, normalized, and evaluated.
• There is overlap between these cadences: some facilities will burst daily, while the remainder will burst only at the monthly boundary.

Burst Load Expectations

Typical steady-state processing is expected to be around 10-20 patients per hour per facility during normal workflow. At reporting boundaries, however, the system may burst up to approximately 250 currently admitted patients per facility.

These burst values are intentionally approximate and should be refined as more production data becomes available so the performance model can better reflect real-world demand.

Peak/Burst Throughput Requirements

The table below shows the same throughput model as above, but sized for peak/burst reporting windows rather than normal operations. It assumes the system must process the currently admitted patient population for each facility within the burst window.

Observation & Tooling

We observe these metrics using the platform's standard telemetry stack as described in Telemetry.

• Prometheus: Collects custom metrics emitted by services (e.g., patient processing counts, phase durations).
• Grafana: Visualizes throughput vs. milestones. We should maintain a "Scale Dashboard" that compares current Patients / Minute against the targets defined in this model.
• OTEL Tracing: Provides end-to-end visibility using correlationId to identify where a specific patient's processing is stalled.
• Report Performance Analysis: Detailed granular tracking of pipeline phases is visualized in the Admin UI (front-end) as proposed in Report Performance Analysis.

Related Entities

• Events: event:ReportStatusUpdated, Resource Normalized
• Services: Report Service, Data Acquisition Service, Measure Evaluation Service, Normalization Service
• Proposals: Report State Machine Enhancements, Report Performance Analysis

Relationships

flowchart LR
nperformance_model_6F4F9EFC["Design: Performance Model"]