Financial Data Aggregation Platform

Technical Architecture

The system implements a layered architecture with clear separation between command layer (cron jobs), service layer (business logic), repository layer (data access), and entity layer (domain models). At its core is a factory-based provider architecture with common interface contracts enabling parallel HTTP requests and graceful error handling.

Multi-Provider Aggregation

Each data provider has a corresponding factory class that creates configured provider instances. All providers implement a common interface with clear separation between request and response handling, enabling all providers to initiate requests before any begin processing responses.

The system handles provider failures gracefully using isolated error handling. Each provider's request/response cycle is wrapped in exception handling to prevent one failing provider from affecting others.

Provider failures are isolated using try-catch blocks around handleResponse(). When a provider throws an exception, the system logs a critical error but continues processing other providers. This ensures one failing provider does not bring down the entire data collection process.

Trust Factor Priority System

Rather than averaging all provider responses, the system uses values exclusively from the highest-trust provider that returned valid data. This prevents low-quality data from polluting the final price calculation. The system falls back to lower-trust providers only when higher-trust sources fail.

The trust factor priority system differs fundamentally from standard averaging approaches used by other aggregation systems. Understanding this difference is key to appreciating the system's data quality guarantees.

Averaging all provider responses seems intuitive but introduces a critical flaw: a provider returning stale or incorrect data pulls the final price toward that bad value. For example, if four providers return £1,800/oz and one provider returns £1,600/oz (stale data from yesterday), the average becomes £1,760/oz, inaccurate despite most providers being correct.

Historic Data Management

The system maintains detailed historical price data with 15-minute granularity. Provider failures or market closures create gaps that need filling. Different commodities may need different timestamp normalisation. Historical queries must be efficient for charting and analysis.

Dual Table Architecture

The system maintains two distinct data tables: LatestCommodityPrice tracks current values per provider (one record per provider-commodity pair, updated in place), whilst HistoricCommodityPrice stores aggregated values with timestamp uniqueness (one record per commodity-timestamp pair, never updated).

A single table approach would require date filtering for “latest” queries and create index conflicts between provider+commodity (for latest) and commodity+timestamp (for historic). Separating concerns prevents these conflicts and simplifies queries.

Timestamp Normalisation

Timestamps are normalised to 15-minute boundaries before storage. A price arriving at 14:23 becomes 14:15. This enables database unique constraints on commodity+timestamp, direct chart queries without transformation, and straightforward gap detection.

Gap Filling Service

When providers fail or markets close, the system fills gaps using the last known good value. For each missing timestamp, the service looks up the most recent historic price and replicates it forward to maintain continuous data coverage.

The gap filling service generates 96 records per day (24 hours × 4 quarters). For each 15-minute interval, it checks if a historic record exists. If not, it uses the last known price from earlier in the day (or the previous day if necessary).

Measurable Outcomes

The platform delivers stable, accurate pricing data whilst gracefully handling provider failures. The architecture choices result in a maintainable, testable codebase with clear separation of concerns.

Code Quality Metrics

• 296source files - Well-organised codebase with clear layering
• 299test files - Near 1:1 test-to-code ratio providing confidence in financial calculations
• 11,175lines of code - Application code (excluding tests and vendor dependencies)
• 20database migrations - Schema evolution spanning 5 years of development
• 27API endpoints - RESTful API for consuming applications

The system aggregates data from 6 commodity providers and 4 currency providers, supporting 10 precious metals (gold, silver, platinum, palladium, rhodium, tellurium, ruthenium, rhenium, indium, gallium) across 14 international currencies (USD, EUR, GBP, AUD, JPY, CAD, SGD, MYR, AED, CHF, HKD, CLP, BRL, MXN).

Architectural Benefits

• Resilience: System continues operating when individual providers fail, with trust factor priority ensuring data quality
• Extensibility: New providers can be added by implementing factory and feed interfaces, no changes to core logic required
• Correctness: Precision arithmetic with bcmath functions and thorough testing ensure financial-grade accuracy
• Observability: Detailed logging tracks which providers succeed/fail and which trust level was used for each calculation
• Maintainability: Clear separation of concerns between layers with well-defined interfaces and near 1:1 test coverage

The trust factor priority system is fundamentally different from standard averaging approaches. Rather than treating all providers equally, it recognises that data quality varies and explicitly prioritises reliable sources over mere availability.

The dual storage model (latest + historic) prevents index conflicts and optimises for different access patterns. Timestamp normalisation at storage time simplifies querying and enables database-level duplicate prevention.

Industry Applications

The patterns demonstrated in this case study are applicable across multiple industries requiring multi-source data aggregation, precision calculations, and graceful degradation.