Building a Modern Market Data Platform for ETRM

Executive summary

Every calculation in an ETRM starts with market data. Pricing, valuation, P&L, VaR, position keeping, scheduling, and settlement all begin from prices, curves, and the wider set of observations that describe the market. If that data is late, wrong, or ungoverned, every downstream number inherits the flaw. Market data is the fuel of the trading platform, and the quality of the fuel sets the quality of everything the engine produces.

Yet market data is often the least deliberately designed part of an ETRM landscape, accreted from vendor feeds, spreadsheets, and point integrations over years. This guide argues for treating it as a proper platform: a metadata-driven, event-driven market data platform with governed ingestion, validation, versioning, forward curves, streaming distribution, and reproducible history. It is written for the people who design this layer, CIOs and CDOs, quant teams, ETRM architects, and market-risk professionals.

It builds on the complete guide to modern ETRM and connects forward to forward curve construction, which goes deeper on turning prices into tradable curves. The recurring theme: trusted, governed, reproducible market data is the precondition for trustworthy pricing, risk, and AI.

What market data is

Market data is all the external and internally derived information used to price, value, hedge, settle, and report trades. It is far broader than exchange prices. It includes broker quotes and reference prices, forward curves and volatility surfaces, FX and interest rates, weather observations and forecasts, production and demand forecasts, storage levels, transportation tariffs, emissions prices, and operational constraints.

Some of this is raw and external; some is derived internally, a constructed forward curve is market data the firm produces from other market data. What unites it all is that it feeds calculations. Treating the full set as one governed domain, rather than a scatter of feeds and files, is the first step toward a platform that can be trusted.

Why market data is the foundation of ETRM

Almost every module consumes market data, which is what makes it foundational and what makes its failures so costly. Prices and curves flow into trade capture for enrichment, into position valuation, into mark-to-market, into VaR and stress testing, into scheduling, settlement, and reporting.

The implication is stark: a single stale or incorrect price does not cause one error, it causes a cascade. A bad curve mis-values every position that references it, which mis-states P&L, which distorts VaR, which misinforms hedging. This is why a market data platform must prioritise correctness and governance above all, and why the effort spent getting this layer right pays back across the whole system.

Types of market data

A market data platform has to handle a wide taxonomy, each type with its own business use. Exchange prices from venues such as ICE, CME, EEX, MCX, and NYMEX provide settled, authoritative references. Broker quotes fill the over-the-counter gaps. Forward curves express the term structure of price. Spot prices anchor the near term. FX and interest rates convert and discount. Volatility surfaces price optionality.

Beyond prices sit the data that increasingly drives energy markets. Weather data, temperature, wind, solar irradiance, rainfall, drives both demand and renewable supply. Fundamental data, demand and generation forecasts, storage levels, pipeline capacity, LNG vessel movements, describes the physical balance. And environmental-market data, carbon allowances, renewable energy certificates, guarantees of origin, prices emissions and green attributes. A serious platform governs all of these as first-class data, because in a renewables-heavy market the non-price data often moves the price.

Market data sources and ingestion

Market data arrives through many channels, and a platform needs a flexible connector framework to ingest them: exchange APIs, vendor feeds, broker FTP/SFTP, REST services, FIX market data, CSV and Excel uploads, internal pricing engines, IoT and telemetry, and weather providers.

The engineering considerations are consistent across sources: latency (how fresh must this data be?), licensing (what are we permitted to do with it, and who may see it?), reliability (what happens when a feed fails?), and normalization (how do we reconcile different formats, units, and conventions into one consistent model?). A connector framework that handles these concerns uniformly, rather than as one-off integrations, is what turns a scatter of feeds into a governed ingestion layer.

A modern market data architecture

A well-designed market data platform is layered, with each layer doing one job and passing clean data to the next. External providers feed a connector layer; a validation layer checks the data; a normalization engine reconciles formats and units; quality rules apply; a versioned repository stores it; a forward-curve engine derives curves; and an API gateway distributes everything to consumers.

The value of this layering is that concerns are separated and governed. Validation and quality are not scattered through the system but concentrated where data enters. Versioning is intrinsic, not bolted on. Distribution is through one governed gateway, not a tangle of direct database reads. This is the metadata-driven, event-driven design behind the Gravitas platform, and it is what lets the same data serve real-time trading and reproducible historical analysis without contradiction.

Forward curve management

Forward curves are the most important derived market data an ETRM produces, because they price everything with a future delivery. A curve is the full set of expected prices across future delivery periods, and managing curves means constructing them from market inputs, interpolating and extrapolating across gaps, shaping them to the right granularity, applying seasonal and calendar adjustments, and versioning every published curve.

Curves differ by commodity: power curves carry peak and off-peak shape and hourly granularity; gas curves carry seasonality and storage economics; oil, LNG, and carbon each have their own conventions. Because curves are so central, they deserve their own governed engine with full versioning, so any valuation can be reproduced against the exact curve version used. This is a deep topic in its own right, covered fully in forward curve construction for power and gas.

Reference data vs market data

Market data is often confused with reference data, but they are distinct domains that require different governance and must integrate cleanly. Reference data is the relatively static structure, commodities, products, instruments, locations, counterparties, units. Market data is the dynamic, observed or derived values, prices, curves, volatility, FX rates, weather, benchmarks.

They need separate governance models, reference data changes rarely and through approval, market data changes constantly and is governed by validation and versioning, but they must integrate seamlessly: a price is meaningless without the instrument and location (reference data) it applies to. A platform that models both well, and the relationship between them, is what makes every downstream number coherent.

Data quality and governance

Governance is what separates a market data platform from a pile of feeds. The essential capabilities are source validation, outlier detection, missing-data handling, effective dating, version control, audit trails, approval workflows, lineage, data stewardship, and retention.

In practice this means automated quality rules catch bad data at ingestion, a suspiciously large price move or a missing point is flagged for a steward rather than silently valuing the book. Effective dating and versioning ensure that the data used on any past date can be retrieved exactly. Lineage traces every derived value back to its inputs, so a curve can be explained and audited. This governance is not bureaucracy; it is what makes the data trustworthy enough to price real risk against, and it aligns with the broader governed analytical model.

Real-time streaming

In a fast market, market data has to move as an event stream, not an overnight refresh. Event-driven processing, using technologies such as Kafka, message queues, WebSockets, and streaming ETL, propagates a price update immediately to everything that depends on it: trade capture enrichment, the position engine, valuation, VaR, and dashboards.

The contrast with batch is decisive. A batch refresh means the platform values the book on prices that are hours old; streaming means valuation and risk track the market as it moves. For a desk trading intraday, this is the difference between a platform that keeps up and one that is always slightly behind reality.

APIs and distribution

Market data is only useful if consumers can get it cleanly, which makes distribution a first-class concern. A modern platform distributes through governed APIs: REST and GraphQL for query, webhooks and pub/sub for push, streaming subscriptions for real-time consumers, and bulk exports for analytics, all with authentication, authorization, and rate limiting.

The principle mirrors the rest of a modern ETRM: every capability is a governed service, not a direct database read. Consumers request prices, curves, and historical series through documented endpoints, with access controlled and usage governed. This is what lets many systems, and many users, safely share one trusted source of market data rather than each maintaining its own drifting copy.

Historical time-series and reproducibility

An ETRM must be able to reconstruct the past exactly, and that depends on rigorous historical time-series management: tick data, intraday prices, daily settlements, and curve snapshots, all effective-dated and versioned, with archiving and performance optimization for large histories.

Reproducibility is the goal. For audit, back-testing, and historical valuation, the platform must answer not just what a price is now but what it was, and what curve version was used, on any past date. A valuation rerun for a regulator or an auditor has to reproduce the original number exactly, which is only possible if every input was versioned and retained. This reproducibility is a defining requirement of a serious market data platform, and a common failure point of ad-hoc ones.

Security and compliance

Market data carries real security and licensing obligations. Vendor data is licensed, often with strict limits on who may see it and how it may be redistributed, so entitlement and access control are not optional. Alongside licensing sit encryption, role-based access, audit logging, secrets management for feed credentials, API security, regulatory retention, and business continuity.

Getting this right protects the firm on two fronts: it keeps licensed data within its permitted use, avoiding contractual and legal exposure, and it ensures the data that prices real risk is itself secured and auditable. A platform that treats entitlement and security as core, rather than afterthoughts, is what lets a firm ingest valuable third-party data safely. See how Gravitas approaches security.

AI and advanced analytics

Governed market data is the foundation that makes AI in trading reliable. With clean, versioned, lineage-tracked data, a firm can pursue price and demand forecasting, volatility prediction, anomaly detection, curve completion for illiquid periods, trade recommendations, risk-scenario generation, and natural-language copilots that answer questions over the data.

The essential caveat is that AI quality depends entirely on data quality. A forecasting model on inconsistent, ungoverned data produces confident but unreliable output. This is why the governance described above is a prerequisite for AI, not a parallel concern: the same versioned, validated, lineage-tracked data that makes pricing trustworthy is what makes AI trustworthy. See how Gravitas approaches AI, grounded in governed data.

Market data KPIs

The reliability of a market data platform can be measured, and a few KPIs keep it accountable.

These targets translate the goal of trusted market data into observable measures: availability and completeness ensure the data is there; validation accuracy ensures it is right; latency and publication time ensure it is timely. A platform that meets them gives the rest of the ETRM a foundation it can price real risk against.

Why the Gravitas market data platform is different

Gravitas treats market data as the governed foundation it is, built on the metadata-driven, event-driven architecture this guide describes.

Because the same governed data serves real-time trading and reproducible history, every downstream calculation, valuation, risk, settlement, reporting, rests on trusted, auditable inputs. And it is delivered at economics that suit desks the incumbents priced out. See how it is scoped or request a demo.