Skip to main content

Data platform and API · Analytics

Data platform and public API for a US analytics company

A data platform that joins more than 15 outside data providers into one PostgreSQL warehouse of about 160 tables, keeps it current with more than 50 scheduled sync jobs and serves it through a documented public API.

Illustration, not a screenshot of the product.

At a glance

Client
A US analytics company
Sector
Data and analytics
Location
United States
What we did
Data engineering, Back end and API, Web application
Status
In production
Our role
Data engineering, back end, API and web application

Key figures

  • 15+outside data providers joined
  • 50+scheduled sync jobs
  • About 160tables in one PostgreSQL warehouse

The challenge

The client is a US analytics company that publishes data-driven projections. Its product depends on data from many outside providers, each with its own format and update schedule.

One reliable, current source of data for the company’s web application and for a public API, plus the means to run large-scale simulations and send alerts in real time.

What we delivered

  • One data warehouse

    PostgreSQL, about 160 tables, joining 15+ outside data providers that each have their own format and schedule.

  • Scheduled sync jobs

    50+ jobs keep every source current, with alerts when one fails.

  • Public REST API

    Documented with OpenAPI, so outside developers build on the same data.

  • Simulations at scale

    Large-scale simulations run directly on the data to produce the company’s projections.

  • The web application

    User accounts and real-time alerts, built on the same data.

How it’s built

Built with Next.js, React, TypeScript, Express, PostgreSQL and Redis, and hosted on Vercel and Render.

  • Next.js
  • React
  • TypeScript
  • Express
  • PostgreSQL
  • Redis
  • OpenAPI
  • Vercel
  • Render
How the data platform fits together More than 15 outside data providers feed more than 50 scheduled sync jobs, which load a PostgreSQL warehouse of about 160 tables. The warehouse serves a public REST API, the web application with accounts and alerts, and large-scale simulations. 15+ outside data providers each with its own format and schedule 50+ scheduled sync jobs PostgreSQL warehouse about 160 tables, kept current Public REST API OpenAPI docs Web application accounts, alerts Large-scale simulations run on the data Hosted on Vercel and Render
A simplified view of the platform.

Results

In production, with development continuing. A walkthrough is available to evaluators on request.

References available on request.

Want something similar?

Send us the work package. We reply with a written plan, team and price within 2 working days.