Skip to content

06Team Member

StatLens

AI-Powered Statistical Data Query Engine

Role
Backend Developer / Full-Stack Data Engineer – Team Member
Stack
PythonFastAPIPostgreSQLpgvectorReactNext.jsOpenAI GPT
Links
Private / internal
statlens / ask?q=GDP+by+state
GDP by stateAsk
  1. Parse question
  2. Decompose
  3. Select datasets
  4. Hybrid search
  5. Retrieve data
  6. Cited answer
State A has the highest GDP in the latest release, followed by State B and State C [1]. Figures are from the most recent annual series; two states report with a one-year lag [2].

GDP by state (index, illustrative)

State A100
State B86
State C74
State D69
State E57
State F44
State G38
StateIndexSrc
State A100[1]
State B86[1]
State C74[1]
State D69[1]
State E57[1]

Sources

[1] State GDP, annual series · official statistics dataset · retrieved live

[2] Release calendar / reporting notes · same publisher

Natural-language query with cited answer. Representative UI mockup with sample data.

Overview

A natural-language query engine for government statistics: users ask questions like “GDP by state” and receive structured, cited answers backed by hybrid search and an LLM query pipeline.

Problem

Government statistics are spread across many sources; users should be able to ask a question like “GDP by state” and get a structured, cited answer.

Solution

A natural-language query engine: a 10-phase ingestion pipeline normalizes datasets from 10+ sources, hybrid search (pgvector + BM25) finds relevant datasets, and an LLM pipeline decomposes the question, fetches data and writes a cited answer.

Architecture

    • Next.js / React UI
  2. FastAPI
    • Query pipeline — decompose → select datasets → fetch → answer (OpenAI GPT)
  3. hybrid search
    • PostgreSQL + pgvector
    • BM25 keyword search
    • Precomputed query slices
  4. 10-phase ingestion
    • 10+ sources — Census, WHO, World Bank, …

Features

  • Natural-language statistical queries
  • Multi-stage data ingestion
  • 10+ global data sources
  • Hybrid search
  • pgvector
  • BM25
  • LLM-powered query pipeline
  • Dataset discovery
  • Query decomposition
  • Real-time data retrieval
  • Precomputed query slices
  • OpenTelemetry monitoring

Engineering work

  • Architected a natural language query engine for government statistics, returning structured, cited answers.
  • Built a multi-stage data ingestion pipeline (10 phases) to collect and normalize datasets from 10+ global data sources (Census, WHO, World Bank, etc.).
  • Implemented hybrid search (semantic + keyword) using pgvector + BM25 for fast and accurate dataset discovery across millions of records.
  • Developed an LLM-powered query pipeline that decomposes user queries, selects relevant datasets, fetches real-time data, and generates human-readable insights.
  • Designed precomputed query “slices” to reduce latency for frequently asked statistical queries.
  • Integrated OpenTelemetry and monitoring tools for tracking LLM usage, latency, and system performance.

Engineering decisions

  1. 1Hybrid search — semantic (pgvector) plus keyword (BM25) — for dataset discovery.
  2. 2Precomputed query “slices” to cut latency on frequently asked questions.
  3. 3OpenTelemetry for tracking LLM usage, latency and system performance.

Code

search/hybrid.pypython
def find_datasets(question: str, k: int = 20):    semantic = vector_search(embed(question), limit=50)  # pgvector    keyword = bm25_search(question, limit=50)           # BM25    # combine both rankings into one ordered list    return merge_rankings(semantic, keyword)[:k]
Simplified sketch of the approach — not production code.

Screens

statlens / ask?q=GDP+by+state
GDP by stateAsk
  1. Parse question
  2. Decompose
  3. Select datasets
  4. Hybrid search
  5. Retrieve data
  6. Cited answer
State A has the highest GDP in the latest release, followed by State B and State C [1]. Figures are from the most recent annual series; two states report with a one-year lag [2].

GDP by state (index, illustrative)

State A100
State B86
State C74
State D69
State E57
State F44
State G38
StateIndexSrc
State A100[1]
State B86[1]
State C74[1]
State D69[1]
State E57[1]

Sources

[1] State GDP, annual series · official statistics dataset · retrieved live

[2] Release calendar / reporting notes · same publisher

Natural-language query with cited answer. Representative UI mockup with sample data.
statlens / datasets?q=GDP+by+state

Dataset discovery

Hybrid search: pgvector (semantic) + BM25 (keyword)
DatasetSourceSemanticBM25

State GDP — annual, current prices

Precomputed slice available

Census0.8612.1

Regional accounts — gross value added by state

Census0.798.4

GDP (current US$) by country

Precomputed slice available

World Bank0.719.7

Health expenditure as share of GDP

WHO0.526.3

Query decomposition

measureGDP

geographystate

periodlatest

unitcurrent prices

Top result selected → data fetched → answer generated with citations.

Dataset discovery with hybrid search scores. Representative UI mockup with sample data.

Challenges

  • Fast, accurate dataset discovery across millions of records.