Skip to content

05Team Member

StatSignal

Data Impact & Research Monitoring Platform

Role
Data Engineer / NLP Pipeline Developer – Team Member
Stack
PythonFastAPIPostgreSQLDagsterNLP ModelsOpenAI APIsClaude APIs
Links
Private / internal
localhost:3000/asset-groups/statsignal
Dagster/Asset lineage · statsignalMaterialize all

Sources

News
Research papers
Blogs
Policy documents

multi-provider adapters

L0 · raw

raw_documents

Fetched as-is, with source metadata

Materialized · 2h ago

L1 · normalized

documents

One schema across all providers

Materialized · 2h ago

L2 · enriched

entities

Named entity recognition

Materialized · 1h ago

sentiment

Sentiment per mention

Materialized · 1h ago

claims

Claims extracted with LLM APIs

Running · started 4m ago

L3 · marts

impact_scores

Weighted by channel: news, citations, policy

Materialized · 1h ago

campaign_mentions

Per campaign, per dataset

Materialized · 1h ago

PostgreSQL · lineage and every layer are kept for reproducibility and audit
Layered ELT pipeline (L0–L3) as Dagster assets. Representative UI mockup with sample data.

Overview

A multi-source data monitoring platform that tracks dataset usage across news, research papers, blogs and policy documents, with a layered ELT pipeline and NLP enrichment.

Problem

Tracking how datasets are used and cited across news, research papers, blogs and policy documents — reproducibly and auditably.

Solution

A multi-source monitoring platform with a layered ELT pipeline (L0–L3), NLP enrichment (NER, sentiment, claims extraction), campaign-based tracking of organizations and datasets, and a weighted impact score across channels.

Architecture

    • News
    • Research papers
    • Blogs
    • Policy documents
  2. multi-provider adapters → one schema
    • Layered ELT in Dagster — L0 → L1 → L2 → L3
  3. NLP enrichment (OpenAI / Claude APIs)
    • NER
    • Sentiment
    • Claims extraction
    • Weighted impact scoring
    • PostgreSQL
    • FastAPI

Features

  • Multi-source data monitoring
  • ELT pipeline
  • Data lineage
  • Reproducibility
  • Auditability
  • NER
  • Sentiment analysis
  • Claims extraction
  • Campaign-based tracking
  • Weighted impact scoring
  • Multi-provider adapters

Engineering work

  • Developed a multi-source data monitoring platform to track dataset usage across news, research papers, blogs, and policy documents.
  • Designed a layered ELT pipeline (L0–L3) ensuring data lineage, reproducibility, and auditability.
  • Implemented an NLP-based enrichment pipeline with Named Entity Recognition (NER), sentiment analysis and claims extraction.
  • Built a campaign-based tracking system to monitor specific organizations and datasets automatically.
  • Created a weighted scoring model to measure dataset impact across multiple channels (news, citations, policy).
  • Integrated multi-provider adapters to unify heterogeneous APIs into a single schema.

Engineering decisions

  1. 1Layered ELT (L0–L3) for data lineage, reproducibility and auditability.
  2. 2Multi-provider adapters unify heterogeneous source APIs into a single schema.
  3. 3Impact measured with a weighted score across news, citations and policy channels.

Code

pipeline/assets.pypython
@asset(deps=[l1_documents])def l2_enriched_mentions():    for doc in load_unenriched("l1_documents"):        save_enrichment(            doc.id,            entities=ner.extract(doc.text),            sentiment=classify_sentiment(doc.text),            claims=extract_claims(doc.text),        )
Simplified sketch of the approach — not production code.

Screens

localhost:3000/asset-groups/statsignal
Dagster/Asset lineage · statsignalMaterialize all

Sources

News
Research papers
Blogs
Policy documents

multi-provider adapters

L0 · raw

raw_documents

Fetched as-is, with source metadata

Materialized · 2h ago

L1 · normalized

documents

One schema across all providers

Materialized · 2h ago

L2 · enriched

entities

Named entity recognition

Materialized · 1h ago

sentiment

Sentiment per mention

Materialized · 1h ago

claims

Claims extracted with LLM APIs

Running · started 4m ago

L3 · marts

impact_scores

Weighted by channel: news, citations, policy

Materialized · 1h ago

campaign_mentions

Per campaign, per dataset

Materialized · 1h ago

PostgreSQL · lineage and every layer are kept for reproducibility and audit
Layered ELT pipeline (L0–L3) as Dagster assets. Representative UI mockup with sample data.
statsignal / campaigns / labour-force-survey

Campaigns

Labour force survey dataset · sample campaign

Last 90 days
All channels

Impact score

64 / 100

Contribution by channel

News42%
Citations33%
Policy18%
Blogs7%

Pipeline

L0–L1 up to date

Claims extraction running

SourceMentionEntitiesSentiment
Policy

State skills policy draft cites survey figures

Claim: “Youth unemployment fell compared with the previous survey round.”

Labour Dept.Survey datasetNeutral
News

Regional daily reports on employment trends

Claim: “Urban female participation remains low.”

Survey datasetNegative
Research

Working paper on informal employment

Claim: “Informal work share is estimated from unit-level data.”

Survey datasetStatistics officeNeutral
Blog

Explainer on how the survey is sampled

Claim: “The quarterly bulletin improves timeliness.”

Statistics officePositive
Campaign monitoring — mentions, entities, sentiment, claims. Representative UI mockup with sample data.

Challenges

  • Normalizing heterogeneous sources into one schema.