Automated Multistage Data Pipeline Platform

An end-to-end data engineering platform that ingests SEC financial statement data, orchestrates multi-format ETL pipelines via Apache Airflow, stages data in AWS S3, and serves it through a cloud data warehouse (Snowflake) — complete with a REST API and an interactive analytics dashboard.

Overview

Public SEC financial filings contain high-value data but querying them at scale requires a proper data engineering foundation. This platform automates the full lifecycle:

• Ingestion — scrapes quarterly ZIP archives from SEC.gov programmatically

• Multi-format transformation — processes the same source data three ways (raw TSV, JSON, and denormalized fact tables) to benchmark storage and query tradeoffs

• Cloud staging — uploads all artifacts to AWS S3 before loading

• Warehouse loading — bulk loads into Snowflake using COPY INTO for performance at scale

• Data access layer — FastAPI REST API for programmatic access

• Analytics UI — Streamlit dashboard for interactive SQL exploration and CSV export

This project demonstrates core data engineering competencies: pipeline orchestration, ELT design patterns, cloud storage integration, data modeling (star schema fact tables), and serving structured data through APIs.

https://github.com/pjsk02/Automated-Multi-Stage-Data-Pipeline-Platform