Associate DataEngineer
Associate Data Engineer building ETL pipelines with Python, pandas, and SQL — from raw data ingestion through to clean, tested datasets ready for analytics.
London-based · no sponsorship required (EU Settlement Scheme)
Daud Abdi
Associate Data Engineer
12
Projects
8
DE Pipelines
2 Yrs
Experience
Core Expertise
End-to-end data pipeline development
dbt modelling, testing & Snowflake deployment
Airflow orchestration & Docker containerisation
Source-aware design: schema contracts and data quality upstream
Engineering Data at Scale
Building a strong foundation in data engineering, one pipeline at a time
Associate Data Engineer
I'm an Associate Data Engineer based in London, building ETL pipelines with Python and pandas — ingesting raw data, cleaning and transforming it, and converting it into efficient formats like Parquet for downstream use. I'm also comfortable with dbt, Airflow, and PostgreSQL, and I'm always looking to build on that foundation.
My approach is hands-on and production-focused: I understand the systems that produce the data, not just the pipelines that move it — which means I design for data quality and schema contracts at the source. Real APIs, real databases, real orchestration, clean architecture.
12
Projects
2 Yrs
Experience
Data Pipeline Development
End-to-end pipelines from ingestion to analytics with Python & dbt
dbt & Snowflake
SCD Type 2 snapshots, dbt models and tests, dual-target deployments
Airflow Orchestration
DAG-based pipeline orchestration with Docker Compose deployments
Data Source Understanding
Designing for data quality and schema contracts at the source, not just in the warehouse
Technology Stack
Primary Languages
Pipeline & Transformation
Database & Cloud
Projects & Case Studies
Real-world data pipelines and backend systems built for production
NHS A&E Waiting List Pipeline
End-to-end pipeline ingesting NHS England A&E attendance and emergency admissions data over 5 years (April 2021 – February 2026), at provider level across NHS Trusts and Foundation Trusts — 11,986 rows from 59 monthly files. Python ingestion (BeautifulSoup/Requests), PySpark transformation into Snowflake, dbt modelling, orchestrated via Airflow on Docker.
UK Land Registry Pipeline
Big data pipeline processing 10,057,373 property transactions from HM Land Registry (2016–2026). Python ingestion loads ~1.7 GB of CSV data into Snowflake, with DuckDB for local exploration. dbt models produce national price trends, property type breakdowns, and London borough-level analysis — 7/7 tests passing.
Fintech Fraud Detection Pipeline
Production-grade fraud detection pipeline ingesting 590,540 IEEE-CIS transactions. Features 12 engineered fraud signals, rule-based scoring (0–100), Apache Kafka streaming for real-time alerts, dbt models (6/6 tests passing), and end-to-end Airflow orchestration — all Dockerised.
Crypto Stream Pipeline
Real-time streaming pipeline ingesting live BTC/ETH price ticks from the Binance WebSocket API through Apache Kafka into PostgreSQL. dbt models produce 1-minute OHLC candlesticks and rolling volatility metrics.
UK Labour Stress Pipeline
Databricks-native pipeline ingesting ONS regional labour market data and producing a composite stress score index across UK regions. Uses a medallion architecture (Bronze → Silver → Gold) with PySpark, Delta Lake, and dbt models.
BoE Stats Pipeline
Data pipeline extracting macroeconomic series from the Bank of England IADB API — base rate, GBP/USD, GBP/EUR, mortgage rates, and M4 money supply. dbt project with 14/14 passing tests, staging and mart layers, orchestrated via Airflow.
FCA Register Pipeline
End-to-end data pipeline ingesting the UK FCA financial services register. Features SCD Type 2 snapshots, dbt models and tests (12/12 passing), and dual-target deployment to both local PostgreSQL and Snowflake.
UK Crime Pipeline
Batch data pipeline extracting 17,000+ crime records from the UK Police API. Stores data as Parquet, loads to PostgreSQL, and extends to AWS with S3, Glue, and Athena for cloud analytics.
Tech Stack & Expertise
Modern data and backend technologies for building scalable, production-ready pipelines and systems
Languages
2 technologies
- Python
- SQL
Data Engineering
9 technologies
- dbt
- Apache Airflow
- Apache Kafka
- PySpark
- Pandas
- Parquet
- Delta Lake
- Snowflake
- Databricks
Cloud (AWS)
3 technologies
- AWS S3
- AWS Glue
- AWS Athena
Databases
4 technologies
- PostgreSQL
- MySQL
- Database Design
- Query Optimisation
Data Ingestion & APIs
2 technologies
- FastAPI
- REST APIs
DevOps & Tools
5 technologies
- Docker
- Git
- Linux
- Postman
- CI/CD
Credentials
Formal training that underpins the stack I use day-to-day
DeepLearning.AI & AWS
CompletedData Engineering Professional Certificate
A four-course program covering the full data engineering lifecycle — generating, ingesting, storing, transforming, and serving data — with hands-on labs building batch and streaming pipelines on AWS. Taught by Joe Reis, co-author of Fundamentals of Data Engineering.
Meta
CompletedVersion Control (Git & GitHub)
Git and GitHub fundamentals for collaborative software development — local repositories, commits, branching and merging, conflict resolution, and team workflows using remote repositories and pull requests.
Shipping value across teams & clients
Production-focused delivery with clean architecture, documentation, and reliability top of mind.
Mar 2025 — Present
Associate Data Engineer
Upwork · Freelance
- Built ETL scripts in Python and pandas to extract, clean, and transform structured and semi-structured datasets.
- Converted raw CSV/JSON exports into Parquet files to reduce storage size and speed up downstream processing.
- Automated data ingestion and transformation workflows, scheduling runs to keep datasets up to date.
- Validated data quality with checks on schema, nulls, and duplicates before loading into target tables.
Oct 2023 — Feb 2025
Money Transfer Agent
Taaj Services · Full-time, Remote
- Built Excel reconciliation trackers to validate daily transaction volumes and balances across multiple currency corridors.
- Automated recurring reporting workflows in Excel using formulas, pivot tables, and conditional formatting to cut manual reporting time.
- Cleaned and structured raw transaction exports in Excel ahead of loading into internal systems, improving downstream data accuracy.
Contact
Let's build something together
Whether you're looking for a data engineer for your next project, or just want to discuss an idea, feel free to reach out. I usually reply within 24 hours.