Seven years turning slow processes into systems.

Colombia's national census arrived as raw, inconsistent extracts that had to be hand-cleaned and joined to cartography before anyone could put it on a map. Each delivery took weeks, and every refresh started from zero. COVID-19 raised the stakes: case data had to reach a public map every single day.

Python pipelines that normalized the census tables, validated every record-to-geometry join and published map-ready layers automatically — built on ArcPy and the ArcGIS API for Python, with R and Bash handling the heavier reshaping. For COVID-19, a scheduled job read the daily case feed, reshaped it and pushed it straight into a REST feature service with no human in the loop.

Batch ingestion ran on Apache NiFi. A full cycle took around six hours, which meant the business opened its day without fresh data — and the flow-based setup was hard to scale, hard to observe, and hard to reason about when a run failed at 4 a.m.

I migrated the batch path to Apache Spark jobs on AWS Glue, orchestrated by Apache Airflow. That brought partitioned and incremental loads, dependency-aware scheduling, retries, and real visibility into every run — feeding the Snowflake data marts used by BI, Data Science and Operations. NiFi came out of the batch path entirely.

RappiPay was applying for a banking license in Colombia. That meant regulatory reports in exactly the formats the regulator expects, on a fixed calendar, assembled by hand every cycle — and cardholders who needed their credit-card statement records in near real time, not the next morning.

I modeled the regulatory reports in Snowflake — dimensional models plus stored procedures — and automated the whole chain end to end: generation, formatting, and delivery of the report files into the government's shared folder. Alongside it, a streaming path built on Apache NiFi and Apache Kafka delivered credit-card statement records in near real time.

A conflict-of-interest application needed to answer questions spanning around ten disconnected sources — Oracle, PostgreSQL and flat files — with relationship traversal between entities and instant per-company lookup. None of that infrastructure existed yet.

The first end-to-end pipeline on the account: ingestion from every source into a multi-layer data lake on Apache Iceberg — raw, curated and consumption — processed with Spark on AWS Glue, queried through Athena and orchestrated with Step Functions, Glue and Lambda. From the consumption layer I bulk-loaded into Amazon Neptune for the relationship graph and OpenSearch for fast per-company lookups: the two engines the .NET application reads from.

A consultancy's services business had no shared data model. Every report was rebuilt from scratch, a full reporting run took around ten hours, and when a number looked wrong nobody could trace where it came from.

I designed and developed the facts and dimensions for the services model natively in Snowflake — roughly 20 models across ~50 tables — with ingestion automated through Airbyte, orchestration in Airflow, and transformation in Snowpark scripts and stored procedures, all inside the Snowflake ecosystem. Then I built a Streamlit app that renders lineage, so an analyst can trace any metric back to its source table instead of asking an engineer.

A business-critical platform ran its ELT as hand-written Spark jobs on AWS Glue, with legacy SSIS packages still sitting in the path. It was expensive, memory-bound, opaque when it broke — and one critical pipeline took eight hours to finish.

I completed the migration to a governed ELT stack: Airbyte for extract and load, dbt for transformation, Amazon Redshift as the warehouse — infrastructure deployed with Terraform and everything orchestrated by Airflow. Around 10 GB/day now flows in from Oracle, SQL Server, OpenEdge and IBM DB2. SSIS packages were replaced with Airflow DAGs, and the Spark pipelines that stayed were re-engineered to eliminate memory bottlenecks.