Before writing a single line of code, spend time understanding your data landscape. I’ve seen too many projects fail because teams jumped straight into implementation without properly mapping their data sources, understanding data quality issues, or defining clear success metrics.
Key questions to ask:
Every pipeline component should be idempotent. If you run the same process twice with the same inputs, you should get the same outputs. This isn’t just good practice—it’s essential for debugging, recovery, and maintaining sanity during 3 AM incidents.
Design your pipelines to fail quickly when something goes wrong, and make sure the error messages are actionable. Nothing is worse than a pipeline that silently produces incorrect data or fails with cryptic error messages.
Monitoring isn’t something you add later—it’s part of the architecture. Every pipeline should emit metrics about data volume, processing time, and data quality. Your future self (and your teammates) will thank you.
I’ve built successful pipelines with everything from cron jobs and Python scripts to sophisticated orchestration platforms like Airflow and Prefect. The tool choice matters less than having clear requirements and good engineering practices.
That said, here are some patterns that have served me well:
I’ve never seen a successful “big bang” data migration. Always plan for incremental migration with parallel systems running during the transition period.
Build for your current scale plus one order of magnitude, not for Google-scale unless you’re actually Google. Premature optimization in data pipelines often leads to unnecessary complexity.
Data quality issues compound over time. Implement data quality checks early and make them part of your pipeline, not an afterthought.
The data engineering landscape continues to evolve rapidly. New tools and paradigms emerge regularly, but the fundamental principles remain constant: understand your requirements, design for maintainability, and always keep the end user in mind.
What challenges have you faced building data pipelines? I’d love to hear about your experiences and lessons learned.
Have questions about data pipeline architecture or want to discuss a specific challenge? Feel free to reach out—I’m always happy to chat about data engineering problems.
]]>Before optimizing anything, you need to understand how your database is executing queries. Every major database system provides tools to examine execution plans:
EXPLAIN ANALYZESET STATISTICS IO ONEXPLAIN FORMAT=JSONThe key is learning to read these plans and identify bottlenecks. Look for:
Instead of indexing entire columns, create indexes on subsets of data:
-- Instead of indexing all orders
CREATE INDEX idx_orders_status ON orders(status);
-- Index only active orders
CREATE INDEX idx_active_orders ON orders(customer_id, order_date)
WHERE status = 'active';
Include frequently accessed columns in your index to avoid table lookups:
CREATE INDEX idx_orders_covering ON orders(customer_id, order_date)
INCLUDE (total_amount, status);
Index computed values that you frequently filter on:
CREATE INDEX idx_orders_month ON orders(EXTRACT(MONTH FROM order_date));
Replace expensive self-joins with window functions:
-- Instead of this expensive self-join
SELECT o1.customer_id, o1.order_date, o1.total_amount
FROM orders o1
JOIN (
SELECT customer_id, MAX(order_date) as max_date
FROM orders
GROUP BY customer_id
) o2 ON o1.customer_id = o2.customer_id
AND o1.order_date = o2.max_date;
-- Use window functions
SELECT customer_id, order_date, total_amount
FROM (
SELECT customer_id, order_date, total_amount,
ROW_NUMBER() OVER (PARTITION BY customer_id ORDER BY order_date DESC) as rn
FROM orders
) ranked
WHERE rn = 1;
For large datasets, EXISTS often performs better than IN:
-- Instead of IN
SELECT * FROM customers
WHERE customer_id IN (SELECT customer_id FROM orders WHERE order_date > '2024-01-01');
-- Use EXISTS
SELECT * FROM customers c
WHERE EXISTS (SELECT 1 FROM orders o
WHERE o.customer_id = c.customer_id
AND o.order_date > '2024-01-01');
Choose the right data types—it matters more than you think:
INT instead of BIGINT when possibleVARCHAR lengthsENUM types for limited value setsFor very large tables, partitioning can dramatically improve performance:
-- Partition by date range
CREATE TABLE orders_2024 PARTITION OF orders
FOR VALUES FROM ('2024-01-01') TO ('2025-01-01');
-- Distribute data evenly across partitions
CREATE TABLE orders_hash_1 PARTITION OF orders
FOR VALUES WITH (MODULUS 4, REMAINDER 0);
Set up monitoring for:
pg_stat_statements for query analysisBRIN indexes for time-series dataRemember that the most optimized query is useless if it doesn’t solve the right business problem. Always:
SQL optimization is both an art and a science. While these techniques can dramatically improve performance, remember that premature optimization is the root of all evil. Focus on the queries that matter most to your users and business outcomes.
The best optimization is often the simplest one—sometimes the answer isn’t a more complex query, but rather a different approach to the problem entirely.
What SQL optimization challenges are you facing? I’d love to hear about your experiences and discuss specific scenarios.
]]>These suggestions are not written in stone and, if you have a suggestion or disagreement, I would love to chat about it.
Update: you don’t have to memorize any of this. Download my
.sqlfluffconfig and let SQLFluff enforce it, or paste a query into the SQL formatter in the lab to see it in action.
with date_spine as (
select
day
from date_tables
)
, account_revenue as (
select
account_id
, created_date as joined_date
, revenue
from customers.revenue
where revenue > 0
and date(created_date) >= (select min(g.created_at) as first_at from customers.groups as g)
)
select
c.audience_id
, d.day as date_of_interest
, sum(ar.revenue) as daily_audience_revenue
from customers.groups as c
inner join date_spine as d
on c.created_at <= d.day
left outer join account_revenue as ar
on c.account_id = ar.account_id
where c.created_at <= ar.joined_date
group by
c.audience_id
, d.day
order by
c.audience_id
, d.day
CONTEXT DECOUPLED UNDERSTANDABILITY!!
The goal of constant SQL formatting is to improve development, review, and understanding time.
no capitalizing of keywords
keywords are already highlighted in the editor. the additional keystrokes are redundant.
as when aliasing columnsis_ prefix when naming boolean fieldswhere, on, when, etc.), leave on same line. For multiple, use new line2 spaces for indents
easier to manually do. no super long lines due to front spacing
Start columns below the select indented
Readability.
in values into multiple linesAlways end on new line
, & and)selectStyle guides only work if nobody has to think about them. Everything above (except the naming judgment calls) is encoded in a SQLFluff config:
Drop it in the root of your project (next to dbt_project.yml if you’re using dbt), then:
pip install sqlfluff
sqlfluff lint models/ # what's wrong
sqlfluff fix models/ # fix what can be fixed
It defaults to the Snowflake dialect and the Jinja templater. Change dialect for your warehouse, and install sqlfluff-templater-dbt if you want it to compile dbt refs.
“
The US standard railroad gauge (distance between the rails) is 4ft, 8.5in. This gauge is used because the English built railroads to that gauge and US railroads were built by English expatriates.
Why did the English build railroads to that gauge?Because the first rail lines were built by the same people who built the pre-railroad tramways, and that’s the gauge they used.
Why did those wheelwrights use that gauge then?Because the people who built the horse-drawn trams used the same jigs and tools that they used for building wagons, which used that wheel spacing.
Why did the wagons use that odd wheel spacing?For the practical reason that any other spacing would break an axle on some of the old, long distance roads, because this is the measure of the old wheel ruts.
So who built these old rutted roads?The first long distance roads in Europe were built by Imperial Rome for their legions and used ever since. The initial ruts were first made by Roman war chariots, which were of uniform military issue. The Imperial Roman chariots were made to be just wide enough to accommodate the back-ends of two war horses.
This story does not end there, however.Look at a NASA Space Shuttle and the two big booster rockets attached to the sides of the main fuel tank. These are solid rocket boosters or SRBs. The SRBs are made by Thiokol at their factory at Utah. The engineers who designed the SRBs might have preferred to make them a bit fatter, but the SRBs had to be shipped by train from the factory to the launch site in Florida. The railroad line from the factory runs through a tunnel in the mountains and the SRBs have to fit through that tunnel. The tunnel is slightly wider than the railroad track.
So, the major design feature of what is arguably the world’s most advanced transportation system was determined by the width of a horse’s ass.
Now, this story may not be 100% factual but, I like to believe it is pretty damn close and provides plenty of food for thought.
The headline in my mind is that design decisions propagate.
The initial constrain of “accommodate the back-ends of two war horses” got passed along over and over simply because the next step relied on the previous. So by the time NASA was designing their rockets it had to be taken into account.
At a macro level, we (humanity) have been building on top of what came before us for all of history. At each stage in the design tree of horse’s ass to space ship, no one said “welp gotta keep this to the original ass spec”. They stayed pretty close to it because it was already built. No need to re-do the work. What gets my attention though is that the rocket’s size could have been larger if the constraint wasn’t there. Would that have made them more effective? I don’t know but it doesn’t sit well with me that the design was limited based on the needs of the inital design.
Back down on a micro level, the decisions we make can propagate farther than we anticipated. Writing new backend code for your company’s web app? Well whatever comes next is probably going to build on top of it. Designing a database structure? Whatever comes next is probably going to build on top of it.
All of this is my long-winded way of saying: when you’re making something, take the time to understand constraints that might be piggy-backing in your work and if they are holding you back.