Towards Data Freedom: ETL Pipelines

Welcome to the first installment in a series of posts we’re calling Towards Data Freedom! Our goal here is simple: cutting through the noise and gatekeeping around data, by eliminating the need for data practitioners to understand a million buzzwords and letting anyone be a data cowboy.

We’re starting with a doozy because we at Structify believe the biggest problems have the coolest solutions.

‍

What is an ETL Pipeline?

So, what IS an ETL pipeline? Starting with the acronym, ETL stands for Extract - Transform - Load (for those in the know, ELT because often the loading happens before the transformation step). Each of these pieces have their own intricacies and unique pain points, so stay tuned for follow-ups where we’ll do a deep dive into each of these pieces.

At a high level:

• ‍Extract: grab your data, be it from a CSV, internal database, or an API call‍
• Transform: manipulate, clean, re-format, derive on, your data‍
• Load: put your data where it needs to be, be it an S3 bucket or a simple database to load to a pretty Tableau visualization (link to tableau explanation)

Historically, ‘ETL pipeline’ is an incredibly loaded term. Its usage ranges from various one-off scrapers and ingestors that small go-to-market and similarly scoped teams use to get their data ALL THE WAY to robust, production-ready pipelines that sit as the backbone for companies as big as Amazon.

‍

Why do we care?

No matter which way you slice it, these are painful to build and maintain. A host of very similar and repetitive problems tend to arise:

‘Oh, this data isn’t clean at all’

‘The API’s data dictionary changed and now all my visualizations are broken’

‘My custom scraper broke this morning and I have to burn 3 hours to fix it before I actually start my day’

‘My upload is broken because an input contained unexpected null values my transform was supposed to catch but for some reason didn’t ‘

...and the list goes on.

We listened to our customers and turns out the biggest needs for them in any sort of tooling that helps with ETL is:

1. Automates the building
2. Makes iteration on the different pieces of the workflow a breeze
3. Makes inspectability easy and consistent

By giving users access to this suite of tools, we believe that the big data landscape is going to change positively and forever. Check us out to learn more and if this sounds useful to your team, send us a message: let’s build together.

Yours in data,
Alex Reichenbach