But which option should you go with? In this post, we’ll walk you through the seven things you’ll want to consider before you make a decision on whether you should build or buy your marketing data pipelines.
1. What are the business objectives and requirements?
When you’re evaluating your options, the very first thing you need to consider is the business objective: what type of data are you trying to move, from which data sources, and to which reporting, visualization, or data warehouse platform.
Building your own marketing data pipelines offers you free hands in terms of which data sources, metrics, and dimensions you can pull and to which destinations. However, in most cases, you’ll quickly notice that there’s already an existing solution out there that matches your exact needs. And if that’s the case, there’s usually no reason to reinvent the wheel and go through the trouble of building your own pipelines.
💡 TL;DR: If none of the existing marketing data pipelines on the market come even close to what you need, you might be better off building your own. However, in most cases it’s better to start with an off-the-shelf solution and customize it to your needs than to start from scratch.
2. Do you have the skills and resources in-house to build a reliable data pipeline?
By now, you should know exactly what kind of pipeline you need. The next thing you need to pay attention to is whether any of your existing employees know how to make that happen — and more importantly, whether they can be freed up to a big project like building and maintaining marketing data pipelines.
At first, you’ll need at least one person to work full-time on building the integrations. But depending on the number of connections you need, that number may have to grow exponentially. In general, creating your pipelines from scratch can take anywhere from a few weeks to several months.
Getting a managed pipeline up and running, on the other hand, typically takes somewhere between minutes and days, depending on the solution you choose. For reference, setting up a data warehouse in BigQuery with Supermetrics can be done in less than 10 minutes.
💡 TL;DR: Only commit to building your own data pipelines if you can free up the necessary resources for building and maintaining them.
3. How much downtime is too much?
Speaking of maintenance, you’ll also need to factor in resources for regular pipeline maintenance. The problem is that maintenance needs are difficult to predict, since the marketing platforms you’ll connect to can (and do) frequently make changes to their APIs. This means that you’ll either have to have someone on-call to fix any issues as they arise or alternatively get comfortable with the idea of some serious downtime in your data transfers.
From the maintenance perspective, buying your data connectors from an external partner nearly always comes out as the preferred solution. This is simply because managed data pipelines come with dedicated support and engineering teams that are there to answer your questions and fix any sudden issues.
💡 TL;DR: If you’re at all worried about downtime and can’t afford a full-time pipeline team, it’s almost always better to go with a ready-made solution.
4. Are pipeline building and troubleshooting really the best ways to use your team’s time?
The hard truth is that building reliable, secure, and user-friendly marketing data pipelines is quite difficult. And while your data engineers may be up for the challenge, it might not be the best use of their time. After all, weren’t they originally hired for conducting more strategic tasks like building your data strategy and overseeing its implementation?
Instead of making your well-paid data engineers build and fix data pipelines, why not let them focus on more strategic data initiatives? With fully managed pipelines, you won’t have to spend a lot of time on troubleshooting, since your partner will take care of everything from diagnostics to bug fixing for you.
💡 TL;DR: Most data engineers aren’t software developers, which means that their time is better spent on more strategic tasks than pipeline building and troubleshooting.
5. Can you build a sustainable solution?
Because building data pipelines from scratch is so time-consuming, companies tend to use the same pipelines for years. The problem is that older solutions typically contain legacy code that only one or two developers understand. And when these people change jobs, no one knows how to work the data pipelines anymore.
Instead of building new pipelines every couple of years with a new team, you’re better off with a vendor that has the resources to constantly update the pipelines and make sure your data is moving quickly and securely to where you want it.
💡 TL;DR: Data pipelines that are built in-house usually have a relatively short lifetime. Managed pipelines are typically a more sustainable solution.
6. What’s the most secure solution?
If data security is a big thing for you — like it should be — you might actually be better off with a managed solution that complies to your company’s data policy. And since we’re talking about moving data, you may want to choose a partner like Supermetrics that doesn’t store your data at all or move it to a 3rd party platform but simply encrypts it at the source and decrypts it at the destination of your choosing.
Instead of having your team go through the trouble of ensuring pipeline security, simply define your requirements and make a dedicated partner sweat the small stuff.
💡 TL;DR: Most data pipeline companies, including Supermetrics, are extremely conscious of data security. That’s why trusting a partner is often a more secure solution than building your own pipelines.
7. What’s the most cost-effective way of getting a working data pipeline?
Last but definitely not least, you’ll also want to think about the total cost of building your own pipelines versus buying them fully managed — including the time your data analysts may have to spend on digging data for marketers.
In the cost column, managed pipelines usually come out on top. Why? Because implementation only takes minutes, all the necessary maintenance is included in the price, and marketers can pull the data they need without bothering data analysts.
💡 TL;DR: While cost isn’t everything, buying managed pipelines is typically the less expensive solution.
Deciding whether to build or buy your marketing data pipelines comes down to these seven things:
And remember: if you’re leaning towards buying your data pipelines, you can always take a free 14-day trial of Supermetrics before you commit to anything.