It was a rumor by The Information and now you know why their reputation/track record is so good:
It’s true and official now.
Here, I want to break down some thoughts and potential implications for us data practitioners (as usual, in an opinionated way).
1. Context - It makes a lot of sense
Have a look at the overlap/distribution across the different “disciplines” in the Data Engineer Lifecycle:
The idea behind the MDS (Modern Data Stack) - which both companies heavily promoted - was specialized, but open-tooling to do a few things, but to do them really good compared to the All-In-One alternatives.
The classic Best-of-Breed vs Suite trade-off you have in the software landscape.
The often-mentioned flaw of the best-of-breed approach is the interoperability between systems, which varies from little to big depending on the setup.
Out of these ideas spawned vendors that offer a managed, bundled MDS, but due to various factors, none of them has so far established itself as a dominant force, it seems.
During the ZIRP era, when funding was infinite, there was a specialized tool popping up for everything and all kinds of new “layers” were invented besides the classic “Integrate, Transform, Visualize, Orchestrate”. The MDS-Philosophy was taken to the extreme. Too many different solutions and too many tools for the sake of new tools. When the financial tailwind was gone, a lot of players expanded beyond their original specialization scope within the MDS to gain more revenue, clients, and ultimately justify their funding and keep growing.
The big consolidation was certain, but took a bit longer than expected.
As it’s widely agreed, in SAAS money is earned in the Enterprise-Space and less with notoriously broke Start- and Scale Ups. And as you see it with Informatica, Talend or the Microsoft Fabric offering, they simply prefer Bundles and Suites (along with visual drag-and-drop transformation pipelines for whatever reason)
As both players were clearly focusing on landing the big homerun deals instead of hundreds of small contracts in recent years, being now able to offer a unified offering just makes sense.
Since they were both VC-backed (and these positions are aimed to be closed within a 7-10 year period), it also makes sense from a shareholder perspective to do it now, as we see an AI-driven peak in the tech scene. In this context, I wouldn’t be surprised if Airbyte gets acquired by e.g. Snowflake in the near future.
2. Implications
So let’s go through the details mentioned in the press release.
“dbt will still be dbt. Fivetran will still be Fivetran.
(…)
I like to think about it like the Apple ecosystem: it all just works together, no duct tape”
As a convinced Android-User, I didn’t have a positive thought about it first.
I can imagine what it means (aka “it just works smoothly out of the box”)
vs how it first seems to me (aka “big brand, closed standards for a big premium”).
But in the end, I expect nothing less, than dbt transformations to be “natively” integrated with Fivetran syncs and vice-versa.
If so, the two-brand approach might turn out to be confusing for Enterprise Clients not familiar with all the history behind these brands, but let’s see.
Lots of guessing is involved before we know what will actually happen.
Here I have some doubts:
You can read George’s own post on this topic, but part of what made this merger such an obviously good idea to me was Fivetran’s complete alignment in how it sees this. Our collective commitments are as follows:
dbt Core and Fusion will both continue to be shipped under their current licenses.
We will continue to actively maintain dbt Core.
We will continue to support the dbt Community, via Slack, meetups, etc.
Is it sustainable to maintain 2 code bases (dbt Core + dbt Fusion), which more or less solve the same problem in the mid- and long-term?
Can you “simply” translate new features from dbt Fusion to dbt Core with AI? How much new functionality can we actually expect in dbt Core?
These doubts I had also before the merge.
Without knowing the internals, we need to trust these statements around dbt Core support, but at the very least, I’d expect more push and nudging into Paid Plans and Enterprise Plans (there’s also “Enterprise+” now)
And this is fine in general, as the current dbt Core is a very powerful, lightweight, and simply “good enough” solution for a lot of use cases which brings me to the last point:
3. Prepare for the future
Oliver Laslett from Lightdash posted the “fear index”:
A time-series of forks of the dbt-core-Repository. So, in other words, how many people quickly do a copy of the whole thing to be able to develop further on it on their own? As you can see, starting with the rumours in late September, the number of forks also jumped up.
Personally, I understand, but don’t share the same panic.
Call it naive, but I neither expect the repository to be taken private anytime soon nor that features will be “taken away”.
What will happen? Simply, nobody knows yet.
As mentioned, I expect a continued/strong push towards paid and enterprise features.
Worst-Case Scenario:
Irrelevant features (large Enterprise) or “stale” development without meaningful progress in the open-source space, but all innovation behind a paywall.
Core open-source improvements that don’t directly monetize will slow down.
Think of it more of a maintenance of adapters and functionalities with the developments of the data clouds (Snowflake, Databricks, BigQuery) in the ecosystem. We might also see other smaller players in e.g. Governance go out of market or be acquired themselves as their offering relied on a broad open-source userbase tightly coupled with dbt-core.
Best-Case Scenario:
Some new useful open-source features (hit me up if you need some nice ideas around data modeling) and maybe even some Airbyte/dlt-like Integration snippets chipped in by Fivetran.
We are already thinking about what existing software can be shipped under an OSS license (connectors? connector development kit?) and what standards we can invest in (or create).
Perhaps, the “bundled” price might even be a discount over purchasing Fivetran + dbt on its own.
So in my view, there is not that much to lose.
dbt Core will remain a viable solution, I am fully convinced of that.
And depending on the pricing strategy, maybe also the paid offering will continue to serve smaller companies, too. But we simply don’t know yet how the M&A will truly materialize in a few months.
But it’s not that you don’t have alternative options:
I recently looked into Bruin (open-source) and Bauplan Labs, which offer interesting new approaches, respectively, even though you can only partly compare them. If you want to stick to dbt, there is also Paradime.
So yes, we lost some competition with SQLMesh (also Dataform in the pre-Google days), but I still believe in enough viable options for us practitioners to choose from and enough competitive pressure to drive innovation by the vendors. I’m still asking myself how the Tobiko/SQLMesh acquisition fits into big picture (taking competition away? or just a safe bet in case the dbt Labs Merger doesn’t work out?), but we might never find out.
Will SQLMesh be integrated with dbt? Will it be the “native” Transformation-offering in Fivetran next to a “dbt integration”?
A theory I read online was that the acquisition was rather about SQLglot (which was open source anyway) and they want to deeply embed it into the Fivetran Iceberg REST Catalog to attack the Compute space (MotherDuck acquisition next?), but I personally doubt that.
How do you see the situation?
I’m very curious for the more strategic talk on dbt’s Coalesce kicking off today at least :)