The Final Days of the Legacy Data Warehouse
Nearly 6 in 10 tech professionals are looking to switch and modernize their data warehouse environments, according to a new survey.
Out with the Old…
A new report from Ocient has me thinking about the state of data warehouses. But before I get into the report’s details, let me turn back the clock a few years.
I’ve been covering the data warehousing market since the mid 1990s when, as a reporter at InformationWeek, I attended an industry conference where I met Randy Mott, who was CIO of Walmart at the time. (From there, Randy went on to Dell, HP, and GM.)
Walmart had built a data warehouse that was the talk of the industry, in part because it was one of the largest in the world, but also because Walmart was using the system for market-basket analysis, sales history, inventory, and other business intelligence, which the retailer—in a strategy that was ahead of its time—shared with its suppliers and partners.
For the past 25 years, data warehouses like Walmart’s—big, centralized, on-premises systems—have been the model of choice at many companies. However, the heyday of those old-style enterprise data warehouses is coming to an end, or already has.
Today, a growing number of businesses are investing in newer, cloud-native platforms that offer faster queries, auto scaling, separation of compute and storage resources, built-in AI/ML, and multi-cloud availability—often at lower costs and sometimes managed by the cloud database vendor.
That brings me to the Ocient report. Ocient commissioned Propeller Insights to survey 500 data and IT professionals who manage workloads of 150 terabytes or more. The report, titled “Beyond Big Data: The Rise of Hyperscale,” is broadly about modern analytics, not just data warehouses. But it’s directly relevant.
According to Ocient, 59% of survey respondents are actively looking to switch data warehouse providers. Their reasons:
40% want to modernize legacy platforms
42% feel their existing system isn’t comprehensive enough
36% say it’s not flexible enough
…In with the New
No one should be surprised. Many businesses are transitioning from old-style analytics to newer, cloud-native platforms for the reasons I mentioned above.
This explains why Snowflake has become a tour de force. Its data cloud, data marketplace, and industry & departmental solutions are intuitive and accessible to data engineers and business managers alike.
Other disruptors include Databricks, Firebolt, Imply, Panoply, SingleStore, TileDB, and Yellowbrick Data. And the Big 3 cloud providers—AWS, Google Cloud, and Microsoft Azure—are introducing new analytics and data warehouse technologies, as well.
Ocient has developed its own high-end solution, which is capable of managing exabytes of data and analyzing trillions of records. Its software is available on AWS or Google Cloud, as a hosted solution, or for deployment on premises.
For more on Ocient, here’s my podcast with CEO Chris Gladwin.
Changing the Conversation
So, the days of the traditional data warehouse are numbered. In fact, pundits have been predicting the demise of the data warehouse for years. But maybe it would be more accurate to say that data warehousing is being reinvented and transformed, rather than ground into metal shavings.
The industry is adopting new terminology—data clouds, data lakes, etc.—to describe the transition that’s underway.
In a just-published article on CIO.com, tech journalist Stan Gibson (my former colleague) writes about “the rise of the data lakehouse” as an alternative to managing both data warehouses and data lakes, as many organizations do. He offers the example of Walgreens using a data lakehouse that is built with Databricks on Microsoft Azure—the epitome of a modern cloud data environment.
Here’s what Stan says about the challenges of the data warehouse: “The data warehouse requires a time-consuming extract, transform, and load (ETL) process to move data from the system of record to the data warehouse, whereupon the data would be normalized, queried, and answers obtained.” Many organizations, he writes, “are finding this paradigm of relying on two separate systems of insight untenable.”
Other data warehouse challenges identified by the Ocient survey are security and compliance (63%), scaling cost effectively (49%), and streamlining systems (48%).
You don’t have to look far for other examples of the changing landscape. I talked to two SingleStore customers—Christoph Malassa, head of analytics and intelligence solutions with Siemens, and Mauricio Aristizabal, principal data architect with Impact.
Both users described how they recently implemented new analytics capabilities on SingleStore’s unified database, which combines OLAP (online analytics processing) and OLTP (online transaction processing) in a single platform.
One of their primary objectives was to reduce query response times from seconds to milliseconds. On that, they have a lot of company. In Ocient’s survey, improved speed and performance are the #1 reason people are considering a new data warehouse, mentioned by 57% of respondents.
You can see a replay of our conversation at SingleStore’s [r]evolution 2022 virtual event here.
The Rush to Modernize
Which data warehouses are headed for the scrap heap? At the top of the list of legacy systems that survey respondents are looking to replace:
61% IBM DB2
49% Cloudera
41% Teradata
Traditional data warehouse providers know that their customers want newer cloud-native platforms, and they’re moving ASAP to modernize their offerings to stay in the game.
For example, Oracle’s Autonomous Data Warehouse got a major upgrade last year. And AWS recently announced availability of a serverless version of Amazon Redshift, its nearly 10-year-old data warehouse platform.
Teradata has been making strides with its old-school data warehouse, which is now available on AWS, Google Cloud, and Microsoft Azure. See my blog post below on Teradata’s progress—and stay tuned for more updates from Teradata.
Data Migrations Are Underway
No doubt, the traditional data warehouse has long served a vital function for many businesses. Given the large installed base and the level of investment they represent, they’re not about to disappear completely.
However, vendors are offering a growing number of tools and services for database migration. And you can be sure that those capabilities are being put to use moving terabytes and petabytes of data from legacy data warehouses to the new generation of cloud-native alternatives.
A big welcome to the many new Cloud Database Report subscribers! More people are signing up every day to receive my independent analysis on the good, the bad, and the ugly of the database market. It’s great to have you with us.