A Million Times More Data. Yes, It's Coming.
Data architects must plan for exponential growth as data stores swell from petabytes to exabytes and beyond.
There was a time when a terabyte was considered “big data.” I once wrote a cover story for InformationWeek magazine titled “Towering Terabytes” that explored the emergence of terabyte-size data warehouses. That was 26 years ago.
These days, a terabyte (TB) isn’t a big deal. You can buy a 1TB storage drive for $20. Many organizations now manage petabytes (PB) of data, and some have exabytes (EB). An exabyte is a million times more than a terabyte.
AWS, Google Cloud, Microsoft, and Oracle all talk about exabyte-size data stores, and they’re showing up in more places. For example, Apple reportedly had 8EB of data in Google Cloud in 2021. And AWS has a data migration service, called Snowmobile, for moving exabytes—a 45-foot shipping container on wheels to physically transport the data from a customer’s data center to the AWS cloud.
That brings me to my golden rule of database scalability: Expect—and prepare for—a million times more data over the course of your career.
I’ve …