How Microsoft's 'Scalability Day,' 25 Years Ago, Set the Stage for the World of Big Data
The SQL Server on Windows NT demo provided a glimpse into the future
On May 20, 1997 — exactly 25 years ago — Bill Gates hosted ‘Scalability Day,’ a milestone event in New York City that was intended to demonstrate that Microsoft software and systems could handle the world’s most-demanding data and application workloads.
I was there. The demo, at the Sheraton Hotel & Towers on 7th Avenue, showcased Microsoft’s ability to support one billion transactions per day; terabyte-size databases; 50,000 email accounts; and 100 million website clicks.
Those numbers may sound quaint today, but they were pretty impressive at the time. These days, we increasingly hear about exabyte-size databases, which store a million times more data than Microsoft’s SQL Server 6.5 did back then.
Scalability Day was intended to demonstrate Microsoft’s enterprise computing prowess and change perceptions. In 1997, Microsoft was viewed first and foremost as a PC software company. Its server software — Windows NT, SQL Server, Exchange email, Transaction Server, etc. — were not on par with Unix data center systems from Sun Microsystems and others. At least, that was the general consensus.
Bill Gates, who was Microsoft’s Chairman and CEO at the time, wanted to prove that Microsoft belonged in the big leagues of transactions, applications, and data management.
“As we showed today, any business of any size can now run its enterprise applications on Microsoft software and industry-standard hardware,” Gates said at Scalability Day. “Combining enterprise-class scalability with PC-industry volume economics will radically reshape the enterprise market.”
Gates added that Microsoft and “the industry” would deliver even greater levels of scalability in the future. That future is now.
Scalability Day wasn’t a smash success. One analyst noted that some of the key capabilities weren’t commercially available. “Scalability Day Falls Short,” said one headline.
Yet, history shows that Microsoft succeeded in proving that it could handle big data and data center workloads. Microsoft is now the world’s leading database vendor, having surpassed Oracle and IBM, according to Gartner.
And Microsoft Azure cloud services, including virtualized computing, SQL Server and other databases, Synapse Analytics, AI/ML, Data Lake Storage, and more, are available at “hyperscale.”
Commodity hardware
Microsoft partners — Compaq, Dell, EMC, HP, Intel, NCR, SAS Institute, Seagate, and many others — were in full force at Scalability Day. Hasso Plattner, SAP’s vice chairman and cofounder, was a featured speaker.
Yet, as we look in the rear view mirror, the passage of time and technologies is also evident. Amdahl, Baan, Data General, Digital Equipment Corp. They were there, too.
The stars of the show were Windows NT and SQL Server running on commodity PC configurations.
In one of the benchmark setups, Microsoft simulated a bank ATM network that was configured to support 1.6 billion customers, which is pretty impressive even today. The software/hardware configuration cost about $5 million in 1997 dollars. Assuming 1 trillion transactions over 3 years, Microsoft estimated the amortized cost at 5 millionths of a dollar per transaction.
There were four such prototypes. The following descriptions are taken directly from the Scalability Day press kit, so the time references are circa ‘97:
1.1 billion transactions per day. Microsoft Windows NT Server 4.0, Microsoft Transaction Server 1.0 and Microsoft SQL Server 6.5 were used to process the equivalent of 1.1 billion ATM transactions in a 24-hour period using a multi-node configuration of Compaq hardware equipped with Intel Pentium Pro processors. This is greater than the combined volume handled by the five largest American Banks.
Terabyte-size databases. The next version of Microsoft SQL Server, code-named Sphinx, which goes into beta testing next month [June 1997], was used to access a terabyte-sized database of mapping information and satellite images of the Earth. The database used for the demo contains more than 140 million records, including both global mapping and image data, provided by the US Geological Survey and the Russian Interbranch Association SOVINFORMSPUTNIK (Spin-2).
Large-scale Internet email system. A single Digital AlphaServer 4100 running Microsoft Exchange Server was demonstrated supporting over 50,000 Internet mail boxes — twice the size of the largest known system on a single-node mainframe.
100 million Internet hits per day. Microsoft Internet Information Server, running on a single HP NetServer LX equipped with Intel Premium Pro processors, was used to demonstrate more than 100 million Web hits per day.
Microsoft also rolled out 64-bit memory for Windows NT Server 5.0 and SQL Server Sphinx, with this timeless soundbite: “For data warehousing customers, improved response times means faster decisions.”
And there was more: Microsoft’s new (at the time) clustering software, code-named Wolfpack, provided automatic system failover with SAP’s R/3 software running on Tandem servers.
A million times more
Scalability Day pre-dated cloud computing and big data, and the modern, cloud-native architectures that we now take for granted. Yet, you can see a dotted line from the use cases that were on display at Scalability Day to today’s peta- and exa-scale databases and workloads.
The leaps in scale are remarkable. Twenty five years ago, Microsoft Exchange Server supported 50,000 Web email accounts. Today, Google’s Gmail supports an estimated 1.8 billion users.
Even that pales compared to Website traffic. On Scalability Day, Microsoft demoed Windows NT handling 100 million hits per day. Now, Google’s search engine processes 8 billion queries per day — 80 million times more.
And, as noted above, the business world has begun to move from terabyte-size databases to exabytes. That’s 1 million times more data over the span of 25 years. [As a point of reference, iPhone 13’s are now available with 1 TB of storage.]
These examples are not exactly apples-to-apples comparisons, but they provide a general sense of how data, transaction, application, and web workloads keep growing linearly and in some cases exponentially.
No doubt, the trend will continue. In its most recent DataSphere report, IDC estimates that worldwide data will double from 2022 to 2026.
Where’s Microsoft now?
I asked Microsoft to provide an update on its scalability story for this Scalability Day article, but they missed my deadline. The invitation remains open.
In the meantime, the volume, speed, and complexity of data and transactions continue to exceed yesterday’s thresholds across the industry, enabled by entirely new technologies. Now, open source is used throughout the tech stack. AI & ML drive automation. Serverless allows for auto scaling. Compute instances and storage are available as a service, and they can be decoupled to allow for performance and scale based on requirements. And the list goes on.
Pure, raw scalability is not the proving ground it used to be. If businesses need more scale, they get it from their cloud providers — AWS, Google Cloud, Microsoft — which continue to build expansive global data centers for that very purpose.
I will leave you with one example — Nasdaq, then and now. In 1997, the stock exchange touted its reliance on Microsoft’s BackOffice and server software to support Nasdaq Online, its website for stock issuers, and MarketSite, billed as the world’s first giant-screen digital information system displaying stock market activity in graphical format. MarketSite ran on Windows NT 4.0 and 74 Pentium processors.
Fast forward to last year when Nasdaq announced plans to migrate its North American markets to AWS, deploying AWS edge technology (Outposts) in Nasdaq’s own data center.
In short, the world has moved from running websites and digital displays on servers to entire exchange operations in the cloud. It’s a remarkable transformation of tech infrastructure and data capabilities over 25 years.
If history is a guide, this super scalability will only continue. Which means that 25 years from now, in 2047, an exabyte of data storage could be available on your iPhone and yottabytes in the data center.