New Tools Ease the Pain of Database Migrations

Database migration is a growing issue as more businesses move terabytes and petabytes of data to the cloud.

In the latest real-world example, PayPay's tech team published a blog post detailing their experience migrating data warehouse workloads from on-premises to Google Cloud's BigQuery.

"As we worked through this project, it became clear that data movement is very contextual to our setup, and off-the-shelf tools had limitations in being able to seamlessly copy data to Google Cloud Platform," the authors wrote. "This was the hardest part of the whole project." 

PayPal did not say how many months it took to move 15 petabytes, only that it was "less than a year." That's in the ballpark of other major database migrations I've written about, which have ranged from six months to two years. 

Clearly, these are complex undertakings. Kelly Stirman, Google Cloud director of product management for databases, said not long ago that most IT organizations view database migrations as "about the highest risk thing you can do in all of IT." 

AWS, DataStax, Microsoft, Oracle, and other vendors have introduced new tools to help with the process, which includes planning, data replication/synchronization, data models, and app logic. 

Just last week, Google Cloud announced Datastream, a change data capture and replication service, with an Oracle to PostgreSQL Toolkit. Google Cloud also offered a Schema Assistant for MySQL or PostgreSQL migrations to its Cloud Spanner. 

For more see, “Cloud Vendors Confront ‘Highest Risk’ Projects: Database Migrations”

And listen to the Cloud Wars Live podcast, “Database Migration: One Year or One Click?”
 
While simple database migrations can be accomplished in a few days, enterprise-wide projects sometimes go on for months or years. So it's essential that IT teams define clear timelines for these migration projects and ensure that business leaders understand what to expect.