By October 2019 about 300 TB of audit log data-representing a quarter of all data stored in Edgestore-had been migrated to Alki, which was now in full production. At the end of each day, the team offloads the metadata from these tables into Amazon S3 for permanent storage, after which the tables in Amazon DynamoDB are deleted.īy the beginning of 2019, less than 6 months after the Alki team chose Amazon DynamoDB and Amazon S3, Alki was in its beta stage of production, ingesting all data and serving a subset of the reads. Then each of these tables stores 50–80 GB daily. Amazon DynamoDB acts as the hot storage layer, ingesting audit logging data to six DynamoDB tables at 4,000–6,000 writes per second per table. The Alki team, aided by AWS Solutions Architects, constructed a log-structured merge-tree (LSM tree)–based metadata storage system, which has two layers of data storage: an upper layer for hot metadata and a lower layer for cold metadata. We no longer need to plan for on-premises capacity and budget for hardware purchases and then be stuck with our decisions for 4 years.”īuilding Hot and Cold Metadata Stores Using AWS Solutions Both Amazon DynamoDB and Amazon S3 grow automatically with our capacity needs. But by using Amazon DynamoDB and Amazon S3, we simplify these problems because AWS handles many of the complex tasks like data replication, data durability management, and hardware provisioning. “These are problems that large teams take several years to solve. Amazon DynamoDB and Amazon S3 fit that need well-they are industry standards,” says Lee. “When building a storage system, you have to think about a lot of components, including replication, backups, and capacity management.
AWS Solutions Architects functioned like an extension of Dropbox’s Alki team, providing prescriptive guidance and implementation help. Using Amazon DynamoDB and Amazon S3, Dropbox rapidly prototyped and deployed a cold metadata store on AWS within just a year. Because the small Alki team faced a tight deadline that, if missed, could potentially lead to lost user metadata, it decided to implement managed services from AWS. They focused particularly on audit logging data Edgestore’s top cold metadata use case. “If you’re writing data that’s not meant to be read often, it’s extremely expensive to use-not to mention pointless to store-in mediums that are optimized for retrieval speed,” says Jonathan Lee, tech lead for Dropbox’s Alki team.Īs a result, two employees split off from the database team to build Alki, the solution that would cost-effectively store metadata. Additionally, it no longer made sense to store cold metadata in the same database as hot-or frequently use -metadata. Yet increasing the capacity of the on-premises database would require splitting existing partitions and buying new machines to host them, which would double the cost of Edgestore by adding millions of dollars per year. By mid-2018, the rapidly growing cold metadata-data that is accessed infrequently but needs to be stored durably and available instantly-was less than 2 years away from overwhelming Edgestore.
It has become one of the most successful startups in the world, with over 600 million users uploading more than 400 billion pieces of content.ĭropbox’s metadata stores were originally housed solely within the company’s main data store, Edgestore, hosted in an on-premises distributed database built on top of sharded MySQL clusters. This made room for virtually unlimited user metadata and not only saved the company millions of dollars-since it would not have to increase on-premises storage-but also reduced the cost per gigabyte by a factor of 5.5.įounded in 2007 by two Massachusetts Institute of Technology students, Dropbox is a global collaboration tool and file sharing service. Using Amazon DynamoDB, a fully managed, flexible NoSQL database that delivers single-digit millisecond performance at any scale, and Amazon Simple Storage Service (Amazon S3), a cloud object storage service, Dropbox rapidly developed a new managed storage system called Alki. Those circumstances pushed Dropbox to pursue a managed solution from Amazon Web Services (AWS). Dropbox had less than 2 years until its on-premises system would reach maximum capacity, and the implementation team for the project was made up of just two employees. The third option was the best, but achieving it would be a challenge.
Dropbox’s database team had three choices: double the on-premises storage capacity (which would cost millions of dollars), delete swaths of metadata, or find a new, highly scalable yet cost-effective solution. In the summer of 2018, Dropbox experienced a capacity crunch in its on-premises metadata store due to fast data growth in some of the partitions.
0 kommentar(er)
