Spanner and Planet-Scale Services

Google recently published a much-anticipated paper on Spanner, their globally-distributed database. Spanner is the new pinnacle of Google’s technology stack, supplanting Bigtable at the cutting-edge of scalable databases. The authors will present the paper at the OSDI (Operating Systems Design and Implementation) ‘12 conference in October.

Google’s OSDI papers tend to be seminal: the MapReduce paper from OSDI ‘04 sparked the open-source Hadoop project, which in turn powered the industry-wide “big data” trend. And the Bigtable paper from OSDI ‘06 sparked a host of similar systems (HBase, Cassandra, MongoDB and others) that power services such as Facebook, Twitter, Foursquare and many more. Will Spanner spark a trend in the same way? It’s too soon to say, but there are arguments in both directions.

Spanner offers several novel and highly valuable features. Most of these are too technical to describe in this blog (which is primarily aimed at non-engineers). But one feature worth mentioning is that Spanner manages data across multiple datacenters. It is, in other words, the first truly planet-scale system.

Previously, systems were managed at the level of, at most, a single datacenter. Some companies can run for a while out of a single datacenter, but beyond a certain size you really need multiple datacenters, at different locations around the world, for two reasons:

  • Resilience: If an entire datacenter is down, say because it loses power or network connectivity, you can still keep your service up from another location.
  • Speed: One of the major contributors to slowness of a global service is the time it takes for data to traverse all the network connections between a user and a far-away datacenter. The closer you put the servers to a user, the better their experience. So, for example, you might serve West Coast users from a datacenter in California, but Asian users from a datacenter in Japan. 

Managing services across multiple datacenters has always been a technical challenge. You have to make sure that the data in all locations is consistent, and that you can switch seamlessly between them as needed. One of Spanner’s biggest features is that it takes care of all this complexity for you. It’s hard to overstate how useful this is. This sort of service would be a huge benefit to large-scale services such as Yahoo!, Facebook, Twitter, Foursquare and others.

However there are also reasons to think that Spanner may remain a rarified, Google-only technology for several years:

  • Complexity: Spanner is mind-bogglingly intricate. It took some of the top names in distributed systems years to develop, and the paper, though detailed, was still forced to omit many crucial details for the sake of brevity.
  • Technology stack: Spanner relies to a great extent on the rest of Google’s technology stack. Open-source equivalents of parts of this stack exist, but there are still gaps, and even the parts that exist are relatively immature.
  • Hardware: Spanner relies on two forms of time reference: GPS and atomic clocks, to synchronize operations between remote locations. This requires specialized, non-commodity hardware that is not widely available and with which the open-source community has little or no experience.

This last point is especially interesting. The uncertainty of computer clocks and the limits they place on distributed systems are well-studied but too complex to explain here. I will perhaps dedicate a future post to the issue. But suffice it to say that the requirement of specialized clock hardware in datacenters is a major barrier to open-source implementations of Spanner. 

Nonetheless, I expect the paper to spark debate and inspire the open-source community to create even bigger and better systems, for the benefit of companies large and small across the industry. And that can only be a good thing.