Lydia LeongI was just thinking about the nameserver as a vantage point in the Microsoft CDN study, and I remembered that for the CDNs themselves, the nameserver is normally their point of reference to the customer.
When a content provider uses a CDN, they typically use a DNS CNAME to alias a hostname to a hostname of the CDN provider. For instance, www.nbc.com maps to www.nbc.com.edgesuite.net; the edgesuite.net domain is owned by Akamai. That means that when a DNS resolver goes to try to figure out what the IP address of that hostname is, it’s going to query the CDN’s DNS servers for that answer. The CDN’s DNS server looks at the IP address of the querying nameserver, and tries to return a server that is good for that location.
Notably, the CDN’s DNS server does not know the user’s actual IP. That information is not present in the DNS query (RFC 1035 specifies the structure of queries).
Therefore, what nameserver you use, and its proximity to where you actually are on the network, will determine how good the CDN’s response actually is.
I did a little bit of testing, which has some interesting results. I’m using a combination of traceroute and IP geolocation to figure out where things are.
At home, I have my servers configured to use the UltraDNS “DNS Advantage” free resolvers. They return their own ad server rather than NXDOMAIN, which is an annoyance, but they are also very fast, and the speed difference makes a noticeable dent in the amount of time that my mail server spends in (SpamAssassin-based) anti-spam processing. But I can also use the nameservers provided to me by MegaPath; these are open-recursive.
UltraDNS appears to use anycast. The DNS server that it picks for me seems to be in New York. And www.nbc.com ends up mapping to an Akamai server that’s in New York City, 12 ms away.
MegaPath does not. Using the MegaPath DNS server, which is in the Washington DC area, somewhere near me, www.nbc.com ends up mapping to a server that’s directly off the MegaPath network, but which is 18 ms away. (IP geolocation says it’s in DC, but there’s a 13 ms hop between two points in the traceroute, which is either an awfully slow router or more likely, genuine distance.)
Now, let’s take my friend who lives about 20 miles from me and is on Verizon FIOS. Using Verizon’s DC-area nameserver, he gets the IP address of a server that seems to live off Comcast’s local network — and is a mere 6 ms from me.
For Limelight, I’m looking up www.dallascowboys.com. From UltraDNS in NYC, I’m getting a Limelight server that’s 14 ms away in NYC. Via MegaPath, I’m getting one in Atlanta, about 21 ms away. And asking my friend what IP address he gets off a Verizon lookup, I get a server here in Washington DC, 7 ms away.
Summing this up in a chart:
| My DNS / CDN Ping | Akamai | Limelight |
|---|---|---|
| UltraDNS | 12 ms | 14 ms |
| MegaPath | 18 ms | 21 ms |
| Verizon | 6 ms | 7 ms |
The fact that Verizon has local nameservers and the others don’t makes a big difference as to the quality of a CDN’s guess as to what server it ought to be using. Here’s a callout to service providers: Given the increasing amount of content, especially video, now served from CDNs, local DNS infrastructure is now really important to you. Not only will it affect your end-user performance, but it will also affect how much traffic you’re backhauling across your network or across your peers.
On the surface, this might make an argument for server selection via AnyCast, which is used by some lower-cost CDNs. Since you can’t rely upon a user’s nameserver actually being close to them, it’s possible that the crude BGP metric could return better results than you’d expect. AnyCast isn’t going to cut it if you’ve got lots of nodes, but for the many CDNs out there with a handful of nodes, it might not be that bad.
I went looking for other comparables. I was originally interested in Level 3, and dissected www.ageofconan.com (because there was a press release indicating an exclusive deal), but from that, discovered Funcom actually uses CacheFly for the website. funcom.cachefly.net returns the same IP no matter where you look it up from (I tried it locally, and from servers I have access to in Colorado and California). But traceroute clearly shows it’s going to different places, indicating an anycast implementation. Locally, I’ve got a CacheFly server a mere 6 ms away. From California, there’s also a local server, 13 ms away. Colorado, unfortunately, uses Chicago, a full 32 ms away. Unfortunately, this doesn’t tell us much, beyond the fact that CacheFly has limited footprint; we’d need to look at a CDN with enough footprint that uses AnyCast to see whether it actually return results better than the nameserver method does.
So here’s something for future researchers to explore: How well does resolver location correspond to user location? How much optimization is lost as a result? And how much better or worse would AnyCast be?
Category: Infrastructure Tags: CDN
2 responses so far ↓
1 Adrian Otto November 26, 2008 at 7:57 pm
Network latency and download throughput are inversely proportional on a TCP/IP network. This is an undisputed fact. This is a fundamental principle that made Akamai attractive back in the 1990′s.
However, I think you’re putting a bit too much emphasis on network latency. Internet latency from within the US is not a dramatic problem on today’s internet. Backbones are fast, and have a lot of capacity nowadays. Even with ~100ms of latency, you can still easily deliver a high resolution video stream in real-time. This was not true a decade ago.
If you look at actual download performance rather than ICMP ping times, you will notice that some of the newer CDN’s with a smaller footprint than Akamai or Limelight can actually outperform them in some cases. Having fewer locations means less overall network complexity, and therefore a lower theoretical failure rate. They are easier to troubleshoot and maintain. It’s also less expensive to operate a smaller number of large facilities rather than a larger number of smaller ones. Smaller CDN networks can usually offer superior pricing for these reasons, and sometimes more reliable service and faster problem resolution as well.
Yes, an Anycast CDN design is better than one based on DNS, provided there are enough locations to support the viewer population.
Other factors also affect the performance of a CDN, such as how much they load up their servers with traffic, and how efficient thier HTTP servers are, which can affect the HTTP service response times, and effective throughput of data in spite of how close the servers are to the viewers.
Remember that reduction of network latency is not the only reason CDN networks are used. Sometimes it’s simply about getting enough raw capacity to deliver a high-volume rich-content web site. Content suppliers want to deliver content to viewers with performance that’s a blend of fast-enough and affordable-enough. Selection of a medium sized CDN is a sensible choice in these cases.
When the goal is to support a global viewer audience, having a CDN with “enough” locations (at least a few per continent) is smart, but simply selecting the one with the most pops globally might not result in the best actual performance to the viewer population.
Now that I’ve gone to the trouble of writing this lengthy post It might be ironic to mention that I use one of the larger CDN networks. I still am continually compelled by the smaller competitors as they become more and more established. It will be interesting to see which ones consolidate, and which ones remain over the next few years as budgets shrink due to changing economic tides.
2 Lydia Leong November 26, 2008 at 8:12 pm
Thanks for the thoughts.
If you read the rest of my posts related to the Microsoft study (or my body of published Gartner research, if you’re a client), you’ll see that I repeatedly stress that latency is only part of the overall CDN performance picture. Shorter network latencies certainly are beneficial, but there are a vast number of other factors affecting actual CDN performance. The point of this particular narrowly-focused post was to note that Akamai’s theoretical advantage of having lots of nodes close to end-users could be significantly reduced by the lack of sufficient nameserver granularity — a change from a decade ago when most people would be sitting on a local nameserver for a rinky-dink dial-up ISP that was serving a very local population.
The network failure rate is clearly theoretical, I’d say. The problem is that small CDNs often have less polished operations and haven’t yet come through the fire of scaling challenges, and consequently smaller and simpler frequently does not actually equate to better availability.
I’d also say that the cost argument doesn’t necessarily hold water. Bigger can also mean having achieved better efficiencies of scale. Smaller CDNs don’t necessarily offer lower prices because they’re actually cheaper to operate — they often offer lower prices because they’re willing to take a big hit on margins.
Much of your commentary is actually covered in the CDN selection notes that I’ve written. My work in this area tends to stress the business value of performance — of choosing a CDN that delivers “good enough” performance, rather than the best performance. That can even be a CDN whose performance isn’t better than the origin server, if it’s cheaper to deliver the bits via the CDN than to beef up the origin.