DNS, the Recent Apple iOS update and the Future

A few weeks back it came to our attention that some of our customers were struggling to download the recent update to their Apple iOS devices. All initial diagnostic tests could find no errors or problems. We then became aware of the discussions on various fora that if the user changed their DNS resolver to either OpenDNS or Google DNS that it solved the issue. Again, we tested our DNS resolvers and they could successfully look up the relevant record.

However it was in doing this test that we spotted what the cause was – the packet size! When performing a query for appldnld.apple.com, one of the diagnostic team saw that the size of the returned packet was well over 512 bytes:

;; MSG SIZE  rcvd: 729

However Google and OpenDNS were returning much smaller payloads, e.g. from Google:

;; MSG SIZE  rcvd: 172

At Zen we use BIND9 for our DNS resolvers in a standard configuration. In this configuration the server would also return the “Authority” and “Additional” data sections in the DNS reply, along with the “Answer” section. It is the latter that is the important part of the reply, whereas the former two are optional but useful when it comes to some diagnostics.  Since the optional data may be obtained if required we took the decision to configure the DNS resolvers for “minimal-responses”. This reduced the packet size and fixed the issue.

Now, the question that needs to be asked is “Why did packets greater than 512 bytes in size cause a problem?” The issue here lies with how DNS was implemented originally, and how it needs to behave in a modern Internet.

In the original specification of the DNS protocol, defined in RFC 1035 (Request For Comments) back 1987, was that no UDP packet should exceed 512 bytes in size – the vast majority of DNS lookups are made via the UDP protocol.  Should the packet size exceed 512 bytes, then the process should fall back to using TCP instead. It is this fall back mechanism that failed with regard to the iOS update. Most users will be operating their broadband router in its default configuration in which it forwards their DNS requests for them, eg :

PC <–DNS–> Router <–DNS–> Zen <–DNS –> World

Many xDSL routers don’t implement the TCP side of DNS when they can’t handle large packets, which is something I’ll come back to later. So, when Zen sent back a large packet the router failed to understand it, and also failed to try TCP, resulting in the iOS devices not being able to look up where to obtain their downloads. By changing the router, or even the device, to use Google or OpenDNS resulted in successful lookups.

This problem of large packets was considered many years ago. Back in 1999 RFC 2671 was put forward, suggesting a mechanism for larger packets to be handled, but still using UDP. This mechanism has been accepted, and implemented (to some extent), and is known as EDNS.

EDNS is important in a far wider sense these days, and that is in the realm of DNSSEC. This is the digital “signing” of DNS records. This is relevant to security applications, for example how do you know that the IP address you’ve received for www.your.bank.com is actually that bank and not someone pretending to be it? Consider DNSSEC as a similar mechanism to HTTPS in the browser realm – it provides a trust mechanism. The downside to DNSSEC is it produces much bigger packets – bigger than even that appldnld.apple.com one.

At Zen we made sure that all our infrastructure was capable of handling the larger DNS packet sizes used for DNSSEC, and other EDNS payloads, back at the start of 2010 when the DNS Root servers were to be signed. Unfortunately we’re still bound by what the broadband routers are capable of. A study from Nominet back in late 2008 showed that DNSSEC within router hardware was poorly supported. Much has changed since then, but EDNS support is still poor within consumer hardware.

We can hope that support for EDNS and the like will appear in new hardware, and firmware for older equipment as time goes by. Zen and many other bodies within the ISP realm are working with CPE manufactures to push for support for these “new” mechanisms within the Internet.

This was certainly an unusual problem to diagnose, and we certainly thank our sharp eyed engineer performing the diagnosis for spotting the problem

•