DNS Attacks in the Wild

In a recent conversation with Robert McMillan (IDG), I described a in-the-wild attack against one of AT&T's DNS cache servers, specifically one that was configured as an upstream forwarder for an internal DNS machine at BreakingPoint Systems. The attackers had replaced the cache entry for www.google.com with a web page that loaded advertisements hidden inside an iframe. This attack affected anyone in the Austin, Texas region using that AT&T Internet Services (previously SBC) DNS server. The attack itself was not malicious, did not load malware, and from an operational standpoint, had zero impact. I contacted the ISP, worked with our IT folks to switch forwarding services, and wrote a cache auditing tool. I found the "wild" attack interesting, so in a conversation with Robert McMillan, I brought up the incident and forwarded the associated logs and notes. Shortly after our conversation, Mr. McMillan published an article with a sensationalist title, that while containing most of the facts, attributed a quote to me that I simply did not say. Specifically, `"It's funny," he said. "I got owned."`

Most of the facts of the article are correct. I have no problem detailing the attack, how it worked, and how we detected and resolved it. I am careful about the wording, because I want to be clear that while this type of attack can be serious, in this case it was a five minute annoyance that was designed as a revenue generator for the folks who launched it (click-through advertisement revenue). No systems were been compromised, no data was stolen, and most importantly, the target of the attack was the ISP, not the company that I work for. Stating that my company was "compromised" leads the reader to believe that there was some sort of security breach, which is reinforced by the fabricated quote. Mr. McMillan has since published a correction, but by the time this trickles down to all of the IDG publications, the damage will have been done. At this time (09:00 CST), the correction is posted, but the articles themselves have not been updated.

To add some content to my whining, I have included further details on the actual attack. The DNS server in question was dns1.austtx.sbcglobal.net (151.164.20.201). This system accepted recursive requests from anywhere (not just subscribers) and is the default DNS server for anyone who purchased SBC Internet Services (in our case, a T1 line that was our primary until our fiber was run). Internally, we use two DNS servers, one going out the fiber, other going out the T1 as backup. Early Tuesday morning, some of the friends and family members of BreakingPoint employees noticed that the iGoogle web page was returning a 404 from their home internet connections. Once our folks got to the office, they noticed that every once in a while, they could also reproduce it from within our network. Digging into it, we discovered that one of our internal DNS servers was still using SBC/AT&T as an upstream forwarder and that this server was returning the wrong results for www.google.com:

$ dig +short -t a www.google.com @151.164.20.201
www.l.google.com.
67.222.48.43

When querying the same address from the other DNS server, we saw the correct results:
$ dig +short -t a www.google.com @InternalDNS
www.l.google.com.
64.233.167.99
64.233.167.104
64.233.167.147

Requesting the main web page from the "poison" www.google.com server returned a very different response from the real Google server. This server was returning four iframes, one of which showed a fake version of the Google web site, the other three loaded automated ad-clickers from three other compromised servers.

$ echo -ne "GET / HTTP/1.1\r\nHost: www.google.com\r\n\r\n" | nc 67.222.48.43 80
HTTP/1.1 200 OK
Date: Tue, 29 Jul 2008 15:30:16 GMT
Server: Apache/2.2.9 (Unix) mod_ssl/2.2.9 OpenSSL/0.9.8g DAV/2 mod_auth_passthrough/2.1 mod_bwlimited/1.4 FrontPage/5.0.2.2635
X-Powered-By: PHP/5.2.6
Transfer-Encoding: chunked
Content-Type: text/html

161

<iframe width=100% height=100% frameborder=0 src="http://72.14.205.99/"></iframe>
<iframe width=0 height=0 frameborder=0 src="http://www.ramhost.org/ads.php"></iframe>
<iframe width=0 height=0 frameborder=0 src="http://www.medicaltechnic.net/ads.php"></iframe>
<iframe width=0 height=0 frameborder=0 src="http://www.funreport.net/ads.php"></iframe>

We changed the upstream forwarder for our internal DNS to point to a patched server (the ubiquitous BBN 4.2.2.x systems (OpenDNS has issues[1]), contacted the ISP, and wrote a cache validator that does not require host access to the DNS server (see the previous post for more information on that). The lesson -- even if your own DNS servers are patched, make sure none of those systems use an upstream DNS that has not. Since we contacted the ISP, this particular DNS server was taken offline. I found a list of regional SBC DNS servers and prodded them with the porttest.dns-oarc.net service. The end result was that of the 19 servers still online, 12 of them are still using static source ports, and each of these can be reached by anyone on the Internet. I wonder if they are waiting on ISC to fix BIND's performance problems.

-HD

1. OpenDNS returns poisoned records. Intentionally poisoned. For example, www.google.com points to a SQUID cache server run by OpenDNS, not the real google.com server. While I admire a service that improves security, I wonder about the impact of diverting private communications through their web cache servers. Does Google's privacy statement apply to the monitoring of traffic by OpenDNS -- I don't think so.

$ dig +short -t a www.google.com @208.67.222.222
google.navigation.opendns.com.
208.69.32.231
208.69.32.230

$ HEAD 208.69.32.231
200 OK
Cache-Control: private, max-age=0
Connection: close
Date: Wed, 30 Jul 2008 06:49:13 GMT
Via: 1.0 .:80 (squid)