Wednesday Feb 9th 2005
A new breed of online extortionists is threatening to attack Web sites unless the companies pay up. Some big-name sites, including Microsoft and Google, were saved from harm last June with concerted combat against an army of bots.
At 8:29 a.m. Eastern Daylight Time on June 15, 2004, Microsoft.com, Apple.com, Google.com and Yahoo.com, along with almost 1,100 other Web sites, were chugging along nicely. Traffic was moving around the Internet in a typical volume for that time of the day. These customers of Akamai Technologies were, in general, getting faster response times than average because many of the requests for their content were being served locally from one of 15,000 Akamai servers in 69 countries.
Yahooligans were particularly optimistic; the company was preparing for yet another upgrade to its free e-mail service, which competed directly with one from Google.
At 8:30, things started to get ugly.
A botnet army swung into action. Thousands of personal computers around the world had been infected by viruses and turned into zombies, controlled by an unknown attacker through private chat channels using the Internet Relay Chat (IRC) protocol. Acting in concert, the bots flooded a set of Domain Name System servers with requests.
The flood quickly threatened to overwhelm the servers, which belonged to Akamai, a Cambridge, Mass., company that supplies online infrastructure to large Web operations and says it routinely handles 15% of total Internet traffic.
The bot attack forced delays in figuring out what the requests were and where they should be directed. Everyday Web users-customers of the sites-were left waiting online for even routine requests to find their way through the Web.
ZDNet, Silicon.com and other tech news outlets covering the attack at the time described Microsoft, Apple, Google and Yahoo as having been blacked out. Akamai would not confirm the customers targeted, except to say that neither any customers nor any Akamai servers were taken completely offline.
The attack was unusual because of the precision with which the attacker picked his targets. Rather than bomb Apple or Microsoft directly, or attack all of Akamai's servers simultaneously, the bots focused on Akamai's DNS servers. More specifically, those primarily serving Microsoft, Apple, Google and Yahoo, whose volume of traffic and profile make them among the highest-visibility targets on the Web.
Big Business For Cybercriminals
Not unusual, however, was the fact that the botnet attack focused on a company or companies with a major online presence. So-called denial-of-service attacks, which were once the exclusive tool of pranksters and vandals, have become big business for criminals who have found that extortion can be done online as easily as in person.
Online credit-card processing vendor 2Checkout, for example, reportedly rebuffed such a demand for an extortion payment last April and was hit with a series of denial-of-service attacks lasting more than a week. Another credit-card processor, Kentucky-based Card Solutions International, also was hit with a similar attack in April, after its owner refused what he said was a demand for $10,000 from a group of Latvians. Credit-card processing service Authorize.net received a series of attacks in September after refusing to respond to a demand for money that was sent to its general e-mail box.
In August, Saad Echouafni, head of a satellite data reseller called Orbit Communications, was indicted by a federal grand jury in Los Angeles for allegedly launching distributed denial-of-service (DDOS) attacks on three of his competitors, resulting in losses to those companies ranging from $200,000 to more than $1 million. The 37-year-old Moroccan subsequently disappeared, according to the FBI, which suspects he may have fled the country.
The indictment was the first for a distributed attack launched purely for commercial purposes, according to the FBI, which has put Echouafni on its most-wanted list.
Bots also are believed to have helped spawn the growing scourge of financial fraud from "phishing" for credit-card information on the Web. From September to October, the number of phishing sites doubled, according to Websense, which says the spike was probably due to the ability of bot networks to spew out e-mail as well as host the imitation sites that "phish" for individuals' financial information.
Given its high profile, Akamai is not an unlikely target for a bot attack, but it's not an easy target, either. Rather than connect its 15,000 servers by a private network, Akamai locates them in 69 countries and connects them via the fabric of the Internet itself, which uses enough transmission channels that it's almost impossible for one packet flood to choke them all.
The result of the attack wasn't instantaneous, but it didn't take more than a minute or so for people at Akamai to notice, according to Andy Ellis, the company's director of information security. He spotted the slowdown himself when his request for a customer's site was slow to resolve, indicating one of his DNS servers was misbehaving. Another try got the same result.
But even as virtual alarms went off in his head, real ones flashed in the Network Operations Command Center (NOCC) down the hall, where traffic
volumes spiked on Akamai's DNS servers and intrusion-detection specialists started calling for help.
First to respond was the White Hat team, a hand-picked group of Akamai's best architects, operations, security and development engineers. Always on call, the team follows a precisely structured emergency response procedure.
The initial step is a conference call to define the problem and divide up the tasks involved.
"By the time I called the NOCC to say we needed to initiate the White Hat conference bridge, they were already dialing," Ellis says. "By the time I got on as the host a few seconds later, there were already four people waiting."
The first problem is figuring out exactly who and what is under attack, which isn't as obvious as it seems. If Microsoft were under a denial-of-service attack in which thousands of computers each sent bogus requests for connections at the same time, much of the traffic would flow through Akamai's servers. In this case, however, loads were spiking all over Akamai's DNS servers, not just on those for one or two particular customers.
Ellis won't talk about specifics, but typical strategies to fend off a denial-of-service attack include rate limiting (servers are configured to accept only a certain number of requests per second) and packet filtering (suspect packets are simply ignored or turned away by the server). To filter packets, however, you have to know what kind are involved in the attack and at what source they're addressed.
That often requires on-the-fly reconfiguration of filters that are already in place. And rate limiting can slow the response of servers that Akamai's customers use specifically because response times are fast.
Both of those techniques can make the problem worse, in fact, by restricting legitimate traffic and accepting bogus requests that look legitimate because the botnet is programmed to hide the IP address from which they send the packets, and to change the address from which they're pretending to work.
Botnets can also avoid telegraphing their presence. When a virus infects a machine, it often uses Address Resolution Protocol requests to find other potential targets on the network. That jabber helps security administrators discover the problem earlier and nail it down sooner by shutting off the ports the virus attempts to penetrate. Botnets, however, can be programmed to target a specific range of IP addresses; that makes them more stealthy and effective because all the traffic they generate is aimed at specific targets rather than on attempts to expand, as in a virus attack.
Ellis says Akamai has gotten good at identifying distributed attacks quickly, separating legitimate from illegitimate packets, and shutting down access from addresses that seem to be the real source of attacks. The quickest way to respond is to identify the specific attack pattern and get teams of experts working on different parts of the effort to shut it down.
To The Bridge, On The Double
"We had the full team on the bridge in a matter of minutes," Ellis says. "We had about 20 people on the phone. There were about nine of us in an 8-by-8 office with three conference bridges open and a raft of cell phones."
Ellis split the team into subgroups, one of which had charge of forensics-capturing packets and decoding the request pattern to profile the attack and recommend countermeasures.
Other teams fanned out to notify federal law enforcement agencies and Internet service providers, both to help them shut out any subsequent attack and to ask for any help they could offer. A third group evaluated the impact the attack was having on customers, which, after all, hire Akamai to keep attacks like this one from affecting them.
One team launched an application the intrusion-detection group had developed that was designed to filter and identify rogue packets more quickly than the existing set of custom-developed tools Akamai used. "We were waiting to put it in until we needed it," Ellis says.
"We had the attack mostly mitigated within about 90 minutes, but we got a little lucky," he adds.
Normally, address-spoofing makes it hard to identify the zombies in a botnet, let alone the machines that are controlling them. But while an Akamai network architect was warning a colleague in a university data center about the attack, the data-center manager noticed there was not only a lot of traffic streaming from his site to Akamai, but also a lot of IRC traffic. It turned out the attacker was controlling the invasion through corrupted machines within the university's network. It didn't take long to shut the controllers down.
At the request of the data-center manager, who prefers not to be known as the supervisor of the launching point for the most serious attack on Akamai during 2004, Ellis won't reveal the location of the data center: "We sent the FBI that way, though." So far, no one has been arrested for the attack.
Akamai did well by responding to the attack that quickly, according to Johannes Ullrich, chief technology officer for the Internet Storm Center, a unit of the SANS Institute security education company. The attack's focus on the DNS servers made it a serious challenge for Akamai, he maintains. "If they get 10 major attacks a year and this is the only one that had any real impact, that's pretty good," Ullrich says.
Akamai can handle north of a billion bits per second, has a distributed-network design and a full-time, fast-response security team. That's why customers sign up with it rather than try to build similar capabilities themselves. "They're where you go to help avoid DDOS attacks. You don't want to go there to find them," Ullrich says.
Akamai said only 1% of its 1,100 customers suffered enough in the attack that more than 20% of their visitors saw any effect. Only 2% of its customers saw any performance degradation, according to the company.
Still, a problematic attack on Akamai makes the case stronger for botnet masters whose goal isn't just to interrupt Internet traffic, but to make money off it.
"Extortion is big right now," says Marty Lindner, team leader for incident handling at the Computer Emergency Response Team (CERT) Coordination Center at Carnegie Mellon University. "If I have a botnet army and you have a big commercial Web site, I can go to you and say, 'Pay me a lot of money or I'll DOS you off the face of the Earth.'"
Though it doesn't track the number of extortion attempts connected with botnet attacks, Symantec's Internet Security Threat Report estimates that during the first six months of 2004, the number of actively controlled compromised machines on the Internet went up from 2,000 per day to more than 30,000 per day on average, with peaks to 75,000 in one day. About 16% of botnet attacks are against commercial Internet sites, an increase of 400% from 2003, which may indicate a shift toward profit as a goal rather than reputation.
How Much Security Is Enough?
Companies with less sophisticated or less responsive security would have been in much worse shape, according to Ullrich. But no one can prevent a bot attack completely. "You just do what you can to defend yourself," he says. "If you're hardened from the beginning, maybe they'll go on to someone else."
But no matter how many security people a company hires and how secure its own network might be, no organization is sophisticated enough to remain unscathed in the face of a denial-of-service attack from tech-savvy extortionists, Lindner says. "At the end of the day, the problem is compromised computers," he explains. "If they control those, they can send 10 times as much traffic as you have the capacity for. No matter how many security pros you hire, you still lose."
Akamai Base Case