Advanced Persistent Bot activity on the rise

Bad bots are used by fraudsters and are the key culprits behind web scraping, brute force attacks, competitive data mining, online fraud, account hijacking, data theft, unauthorized vulnerability scans, spam, man-in-the-middle attacks, digital ad fraud, and downtime.

In their annual report that identifies statistically significant data on global bot traffic, Distil Networks identified an influx of Advanced Persistent Bots (APBs). These can mimic human behavior, load JavaScript and external assets, tamper with cookies, perform browser automation, and spoof IP addresses and user agents.

“The persistency aspect is that they evade detection with tactics like dynamic IP rotation from huge pools of IP addresses, using Tor networks and peer to peer proxies to obfuscate their origins, and distributing attacks over hundreds of thousands of IP addresses,” said Rami Essaid, CEO of Distil Networks.

“As Digital becomes the norm, we will witness any attempt to provide a competitive advantage, and this will invariably involve the use of a bot of some description. Such tactics are not common knowledge, but their use is prolific and potential impact significant,” Raj Samani, VP and CTO EMEA at Intel Security, told Help Net Security.

Bot traffic

For the first time since 2013, humans outnumbered bots for website traffic. 46 percent of all web traffic originates from bots, with over 18 percent from bad bots.

Medium-sized websites (10,001 to 50,000 Alexa ranking) are at a greater risk, as bad bot traffic made up 26 percent of all web traffic for this group.

The rise of Advanced Persistent Bots:

• 88 percent of all bad bot traffic has one or more characteristics of an Advanced Persistent Bot
• 53 percent of bad bots are now able to load external resources like JavaScript meaning these bots will end up falsely attributed as humans in Google analytics and other tools
• 39 percent of bad bots are able to mimic human behavior so tools such as WAFs, web log analysis, or Firewalls, which perform less detailed analysis of clients and their behavior, will likely result in huge amounts of false negatives
• 36 percent of bad bots disguise themselves using two or more user agents, and the worst APBs change their identities over 100 times
• 73 percent of bad bots rotate or distribute their attacks over multiple IP addresses and of those, a whopping 20 percent surpassed 100 IP addresses.

Bot targets: Digital publishing and real estate websites

• Real estate websites saw a 300 percent increase in bad bot activity, with large real estate sites experiencing the most pain
• As an industry, digital publishers were hit hardest by bad bots, which make up over 31 percent of all their traffic
• For small digital publishers (Alexa 50,001 – 150,000) 56 percent of traffic originates from bad bots.

Huge increase in bad bot traffic from China

• Maldives, Israel and Kyrgyzstan had the highest Bad Bot GDP (number of bad bots per online user) at 526, 168, and 94 respectively
• China, Norway, Germany, and the Netherlands are the most blocked countries for web traffic
• The United States boasts the largest originator of bots again, with over 39 percent of bot traffic, while India and Israel moved up to number two and three, respectively.