Distil Network’s Key Bot Traffic Findings for 2016

Categories

Distil Networks has just released its third annual Bad Bot Landscape Report. Distil has the largest library of bots in the industry and specializes in bot mitigation security. Bots can be a diverse group of website and API predators. They can perform high-speed attacks and a variety of nefarious activities including “web scraping, competitive data mining, personal and financial data harvesting, brute force login and man-in-the-middle attacks, digital ad fraud, spam, transaction fraud, and more.”

Not only are these attacks a nuisance to websites and users, but they are extremely costly to both productivity and the bottom line. A single attack last year to Dridex Banking cost the company a total of approximately $40.5 million. Analyzing the bot traffic of 2015 through data collected from its 17 data centers, several hundred customer and 74 billion bot requests that are contained in its Hadoop cluster, Distil hopes to shed some light on the trends in bot traffic and on how to mitigate bot attacks.

Page 13 - Onlien Users vs Bad Bots - Table

Distil Network’s Key Bot Traffic Findings for 2015

  1. Human Internet traffic has increased (now 54.4%) and good and bad bot traffic decreased (from 36.32% to 27.04% , 22.78% to 18.61% respectively)
    1. Humans now make up the majority of Internet traffic, a first since 2013, likely due to new Internet users in Indonesia, China, India
    2. Bad bots are mimicking our browser choices and Chrome was most popular in 2015 followed by Firefox
  2. Though fewer in number, bad bots become more advanced (advanced persistent bots; APBs)
    1. 88% of bad bot traffic (up from 77% in 2014) were evasive bot s(an 11% increase to 42%) or APBs (46%)
    2. Simple bots decreased from 23% to 12%
    3. To avoid security, APBs can mimic human behavior, load JavaScript and external resources, support cookies, perform browser automation, and spoof IP addresses and user agents
    4. 73% of APBs rotate through IP addresses and change their identities
      1. 36% of those APBs change IPs more than twice, 20% more than 100 times
  3. Bad bot attacks were seen most prevalently on medium-sized sites but humans remained the dominant traffic source on all site types and sizes
    1. Bas bots comprised 26% of all traffic on medium-sized sites (up from 17% in 2014)
    2. Small digital publishers (31%) and large real estate website (31% up 300% from 2014) were most prone to attack
      1. Likely due to bot attacks aimed at scraping of aggregate data by small startups
  4. 6 of the top 20 malicious ISPs originators were from China
    1. 90% of China Unicom IP Network’s traffic was malicious and all 6 averaged 72%
  5. Amazon remained on the top 20 malicious ISP originator list for the third year in a row likely due to its simple scalability and ease of use
    1. Verizon Business, Comcast and Time Warner are not on the list for 2015
  6. Netherlands, Korea and the U.S. topped the mobile bad bot traffic list
    1. Netherlands and U.S. were the only country with multiple carriers on the list
    2. Mobile-only traffic surpassed desktop-only traffic in 2015
  7. U.S. was the country in which the most bad bots originated
    1. Likely due to cheap cloud computing (Amazon, Google Cloud, Azure)
    2. Spain, Brazil, Ukraine, Switzerland, Australia added to the list of significant origins
    3. Many industrial companies made the top blocked list (Top 5: China, Norway, Germany, Netherlands, France)
    4. Maldives remains a bad bot hotspot with 536 bad bots/ person
    5. Worldwide average of bots per person rose from 26.1 in 2014 to 99.2 in 2015

Takeaways

Bots are becoming more advanced than ever with 46% of bad bots registering as APBs. Bot characteristics such as IP switching and loading of resources are becoming increasingly prevalent. If bots have the ability to load resources such as Java, they can skew analytic tools into thinking that they are human traffic. According to the this year’s report, “53% of bad bots will end up falsely attributed as humans in Google Analytics and similar tools” and will escape typical bot detection programs.

Another important characteristic of these bad bots is that they strive to find clever ways to mimic human behavior. For example, it was determined that spikes in bot traffic is typically seen between the hours of 9 am and 5 pm on weekdays, mimicking the typical work day. Approximately 39% of bots in 2015 were exhibiting human-like behavior. Without proper bot mitigation, many businesses, especially those with medium-sized websites, can suffer severe bot related impacts such as lost revenue. Though overall bot traffic has decreased, bots still pose a serious threat to websites worldwide.

Full report here

Scroll to Top