Today, Distil Networks, the San Francisco-headquartered cybersecurity company focused on bot detection and mitigation, published its 2018 Bad Bot Report, titled “The Year Bad Bots Went Mainstream”.
The report looks in-depth at the nature and impact of automated threats across 2017 from the perspective of Distil’s Threat Research Lab. The new lab analyzed 2017 data collected from its global network, including hundreds of billions of bad bot requests at the application layer, anonymized over thousands of website domains. The report’s focus on automated attacks to the application layer (layer 7 of the OSI model) rather than lower level volumetric DDoS attacks is what sets it apart from rival bot reports.
Bad bots are difficult to detect as they interact with applications in a similar way to legitimate users; however, they enable rapid abuse and attacks on websites and APIs, which can be used for a wide range of malicious activities from financial data harvesting to transaction fraud to account takeover.
Key findings of the 2018 report included:
- Bad bots made up 21.8% of total website traffic in 2017, which represents a 9.5% growth from the previous year.
- The gambling and airline industries have a significantly higher amount of bad bot traffic than other sectors. Bad bots account for 53.1% and 43.9% of all traffic, respectively. Next in line are the finance, healthcare and tickets industries.
- Russia became the most blocked country for the first time, making up over 20% of country-specific block requests. China, who was previously in the lead, fell to sixth place.
- Data centers became an even more significant breeding ground for bad bot traffic, which rose to 82.7% in 2017 – a 37% increase over 2016, largely as a result of the availability and low cost of cloud computing accounts. As most bad traffic emanates from data centers, the U.S. remains “the bad bot superpower”.
- Web browsers (Chrome, Firefox, Safari, Internet Explorer) act as the user agent for 83.2% of bad bots whereas mobile browsers (Safari Mobile, Android, Opera) only account for 10.4%.
- 74% of bad bot traffic is comprised of moderate or sophisticated bots, which avoid detection by mimicking browser technology (e.g. through the execution of JavaScript) or producing mouse movements that impersonate human behavior.
Distil Networks summarized their findings as “the year that bots went mainstream”, saying “No longer are bots the preserve of cyber security experts. Instead, even the FBI is investigating their use into influencing the results of the last US presidential election”. Bad bots accounted for more than one-fifth of all Internet traffic last year.
Twitter’s founder Jack Dorsey described the problem his platform and other similar companies faced in a recent Tweet: “We have witnessed abuse, harassment, troll armies, manipulation through bots and human-coordination, misinformation campaigns, and increasingly divisive echo chambers. We aren’t proud of how people have taken advantage of our service, or our inability to address it fast enough.”
Not only are social media, the wider media and political worlds reckoning what the impact of bad bots on democracy; but their wider impact on the economy is “grossly underestimated” in Distil’s words.
Legislation in relation to bot behavior is rapidly being created around the world, for instance, in concert and sporting event ticketing. The U.S. Congress banned the use of software that circumvents security on ticket seller websites in 2016, and the UK proposed altering the Digital Economy act to put a halt to bulk ticket purchases by bots.
Moreover, bots are targeting businesses with an online presence daily, for a variety of reasons, whether account takeover, denial of service or price scraping. Large and medium-sized sites are hit the hardest by bad bots. Distil’s report ends with a series of comprehensive recommendations for detecting bad bot activity.
In a press release announcing the new report, Tiffany Olson Jones, CEO of Distil Networks, said, “Despite bad bot awareness being at an all-time high, this year’s Bad Bot Report illustrates that no industry is immune to automated threats and constant vigilance is required in order to thwart attacks of this kind.”
A copy of the full report is available here; an interview with Anna Westelius, Senior Director of Security Research at Distil, summarizing its findings can be viewed here.