Web scraping is not just a security or fraud problem

Bots compose 42% of overall web traffic, and 65% of these bots are malicious, according to Akamai.

Negative effects of scraper bots on business operations

Web scraping is not just a fraud or security problem, it is also a business problem. Scraper bots have a negative effect on many facets of the organization, including revenue, competitive edge, brand identity, customer experience, infrastructure costs, and digital experience.

With its reliance on revenue-generating web applications, the ecommerce sector has been most affected by high-risk bot traffic. Although some bots are beneficial to business, web scraper bots are being used for competitive intelligence and espionage, inventory hoarding, imposter site creation, and other schemes that have a negative impact on both the bottom line and the customer experience.

There are no existing laws that prohibit the use of scraper bots, and they are hard to detect due to the rise of AI botnets, but there are some things companies can do to mitigate them.

Unfortunately, we often hear about consumers who have fallen victim to phishing scams. In this case, scraper bots may have been used to grab product images, descriptions, and pricing information to create counterfeit storefronts or phishing sites aimed at stealing credentials or credit card information. These phishing/counterfeit sites are a form of brand impersonation, in which the intellectual property of victim organizations is being used to establish trust with potential customers.

“Bots continue to present massive challenges resulting in multiple pain points for app and API owners,” said Patrick Sullivan, CTO, Security Strategy at Akamai. “This includes scraping that can steal web data and produce brand impersonation sites. The scraper landscape is also changing due to advancements like headless browser technology, requiring organizations to take an approach to managing this type of bot activity that is more sophisticated than other JavaScript-based mitigations.”

Web scraping side effects

AI botnets have the ability to discover and scrape unstructured data and content that is in a less consistent format or location. Additionally, they can use actual business intelligence to enhance the decision-making process through collecting, extracting, and then processing data.

Scraper bots can be leveraged to generate more sophisticated phishing campaigns by grabbing product images, descriptions, and pricing information to create counterfeit storefronts or phishing sites aimed at stealing credentials or credit card information.

Bots can be used to facilitate new account opening abuse — which, according to recent research, composes up to 50% of fraud losses.

Technical impacts that organizations face as a result of being scraped, whether the scraping was done with malicious or beneficial intentions, include website performance degradation, site metric pollution, compromised credentials attacks from phishing sites, increased compute costs, and more.

Regardless of the intent of the web scraping, organizations have to deal with expenses from its side effects. Some companies pay for beneficial scraping services, but the companies that are being scraped are incurring costs of their
own.