Unlocking internet’s secrets via monitoring, data collection, and analysis

In this Help Net Security interview, Ryan Woodley, CEO of Netcraft, discusses the importance of monitoring, collecting, and analyzing internet data to gain a profound understanding of the internet. This insight plays a vital role in protecting and empowering customers.

Netcraft has been monitoring the internet since 1995 and knows various industry aspects. How do you collect and analyze data on web servers, operating systems, hosting providers, and other related areas?

Netcraft has been mapping the internet’s evolution since its inception in 1995. Our methodology includes running comprehensive monthly internet surveys to visit and inspect as many new and existing websites as possible. The aim is to generate a vast, rich pool of data, which is processed using advanced algorithms and data enrichment techniques. The results can be manipulated and viewed through many different lenses.

Our results offer a multi-faceted view of the internet’s landscape, categorized under several key parameters such as web servers, operating systems, hosting providers, and more. We’ve also channeled this data into our cybercrime detection and disruption service, which launched in 2005. The data and insights from both complement each other and provide a deep understanding of the internet.

Combined with our experience, the data drives our deep knowledge of the internet, both its visible and hidden aspects, to protect and empower our customers. This dual approach provides a holistic, robust, and dynamic perspective on the internet, making it a potent resource for businesses, cybersecurity experts, and researchers.

Netcraft’s data shows many sites, domains, and web-facing computers. How does Netcraft ensure the accuracy and reliability of its data collection and analysis processes? Are there any potential limitations or challenges in collecting such extensive data?

Our primary focus at Netcraft is reporting on the internet as-deployed, and, for example, in our Secure Sockets Layer (SSL) and Transport Layer Security (TLS) data, we report on the certificate used when connecting to a website even when there are many overlapping issued certificates visible in certificate transparency logs.

Equally, we strive to ensure our cybercrime data is as complete as possible by sourcing reports through our anti-cybercrime community, our own searches, and partner data feeds. For example, by consuming zone files from top TLDs and gaining access to potentially malicious content from a geo-distributed fetch network to bypass an attacker’s attempt to evade detection.

Our approach has been to automate this detection process as much as possible, allowing it to run at scale, providing round-the-clock vigilance. There are inherent challenges in handling such an enormous scale of data, but our sophisticated systems and procedures ensure high accuracy and reliability.

What notable emerging trends or technologies in the web server industry did you observe during your recent survey? How do these trends impact the overall landscape, and what can we expect in the near future?

Across the web server industry, one key trend we’ve observed has been the use of Content Delivery Networks (CDNs), which has made a prominent mark in both our web server data and in the cybercrime world where it is a crucial enabler of both legitimate and illegitimate content.

As with many technologies, criminals are often trailblazers, adopting new products and services as they address new and existing needs. However, with a vigilant eye on these trends and patterns, we can equip our customers with the knowledge to safeguard against potential threats and better understand the changing landscape. As CDNs and similar technologies evolve, we foresee a continuing adaptation in how businesses and cybercriminals operate online.

Considering the increase in cyber threats and the importance of cybersecurity, what advice would you give CISOs wanting to keep their websites and customers safe?

In the face of consistent and ever-present cyber threats, there are a few fundamental practices that CISOs should prioritize. The basics are always essential – keeping up to date with security patches, changing the default and admin passwords, and only exposing the minimum set of intended services to the internet. However, this is just the starting point. For large organizations, there is a need for continuous monitoring of their attack surface as well as that of potential adversaries.

Large organizations and their CISOs can employ various tools and platforms, including tracking their own exposed attack surface and that of their would-be attackers’ who are seeking to impersonate the brand to exploit its customers. Both aspects can interact with each other in intriguing ways – for example, monitoring HTTP Referer information on their legitimate website can help identify cloned sites and phishing sites designed to mimic the brand. By leveraging such insights, organizations can take a more proactive and comprehensive approach to cybersecurity, staying ahead of potential threats.