Unlocking Internet Secrets Through Monitoring, Data Collection, and Analysis – Help Net Security

In this Help Net Security interview, Ryan Woodley, CEO of Netcraft, discusses the importance of monitoring, collecting and analyzing Internet data to gain a deep understanding of the Internet. This insight plays a vital role in protecting and empowering clients.

Netcraft has been monitoring the Internet since 1995 and knows various aspects of the industry. How do you collect and analyze data on web servers, operating systems, hosting providers and other related areas?

Netcraft has mapped the evolution of the Internet since its inception in 1995. Our methodology includes conducting comprehensive monthly Internet surveys to visit and inspect as many new and existing websites as possible. The goal is to generate a large and rich pool of data, which is processed using advanced algorithms and data enrichment techniques. The results can be manipulated and viewed through many different lenses.

Our findings offer a multifaceted view of the Internet landscape, ranked by several key metrics such as web servers, operating systems, hosting providers, and more. We also funneled this data into our cybercrime detection and disruption service, which launched in 2005. The data and insights from both complement each other and provide a deep understanding of the internet.

Combined with our experience, the data drives our deep understanding of the Internet, both its visible and hidden aspects, to protect and empower our customers. This dual approach provides a holistic, robust, and dynamic perspective on the Internet, making it a powerful resource for businesses, cybersecurity experts, and researchers.

Netcraft’s data shows many sites, domains and computers connected to the web. How does Netcraft ensure the accuracy and reliability of its data collection and analysis processes? Are there any potential limitations or challenges in collecting such extensive data?

Our main focus at Netcraft is reporting the internet as implemented, and for example in our Secure Sockets Layer (SSL) and Transport Layer Security (TLS) data, we report the certificate used when connecting to a website even when not There are many overlapping issued certificates visible in the certificate transparency logs.

Likewise, we strive to ensure that our cybercrime data is as complete as possible by providing reports through our anti-cybercrime community, our research and partner data feeds. For example, consuming zone files from major TLDs and gaining access to potentially malicious content from a geographically distributed scavenging network to circumvent an attacker’s attempt to evade detection.

Our approach has been to automate this discovery process as much as possible, allowing it to run at scale, providing 24/7 vigilance. There are inherent challenges in managing such a huge scale of data, but our systems and Sophisticated procedures ensure high accuracy and reliability.

What major trends or emerging technologies in the web server industry did you observe in your recent survey? How do these trends impact the overall landscape and what can we expect in the near future?

Across the web server industry, a key trend we have observed has been the use of Content Delivery Networks (CDNs), which has left a major mark on both our web server data and the world of cybercrime, where it is a crucial enabler of both legitimate and illegitimate content.

As with many technologies, criminals are often pioneers, adopting new products and services while responding to new and existing needs. However, by keeping a watchful eye on these trends and patterns, we can equip our clients with the knowledge to protect themselves from potential threats and better understand the changing landscape. As CDNs and similar technologies evolve, we expect the way businesses and cybercriminals operate online to continually adapt.

Considering the increase in cyberthreats and the importance of cybersecurity, what advice would you give to CISOs looking to protect their websites and customers?

Faced with constant and ever-present cyberthreats, there are a few key practices that CISOs should prioritize. The basics are always essential to keep up to date with security patches, change default and admin passwords, and expose only the minimum set of expected services to the Internet. However, this is just the starting point. For large organizations, you need continuous monitoring of your attack surface and that of potential adversaries.

Large organizations and their CISOs can use various tools and platforms, including monitoring their own exposed attack surface and that of potential attackers looking to impersonate their brand to exploit their customers. Both aspects can interact with each other in intriguing ways, for example, monitoring HTTP Referer information on their legitimate website can help identify cloned sites and phishing sites designed to mimic the brand. By leveraging this information, organizations can take a more proactive and comprehensive approach to cybersecurity, staying abreast of potential threats.