last posts

Threat Intelligence Using Web Scraping


Threat intelligence plays a key role in the safety and security of any organization’s online activity and is instrumental in maintaining the integrity of their internal infrastructure.

But to be able to assess potential threats in the cybersecurity landscape at scale, they need data and, more importantly, public web data.

Indeed, web data helps security operators to better understand the vulnerabilities that may be present in their systems, the threats that could come from the networks of outside organizations as well as the potential risks that could target their organization on the World Wide Web. .

Public web data in research, intelligence and security testing

In practice, this collection of public web data is used to automate checks to discover the presence of possible malware, phishing links, different forms of fraud, information leaks and counterfeiting schemes.

Essentially, it provides security operators with the visibility they need to effectively detect, respond to, and prevent real-world threats or intrusions from affecting their organizational security, by mapping potential security vulnerabilities that could target various web-based systems.

Data collection web networks

To achieve this increased visibility, security companies and operators use public web data collection networks to collect large amounts of information, or web data, which they then use to identify, monitor and assess threats in time. real.

Web-based data collection networks (including IP proxy networks) provide a risk-free environment to discover how to prevent digital risks from reaching their organization, without sacrificing the integrity of their internal infrastructure.

To do this, security operators route requests through web data collection networks to assess the risk of potentially malicious websites or URLs.

The requests then return information, or web data. This web data provides details on how the domain responded to the request, which then allows security teams to assess the threat (or lack thereof) and take appropriate action to mitigate it before it occurs. reaches their web systems.

Within applications, this method essentially provides a firewall, opening one-way access to information, taking into account how requests are routed: away from their internal systems — protecting their organization’s internal network.

Security use cases:

Scraping for possible malware and phishing targeting US banks

The security departments of some of the major US banks use public web data collection networks to gather information about possible online threat actors and examine malware.

Additionally, they use web scraping techniques to continuously and automatically scan the public domain for potentially malicious websites or links. For example, security teams can automatically identify different phishing sites that attempt to steal sensitive customer or company information, such as usernames, passwords, or credit card information.

From there, when an email arrives on the organization’s network or a website is approached, the security team already knows the risk parameters attached to it.

Web Scraping for Cybersecurity Companies

A number of cybersecurity companies use web data collection to assess the risk of different domains for malware and fraud.

They generate or buy lists of potentially malicious domains, then route DNS queries to each of those links, servers, or websites to see how they react to the query.

This gives cybersecurity companies the ability to approach potentially malicious websites as a “victim” or real user, and see how the website would target an unsuspecting visitor to properly assess the risk.

Threat Research and Mitigation

Threat intelligence firms are deploying the use of public web data collection networks to mine various sources of information, such as hacker or app forums, public social media channels, blogs, and more. , in order to identify new leads on various potential threats.

This collection of web data is essential to their intelligence information, which they then share with a wide range of clients seeking to bolster their own security operations.

Key points to remember

Overall, integration with web data collection networks improves an organization’s visibility and ability to deal with digital threats in the vast online landscape in real time.

This is particularly important given the recent changes to remote working, as well as the expansion of online operations, strategies and services all around, following the outbreak of the coronavirus, which only add to the list of risks or pathways that can disrupt organizational security.

So, as the task entrusted to security teams has become increasingly difficult, web data collection networks have essentially transformed this once complicated ordeal into a much more manageable undertaking by providing options for automation – helping them to target more sources of information, thereby identifying more risks, while protecting the integrity of their own internal systems in the grand scheme of things.

About the Author

Ou Lenchner is CEO of Bright Data, a position he has held since July 2018. Over the past few years, under his leadership, the company has evolved its product offerings to include first-of-its-kind automated solutions, enabling its more than 15,000 customers to collect and receive public data in minutes. Prior to his career at Bright Data, Lenchner founded and managed several web-based businesses, developing digital assets and online marketing programs.



Font Size
lines height