Web crawling, also known as web scraping, has become an essential tool for companies of all stripes to collect valuable external data at scale. By 2024, the global web scraping services market is projected to reach $10.4 billion, growing at a CAGR of 13.1% ^1. From retail giants optimizing prices to hedge funds seeking alpha in alternative data, businesses are eagerly adopting web crawling.
However, the legal landscape surrounding this practice remains complex and unsettled. As we‘ll explore, while crawling publicly accessible data tends to be legal, various factors like the scraping methods used, intended data use, applicable laws and jurisdiction all affect legality. Let‘s untangle this web and see where things stand in 2024.
Web Crawling in the Business World
It‘s hard to overstate how critical web crawling has become for staying competitive in the digital economy. A few key use cases illustrate its transformative benefits:
E-commerce intelligence: Retailers like Amazon and Walmart constantly crawl each others‘ sites, using pricing and product data to inform dynamic pricing, marketing strategy, and more. 71% of senior e-commerce decision-makers are using web scraping for competitor monitoring and analysis ^2.
Investment insights: Hedge funds and banks leverage web crawling to aggregate and analyze huge volumes of alternative data (e.g. SEC filings, social media, job listings) to generate investment insights. The alternative data market is expected to grow from $1.72 billion in 2021 to $69.36 billion by 2028 ^3.
AI/ML training: Tech giants like Google, Facebook, and OpenAI use web crawling to amass enormous datasets needed to train state-of-the-art AI models for applications like search, content moderation, text/image generation, and more.
The Tangled Legal Web
Despite its ubiquity and value, web crawling is fraught with legal ambiguity. There‘s still no definitive answer to "is web crawling legal?". The short answer is "it depends" – on what exactly is being scraped, how, and why. Generally speaking, crawling public data in a way that doesn‘t harm the host website or infringe on copyrights or privacy tends to be kosher, but many nuances exist.
Several landmark lawsuits have helped define the contours of web crawling legality:
hiQ Labs v. LinkedIn (2022): The U.S. Supreme Court let stand a lower court ruling that scraping data from public LinkedIn profiles likely does not violate the Computer Fraud and Abuse Act‘s prohibition on unauthorized computer access ^4. However, this case didn‘t resolve questions around websites‘ rights to restrict crawling in their terms of service.
eBay v. Bidder‘s Edge (2000): In this early case, a court held that the auction aggregator Bidder‘s Edge had likely committed trespass to chattels by continuing to crawl eBay‘s site without permission. But this pre-dated the rise of crawling and some argue it‘s of limited precedential value today ^5.
Facebook v. Power Ventures (2020): Here the U.S. Ninth Circuit Court of Appeals held that Power Ventures, a now-defunct social media aggregator startup, had violated the CFAA by continuing to scrape Facebook user data after receiving a cease-and-desist letter. The court also ruled that violating a website‘s terms of service, without more, doesn‘t automatically trigger CFAA liability ^6.
These cases demonstrate the critical role that the specifics of each situation play in determining crawling legality. Let‘s break down some of the key legal considerations:
Copyright Infringement
Although raw facts and data aren‘t copyrightable, websites‘ unique selection and arrangement of that information may be. In the U.S., such compilations of data are protected if they exhibit a "minimal degree of creativity" ^7. The EU‘s Database Directive extends even stronger protections to databases created with "substantial investment." So, scraping a site‘s database without permission and reproducing a substantial portion of it could constitute infringement. However, limited, transformative data use for research and analytics may be shielded by fair use principles.
Breach of Contract
Virtually all websites have terms of service (ToS) that limit how they can be used. While violating ToS alone may not trigger liability under laws like the CFAA, as noted in Power Ventures, it could still lead to a breach of contract claim. The enforceability of ToS in web scraping cases remains unsettled, but courts are more likely to side against scrapers who overtly violate a site‘s prohibitions after clear notice.
Data Privacy Laws
The growing wave of data privacy laws like the EU‘s General Data Protection Regulation (GDPR) and the California Consumer Privacy Act (CCPA) add another layer of legal complexity to web scraping. These laws tightly restrict the collection and processing of personal data, with costly penalties for non-compliance. Scraping any data from EU or California residents that could identify them – names, emails, account details, photos, etc. – likely requires affirmative consent. Only 11% of companies doing web crawling say they‘re completely GDPR compliant ^8.
Trespass to Chattels and CFAA
Website owners have asserted trespass to chattels claims by arguing that unauthorized scraping interferes with and diminishes the quality of their services. And the CFAA, intended as an anti-hacking law, has been weaponized against scraping, with some courts finding CFAA violations when scrapers circumvent technical blocking measures. However, as evidenced in hiQ v. LinkedIn, overly broad interpretations of these doctrines in the web scraping context have been reined in.
Walking the Legal Line
So how can companies reap the rewards of web crawling while minimizing their legal exposure? Here are some expert tips:
Read the robots.txt: This file tells crawlers which site pages are fair game. Disobeying it isn‘t necessarily illegal but can weaken your case if challenged.
Check the ToS: If a site expressly prohibits scraping, consider seeking written approval or limiting your crawling to mission-critical data only.
Don‘t be a bandwidth bandit: Aggressive crawling that strains sites‘ technical resources invites trouble. Throttle your bots and consider paid APIs for cleaner, owner-sanctioned data feeds.
Play nice with privacy: Avoid scraping personal data without consent or at least de-identify and safeguard what you collect. Most privacy laws have long arms.
Focus on facts: Stick to scraping discrete factual data vs. large, potentially copyrightable content chunks. Use the data to drive analysis and insights, not to simply reproduce it.
Get a legal gut check: Work with counsel to assess the specific risks of your web crawling program. Regulatory regimes and case law remain fluid, so vigilance and adaptation are vital.
The Road Ahead
Web crawling‘s legality in 2024 remains a tangled, context-dependent thicket. From a practical standpoint, an equilibrium of sorts has emerged: crawling public data respectfully for analytics and research is mostly tolerated, while aggressive, invasive, or infringing scraping is punished. But there are many shades of gray in between.
The trend of court rulings and regulations points to an overdue reconciliation between the immense value web crawling creates and the real risks it can pose to privacy, intellectual property, and internet integrity. With deliberate practices and due diligence, companies can untangle the web and harness this power for good. One expert put it well: "The most successful web scraping projects find ways to create value for everyone involved – the data users, the website owners, and the public. It‘s not just about what‘s technically possible or even legally defensible, but what‘s ethical and sustainable in the long run." ^9
As the law works to catch up with the lightning pace of web scraping innovation, adopting such a mindset will be the smartest strategy. Not just to avoid lawsuits today, but to help chart a course toward an internet that balances open data with respect for intellectual property, privacy, and prosperity for all.