Web Scraping Legality in 2026: What Every Developer Should Know Before Writing a Single Line

Most developers who ask "is web scraping legal?" expect a simple yes or no. The actual answer is closer to "it depends on about six different things simultaneously." That is frustrating to hear, but ignoring the nuance is how teams end up with cease-and-desist letters or GDPR fines buried inside a sprint review.

This guide covers the legal terrain as it actually stands in 2026. Court decisions, privacy law changes, AI training complications, robots.txt misconceptions, and where monitoring fits in when scraping carries too much risk.

Why This Got More Complicated

A few years ago the rough consensus was: public data equals fair game, private data behind a login equals trouble. That framing still holds as a starting point, but several things have happened since.

The EU AI Act enforcement kicked in across 2025 and 2026, adding machine-readable opt-out requirements for AI training use cases. Reddit v. Perplexity AI introduced DMCA Section 1201 into the conversation, arguing that rate limits and anti-bot systems are "technical protection measures." India's DPDP Act entered real enforcement. France's CNIL explicitly ruled that public pages can still contain personal data requiring GDPR safeguards. The legal surface area expanded.

None of this makes scraping illegal. But it does mean the old "it is public, so I am fine" shortcut no longer covers all your bases.

The Core Legal Frameworks

There is no single scraping law. What you are actually navigating is an intersection of several frameworks, each applying differently depending on what you scrape, where, and what you do with the data.

Computer Fraud and Abuse Act (CFAA)

The CFAA is the US federal law most often cited in scraping disputes. It criminalizes accessing a computer "without authorization" or in a way that "exceeds authorized access." For years, platforms tried to argue that violating their Terms of Service revoked your authorization, meaning any scraping after a ToS ban was a federal crime.

Courts largely rejected that interpretation. The landmark case here is hiQ Labs v. LinkedIn.

hiQ Labs v. LinkedIn (9th Cir. 2019, reaffirmed 2022)

LinkedIn tried to block hiQ, a workforce analytics company, from scraping public profile data. The Ninth Circuit ruled that accessing publicly available information does not constitute "unauthorized access" under the CFAA. LinkedIn lost on the CFAA claim. The ruling was appealed, survived a Supreme Court remand, and the core holding has been consistently reaffirmed.

The practical takeaway: accessing genuinely public data, with no login and no circumvention of technical barriers, is not a CFAA violation in the Ninth Circuit's view.

Meta v. Bright Data (N.D. Cal. 2024)

Meta sued Bright Data over Facebook and Instagram scraping. The court found that Meta's Terms of Service only bind users who are logged in and using the platform. When Bright Data scraped public, logged-out pages, there was no enforceable contract governing that conduct. Meta dropped the case.

This reinforced hiQ in a new context: browsewrap terms do not reach logged-off public scraping.

Where the CFAA still bites: If you create fake accounts, bypass login walls, circumvent CAPTCHAs to access non-public content, or ignore an explicit IP block and route around it, you are in materially different legal territory. The CFAA analysis changes the moment technical barriers are involved.

Terms of Service

Nearly every major website prohibits automated scraping in its ToS. The question is how enforceable those terms are against someone who never explicitly agreed to them.

Courts have distinguished two types of agreements:

If you have never created an account, never clicked "I agree," and are accessing public content, ToS enforcement is limited. If you created an account, agreed to terms that prohibited scraping, and then scraped anyway, you have a breach of contract problem even if the CFAA does not apply.

The risk calculus: ToS violations are civil, not criminal. But they create leverage. A company can use a ToS breach as supporting evidence in trespass claims, unfair competition arguments, or as grounds to seek a preliminary injunction. The litigation cost alone can be significant regardless of outcome.

Copyright

When you scrape a page, you are technically copying its content. If that content is original creative work, you may be reproducing something protected by copyright.

The nuances that matter in practice:

• Facts and data are not copyrightable. Prices, temperatures, stock values, product names: not protected.
• Creative expression is protected. An editorial review, a photo, a curated article: the author's original expression is protected.
• Database rights exist in the EU under the Database Directive. Even if individual data points are not copyrightable, a database that represents substantial investment in gathering or organizing data can be protected. Extracting a substantial portion of such a database raises infringement risk.
• Fair use applies in the US. Transformative use, research, analysis, and commentary can provide a defense. But fair use is a defense you raise in court, not a permission slip you grant yourself in advance.

For AI training specifically: the EU AI Act now requires transparency about training data sources and must honor machine-readable opt-outs. In the US, fair use is still being litigated in several active cases, including Reddit's suits against AI companies for using its data without a licensing agreement.

Privacy Law: GDPR and CCPA

This is where public data gets complicated.

GDPR (EU)

The most common misconception about GDPR is that it only applies to private data. It does not. GDPR applies to all personal data, defined broadly as any information relating to an identifiable person. A publicly visible LinkedIn profile, an Instagram post with a username, a forum comment attributed to a real name: all personal data under GDPR.

If you scrape personal data about EU residents, you need a lawful basis. The two realistic options are:

• Consent: Impractical when scraping at scale from sources you do not control.
• Legitimate interest: Possible, but requires a formal balancing test weighing your purpose against the individual's rights. It is not a catch-all.

The CNIL in France clarified in 2025 that even public pages require GDPR compliance when they contain personal data. Clearview AI's experience is instructive: scraping billions of public photos resulted in over 91 million euros in fines across EU jurisdictions by 2025, not because the photos were private but because the processing lacked a lawful basis.

CCPA (California)

CCPA gives California residents the right to know what personal data businesses collect and to request its deletion. If you are scraping personal data about California residents and building products around it, CCPA obligations apply to how you handle and expose that data.

robots.txt: Common Myths vs Reality

robots.txt is probably the most misunderstood piece of web infrastructure in any scraping conversation.

The practical position: respect robots.txt. Not because you are legally required to, but because it signals intent and good faith if a dispute arises, and because it is a reasonable convention that keeps relationships with site operators workable. Courts have occasionally treated Disallow rules as notice that access is unwelcome, which can factor into trespass to chattels arguments.

The Public vs. Private Data Line

This is the clearest boundary in scraping law, but it has edge cases.

Generally lower risk:

• Pricing data displayed publicly without login
• Public business listings
• Government datasets and open data portals
• Public job listings
• Open API responses

Generally higher risk:

• Any data behind authentication
• Personal data about individuals, even if technically public
• Content that required bypassing a paywall or CAPTCHA
• Content where the site has explicitly deployed technical barriers

The key distinction the courts keep returning to is whether you circumvented technical access controls. Accessing public content is reading. Bypassing a login, manipulating rate limit logic, or using credentials you are not authorized to use looks more like unauthorized access.

Legal vs. Illegal Scraping: Scenario Reference

Developer Best Practices for Legal Compliance

Most scraping legal risk is not eliminated by one decision. It is reduced by a pattern of decisions made consistently across a project.

Before you start:

• Identify what type of data you are collecting and whether it includes personal information.
• Check whether the target site has an official API. If one exists, use it. It is lower risk and more stable.
• Read the ToS. Even if browsewrap terms are weakly enforceable, they tell you the site's posture and whether you have been explicitly warned.
• Establish a documented business purpose. Courts and regulators look at purpose and proportionality.

During operation:

• Respect robots.txt Disallow rules. Flag any Crawl-delay directives and honor them.
• Identify your scraper honestly in the User-Agent string. Something like YourCompanyBot/1.0; +https://yourcompany.com/botinfo is reasonable practice.
• Throttle your requests. Aggressive scraping that degrades site performance opens trespass to chattels claims, which do not require circumventing any access controls.
• Log your data sources. Traceability is increasingly required under EU AI Act compliance for training use cases, and it is good practice in general.
• Strip personal data at the extraction layer if your use case does not require it.

Data handling:

• Do not store personal data of EU or California residents without a documented lawful basis.
• Establish a retention policy. Holding scraped personal data indefinitely is harder to defend.
• If someone requests deletion of their data, have a process to honor it.

Developer Risk Mitigation Checklist

• Target data is public and accessible without authentication
• No personal data collected, or a documented lawful basis exists
• robots.txt has been read and Disallow rules are respected
• Request rate is throttled, not crawling at maximum speed
• User-Agent string identifies the bot honestly
• ToS has been reviewed; no clickwrap agreement prohibiting scraping was accepted
• Data is not being stored or redistributed in a way that triggers copyright issues
• If targeting EU users' data, GDPR compliance plan documented
• If AI training use case, machine-readable opt-outs are honored
• Logs track which sources were accessed and when

When Monitoring Makes More Sense Than Scraping

A significant subset of scraping use cases are actually monitoring problems in disguise.

Competitive pricing intelligence, dependency release tracking, inventory restock alerts, regulatory filing detection, API contract drift: none of these require downloading and storing raw HTML at scale. They require knowing when a specific value changes.

Scraping to satisfy this kind of use case introduces unnecessary surface area. You are maintaining parsers, managing retry logic, storing diffs, and building alert logic, all while sitting under the legal and operational overhead of a scraping project.

This is the problem Verid is built around. Instead of returning raw HTML for you to parse and monitor yourself, Verid runs a complete extraction and monitoring loop: it fetches the page, extracts a specific field using CSS selectors, XPath, JSONPath, regex, or an AI prompt, compares it against the last run, evaluates a predicate you define (did the price drop 10%? did stock status change?), and fires a signed webhook only when that predicate is true. You define what matters; it handles everything else.

For use cases like competitor price tracking, dependency monitoring, or regulatory filing detection, this approach is legally cleaner too. You are querying for a specific value on a schedule, not bulk-extracting entire sites. The scope is defined and proportionate.

Here is a basic example of creating a monitor with Verid's REST API to watch a public product page for price changes:

// Create a Verid monitor that fires only when price drops 10% or more
// Docs: https://docs.verid.dev/quickstart

const response = await fetch("https://api.verid.dev/v1/monitors", {
  method: "POST",
  headers: {
    "Authorization": "Bearer vrd_your_api_key",
    "Content-Type": "application/json"
  },
  body: JSON.stringify({
    name: "Competitor product price",
    url: "https://example-store.com/products/widget-pro",
    schedule_interval_seconds: 3600, // Check every hour

    // Extract the price field using a CSS selector
    extract_config: {
      method: "css",
      fields: {
        price: ".product-price"
      }
    },

    // Only fire the webhook when price decreases by 10% or more
    diff_predicate: {
      type: "field_decreases_by_percent",
      field: "price",
      threshold: 10
    },

    // Where to send the alert
    deliveries: [
      {
        type: "webhook",
        url: "https://your-app.com/hooks/price-alert"
      }
    ]
  })
});

const monitor = await response.json();
console.log("Monitor created:", monitor.id);

When the predicate fires, Verid sends a signed POST to your endpoint. The payload includes the before and after values so you know exactly what changed:

{
  "monitor": "Competitor product price",
  "fired": "field_decreases_by_percent",
  "field": "price",
  "before": "149.00",
  "after": "119.00",
  "at": "2026-06-28T14:22:00Z"
}

Verify the X-Verid-Signature header on your endpoint before trusting the payload. The webhook documentation covers the HMAC verification steps.

For pages where the markup keeps shifting, you can switch the extraction method to AI without redeploying:

{
  "method": "llm",
  "fields": {
    "price": "the current sale price in USD"
  }
}

This is especially useful for dynamic pages where class names change frequently a common maintenance headache with traditional scrapers.

Key Takeaways

• Scraping public data is not inherently illegal, but it is not automatically legal either. Six separate legal frameworks apply depending on your context.
• The CFAA risk is largely settled for public data cases, thanks to hiQ and Bright Data. The risk increases sharply the moment technical access controls are involved.
• GDPR applies to publicly visible personal data. The public/private distinction does not map onto the GDPR personal/non-personal distinction.
• robots.txt is a convention, not a law. Respect it anyway.
• ToS enforceability against scrapers depends heavily on whether you agreed to them. Browsewrap terms are weak. Clickwrap terms are enforceable.
• For many use cases, targeted monitoring is more defensible and less operationally complex than broad scraping.
• Document your purpose, scope, and data handling. Compliance is easier to demonstrate when you have made deliberate decisions rather than ad hoc ones.

This post is for informational purposes and does not constitute legal advice. If you are building a production data pipeline with meaningful legal exposure, talk to a lawyer who knows technology and data law in your jurisdiction.

Frequently Asked Questions

Is it legal to scrape public data without using the site's API?

Generally yes in the US, provided the data is genuinely public, you are not circumventing any access controls, and the data does not include personal information about identifiable individuals. hiQ Labs v. LinkedIn and Meta v. Bright Data both affirmed that accessing public data does not violate the CFAA. However, the site's ToS may still create civil exposure, and privacy law applies separately if personal data is involved.

Does ignoring robots.txt make scraping illegal?

No. robots.txt is a technical convention, not a law. Ignoring it does not by itself violate the CFAA or constitute copyright infringement. Some courts have treated it as evidence of notice, which can factor into trespass to chattels claims, but it does not independently create criminal or civil liability. That said, respecting it is sensible practice.

Does GDPR apply if I am scraping publicly visible data?

Yes, when that data relates to identifiable EU residents. GDPR does not distinguish between data people share privately and data that appears on a public page. If you are scraping names, email addresses, usernames, or any other information that identifies a person, GDPR obligations apply regardless of whether that information is publicly visible. You need a lawful basis to collect and process it.

What is the difference between web scraping and website monitoring, legally speaking?

The legal distinction is not always about the label. What matters is scope, intent, and method. Bulk extraction of large datasets is treated differently than targeted polling of a specific value on a schedule. Monitoring a specific field for changes, especially via a structured API or service like Verid, involves narrower scope and is easier to justify as proportionate to the business purpose. It also sidesteps the personal data accumulation risk that comes with storing large volumes of raw scraped content.