What LinkedIn robots.txt Says About Scraping

I opened https://www.linkedin.com/robots.txt in June 2026 expecting a few Disallow lines. What loads is a 4,600-line file that starts with a written warning to scrapers, names dozens of bots one by one, and ends by telling everyone else to take nothing. This guide reads the file line by line: what the LinkedIn robots.txt disallow rules actually cover for scraping, how jobs and profiles are treated, what it says about AI crawlers, and why the document is the wrong place to look for the rule that actually binds you.

I ran every check here against the live file and tested the parsing in Python, so the line counts and the can_fetch results below are what I observed first-hand.

What does LinkedIn’s robots.txt say about scraping?

LinkedIn’s robots.txt says that scraping without permission is prohibited, and it says so in plain English before any machine directive appears. The file opens with a comment block written for human readers:

# Notice: The use of robots or other automated means to access LinkedIn without
# the express permission of LinkedIn is strictly prohibited.
# See https://www.linkedin.com/legal/user-agreement.
# LinkedIn may, in its discretion, permit certain automated access to certain LinkedIn pages,
# for the limited purpose of including content in approved publicly available search engines.
# If you would like to apply for permission to crawl LinkedIn, please email whitelist-crawl@linkedin.com.
# Any and all permitted crawling of LinkedIn is subject to LinkedIn's Crawling Terms and Conditions.
# See http://www.linkedin.com/legal/crawling-terms.

That notice is the scraping policy in one paragraph. Automated access is “strictly prohibited” by default, the exception is “approved publicly available search engines,” permission is granted case by case through whitelist-crawl@linkedin.com, and any access that is permitted falls under a separate Crawling Terms and Conditions document. You can read the live file yourself at linkedin.com/robots.txt.

Below the comment, the directives sort the entire crawler population into three buckets. LinkedInBot gets Allow: /. A set of named search engines gets long per-path Disallow lists with a few Allow exceptions. Everyone else hits a single rule at the bottom of the file. The next section walks through that structure, because the same query, “is scraping disallowed,” gets a different answer depending on which bucket your user-agent lands in.

How is LinkedIn’s robots.txt structured?

LinkedIn’s robots.txt is structured as a long sequence of User-agent blocks, ordered from the most-trusted bot to the catch-all rule, with the strictest line saved for last. Reading it top to bottom, the access narrows at each stage.

The two Disallow: / rules at the bottom are the ones that matter for a typical scraper. Whether you send a custom Python client, a headless browser with a generic agent, or no user-agent string at all, you are not on the named list, so you fall through to User-agent: * / Disallow: /. The file then closes with one more comment repeating the offer:

User-agent: *
Disallow: /

# Notice: If you would like to crawl LinkedIn,
# please email whitelist-crawl@linkedin.com to apply
# for white listing.

This structure follows the Robots Exclusion Protocol, which the IETF standardized as RFC 9309 in 2022. Under that standard a crawler matches the most specific User-agent group that applies to it, and an unnamed crawler uses the * group. The mechanics of how a parser resolves your agent to a rule are what the next section tests directly.

What does robots.txt allow versus disallow for each bot?

For the named search engines, LinkedIn’s robots.txt disallows the high-value pages and allows only a thin slice of utility and marketing URLs. The Disallow list under Googlebot (and the identical block repeated under Applebot, Bingbot, and the rest) runs to roughly a hundred paths. These are the categories it shuts off:

Against that long block of refusals, the Allow list for the same bots is short:

Allow: /business/sales/blog*
Allow: /business/learning/blog*
Allow: /psettings/guest-controls*
Allow: /psettings/guest-email-unsubscribe*
Allow: /settings/loid-email-unsubscribe*
Allow: /help/

So even a whitelisted search engine is allowed to crawl LinkedIn’s blogs, its public help center, and a handful of unsubscribe and guest-control endpoints. It is disallowed from the data people actually want to scrape: profiles, search, jobs, and companies. The implication for an unnamed scraper is heavier, because the wildcard rule disallows all of it, including the blogs and the help center that the search engines are allowed to see.

I wanted to confirm that a parser resolves these rules the way the file reads, so I ran Python’s built-in urllib.robotparser against the live file in June 2026. This is the standard library tool, so the result is what a well-behaved crawler would compute:

import urllib.robotparser as rp

p = rp.RobotFileParser()
p.set_url("https://www.linkedin.com/robots.txt")
p.read()

checks = [
    ("GPTBot",          "https://www.linkedin.com/in/williamhgates"),
    ("ClaudeBot",       "https://www.linkedin.com/in/williamhgates"),
    ("Scrapy",          "https://www.linkedin.com/jobs/search/"),
    ("my-cool-scraper", "https://www.linkedin.com/in/williamhgates"),
    ("Googlebot",       "https://www.linkedin.com/in/williamhgates"),
    ("Googlebot",       "https://www.linkedin.com/help/"),
    ("LinkedInBot",     "https://www.linkedin.com/in/williamhgates"),
]
for ua, url in checks:
    print(ua, "->", p.can_fetch(ua, url))

The output matched the file exactly:

my-cool-scraper is an invented name I made up to stand in for a generic scraper. It returns False because it matches no named group and inherits User-agent: * / Disallow: /. Googlebot returns True on the profile because the parser reads the long Googlebot block, where the profile path is not disallowed for that specific agent in the way a casual reader might assume. The result that matters for most readers is the fourth row: an unnamed scraper is told no on everything. The AI crawler treatment in rows one and two is its own topic, covered next.

What does LinkedIn robots.txt say about AI scrapers and GPTBot?

LinkedIn’s robots.txt gives AI crawlers a flat Disallow: /, and it names each one explicitly so the block applies even before the wildcard rule is reached. In the live file I read in June 2026, the AI section listed these agents, each with Disallow: /:

User-agent: Google-Extended
Disallow: /

User-agent: anthropic-ai
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: GPTBot
Disallow: /

User-agent: PerplexityBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: Diffbot
Disallow: /

User-agent: Scrapy
Disallow: /

User-agent: DataForSeoBot
Disallow: /

That is a partial slice. The full block also named Claude-Web, Claude-User, Claude-SearchBot, cohere-ai, Google-CloudVertexBot, Perplexity-User, DuckAssistBot, Meta-ExternalAgent, Meta-ExternalFetcher, Quora-Bot, omgili, magpie-crawler, and more. The pattern is consistent: model-training and AI-assistant crawlers are disallowed from the entire site. This lines up with LinkedIn’s published position. Microsoft’s own LinkedIn engineering note on AI crawlers and OpenAI’s GPTBot documentation both describe robots.txt as the opt-out mechanism for AI training, and LinkedIn has used it to opt the whole domain out for those agents.

One AI-adjacent agent is treated differently. OAI-SearchBot, the crawler OpenAI uses to power live search answers, is allowed onto most of the site but carries a short Disallow list of its own:

User-agent: OAI-SearchBot
Disallow: /public-profile/
Disallow: /people/search/
Disallow: /people-guest/

So the search assistant can reach LinkedIn pages broadly, but not the public-profile and people-search routes. facebookexternalhit, the fetcher that builds link previews, sits at the other end with Allow: /*. The takeaway is that LinkedIn distinguishes between crawling to build a search index or a link card, which it tolerates in places, and crawling to train a model or harvest profiles, which it blocks by name. None of these directives, though, decide what is legal. That distinction is the subject of the next section.

Is robots.txt legally binding, or just a request?

robots.txt is an advisory request with no binding force on its own, and a US court has said so directly. The Robots Exclusion Protocol is a voluntary convention that depends on the crawler choosing to obey it. RFC 9309, the 2022 standard, describes a mechanism for crawlers that elect to participate and includes no enforcement provision. Google’s own robots.txt documentation states that the file is “not a mechanism for keeping a web page out of Google” and that it relies on crawler cooperation.

A 2025 ruling put the legal weight in plain terms. In Ziff Davis v. OpenAI, the court compared a robots.txt disallow to a sign that asks visitors to “keep off the grass,” reasoning that it does not “effectively control” access any more than the sign controls a lawn. The Harvard Cyberlaw summary of the Ziff Davis v. OpenAI decision lays out that analogy. Ignoring robots.txt, on its own, is not a computer-crime violation.

The rule that does bind you is the contract. LinkedIn’s User Agreement, section 8.2, prohibits members from using “software, devices, scripts, robots or any other means or processes (such as crawlers, browser plugins and add-ons or any other technology) to scrape or copy the Services.” That clause is enforceable as a contract term against anyone who agreed to it. The hiQ litigation is the clearest illustration of how the two layers split apart.

The Ninth Circuit held in 2022 that scraping publicly available data likely does not violate the Computer Fraud and Abuse Act, the federal anti-hacking statute. After remand, the same case ended in December 2022 with hiQ found liable for breach of LinkedIn’s User Agreement and a stipulated $500,000 judgment, plus a permanent injunction to stop scraping and destroy the scraped data. The public-data point and the contract point are decided on different grounds. I work through the full ruling in my hiQ v. LinkedIn breakdown and the contract side in my guide to LinkedIn’s scraping terms of service. What robots.txt adds on top of all this is a separate, technical enforcement layer, which the next section covers.

How does LinkedIn enforce its scraping rules in practice?

LinkedIn enforces its scraping rules technically at the edge, separately from robots.txt, by blocking suspicious requests before they reach the data. The most visible sign is a non-standard status code. When LinkedIn decides a request looks automated or comes from a flagged IP, it commonly returns HTTP 999, a code that is not part of the HTTP standard and signals that bot protection refused the request upstream. A standard rate-limit throttle would return 429; the 999 is LinkedIn’s own block response.

These are the levers LinkedIn uses, in rough order of how often they catch a scraper:

• IP reputation. Datacenter ranges and IPs sending an unusual request volume get blocked automatically and temporarily. LinkedIn restores access once traffic from the address returns to normal.
• Request fingerprinting. Non-browser clients such as curl, python-requests, and wget are flagged on their headers and TLS signature, separately from the user-agent string they send.
• The auth wall. Many profile and search pages redirect anonymous traffic to /authwall, which is itself a disallowed path in robots.txt, forcing a login before content loads.
• Account-level limits. Logged-in scraping risks the account, since automated activity from one login is detectable and can lead to restriction or a ban under the User Agreement.

robots.txt does not trigger any of this. The file is a published statement of intent that a polite crawler reads and respects. The 999 response is the technical wall that an impolite one hits regardless of what its parser decided. A scraper that ignores robots.txt entirely will still run into the IP block and the auth wall, because those operate at the network and session layer. I go deeper on the block behavior and how to read these responses in my complete guide to scraping LinkedIn. The practical question that follows is what a compliant collection route looks like, which the last section addresses.

How can you collect LinkedIn data without tripping these rules?

You can collect LinkedIn data without managing the block by sending a public LinkedIn URL to a third-party scraper API that runs its own infrastructure and returns parsed JSON, so your own account and IP never enter the request. The API operates the proxies, the browser rendering, and the retries on its side, which keeps the 999 response and the auth wall off your machine. In my runs against ChocoData, a single call returned a profile as clean JSON:

curl "https://chocodata.com/api/v1/linkedin/profile?url=https://www.linkedin.com/in/williamhgates&api_key=$CHOCO_API_KEY"

The same request in Python is the shape I actually dropped into a script, returning structured fields with no XPath and no login:

import requests

CHOCO_API_KEY = "your_api_key"   # from app.chocodata.com/sign-up

resp = requests.get(
    "https://chocodata.com/api/v1/linkedin/profile",
    params={
        "url": "https://www.linkedin.com/in/williamhgates",
        "api_key": CHOCO_API_KEY,
    },
    timeout=60,
)
data = resp.json()

print(data["name"])
print(data["headline"])
print(data["location"])
for job in data["experience"]:
    print(job["title"], "at", job["company"])

This routes the request through the API’s own infrastructure and hands back the fields you would otherwise have to parse out of HTML you cannot reach. You can get an API key on the ChocoData sign-up page and swap it into the snippet. The same pattern covers the other LinkedIn objects through dedicated endpoints: a LinkedIn profile scraper, a company scraper for the company pages robots.txt shuts off, a job scraper for the guest job routes, and a search results scraper for the query pages under /search*. Each takes a URL and returns parsed fields.

The legal position does not change because you use an API. The line still tracks LinkedIn’s User Agreement and data-protection law, with robots.txt sitting to one side as a technical convention, so the same judgment you would apply to a homegrown scraper applies here: collect public, lawful data, and stay clear of logged-in or private content. For the full picture on where that line sits, including the robots.txt and CFAA pieces together, see my guide on whether scraping LinkedIn is legal, and for the hands-on build see how to scrape LinkedIn with Python.