Research

Which S&P 500 companies block AI?

We checked the websites of all 500 companies to see which AI crawlers they let in, and which ones publish an llms.txt file.

Christopher LierFounder and CEO, Sentosa LimitedPublished October 2026

Key findings

Almost every large US company lets AI tools read its website. Very few have made it easy for them.

97%
of the websites we could read let ChatGPT’s crawler in: 15
companies block at least one AI crawler from their whole website: 14%
publish an llms.txt file, a short guide for AI tools: 100
websites turned away our automated check before we could read their rules

Finding 01

Companies block AI training more than AI search

Of the 400 websites we could read, 11 block GPTBot, the crawler OpenAI uses to train its models. Only 2 block OAI-SearchBot, the crawler behind ChatGPT search.

Nine companies, including Airbnb, eBay and Northrop Grumman, block training but still let ChatGPT search in. They want to appear in AI answers without handing over their content to train the next model.

Websites blocking each AI crawlerNumber of the 400 S&P 500 websites whose rules we could read that block the crawler from every page.

Who blocks AI

15 companies block at least one AI crawler

These companies tell at least one AI crawler not to read any page on their website. A cross means the crawler is blocked.

Company	ChatGPT training	ChatGPT search	Claude	Perplexity	Gemini	Common Crawl
Amazon	✕	✕	✕	✕	✕	✕
Simon Property Group	✕	✓	✕	✕	✕	✕
Ares Management	✕	✓	✕	✓	✕	✕
eBay Inc.	✕	✓	✕	✕	✓	✕
Gartner	✕	✕	✓	✓	✕	✕
Kimberly-Clark	✕	✓	✕	✓	✕	✕
Northrop Grumman	✕	✓	✕	✓	✕	✕
Prudential Financial	✕	✓	✓	✓	✕	✕
Airbnb	✕	✓	✕	✓	✓	✓
Fortinet	✓	✓	✕	✓	✓	✕
C.H. Robinson	✓	✓	✓	✓	✕	✓
Garmin	✓	✓	✓	✕	✓	✓
Mettler Toledo	✕	✓	✓	✓	✓	✓
Netflix	✓	✓	✓	✓	✓	✕
PG&E Corporation	✕	✓	✓	✓	✓	✓

Finding 02

Consumer brands guard their content, B2B companies want to be found

Four of the 15 companies that block AI crawlers sell to consumers: Amazon, Airbnb, eBay and Garmin. Their product listings, prices and reviews are valuable data that AI tools would otherwise use for free.

Only two industrial companies block any AI crawler. For most B2B companies, being read and cited by AI tools is a way to reach buyers, not a risk.

Companies blocking at least one AI crawler, by sectorCount of the 15 companies that block at least one AI crawler from their whole website.

Consumer Discretionary4

Financials2

Industrials2

Information Technology2

Consumer Staples1

Health Care1

Communication Services1

Utilities1

Real Estate1

llms.txt

70 companies publish an llms.txt file

An llms.txt file gives AI tools a short guide to a website’s most important pages. Adoption is highest in information technology and lowest in utilities and energy.

Information Technology27%20 of 75

Health Care18%11 of 60

Industrials17%14 of 83

Financials13%10 of 76

Materials13%3 of 24

Consumer Staples12%4 of 34

Consumer Discretionary11%5 of 47

Real Estate7%2 of 30

Communication Services5%1 of 19

Utilities0%0 of 31

Energy0%0 of 21

Finding 03

llms.txt does not yet mean more AI citations

Among the 88 S&P 500 industrial companies in our Industrial SEO Report, the 14 with an llms.txt file get a median of 465 AI citations. The 74 without one get 559.

None of the large AI companies has said it reads llms.txt. For now, AI citations follow the same signals as Google rankings, above all the number of websites linking to you.

Median AI citations, industrial companiesAI citations across AI platforms, by whether the company publishes an llms.txt file. Source: Ahrefs.

465median AI citations
with llms.txt (14 companies)

559median AI citations
without llms.txt (74 companies)

Finding 04

One in five websites turns automated visitors away

100 of the 500 websites refused our automated request before we could read their rules. Firewalls and bot protection services block unknown automated visitors by default.

The same protection can stop AI crawlers without the company knowing. Cloudflare, one of the largest providers, started blocking AI crawlers by default for new customers in July 2025. If your firewall turns AI tools away, your rules file does not matter.

Websites that turned away our check, by sectorShare of each sector’s websites that refused our automated request.

Consumer Discretionary28%

Real Estate27%

Information Technology25%

Energy24%

Financials22%

Industrials22%

Materials17%

Health Care13%

Communication Services11%

Utilities10%

Consumer Staples9%

Is your website open to AI?

See which AI tools can read your site and how often they cite it.

Check your website free

Full data

All S&P 500 companies

Search for a company or click a column to sort. A dash means we could not read the website’s rules.

Company	Sector	ChatGPT training	ChatGPT search	Claude	Perplexity	Gemini	Common Crawl	llms.txt
3M	Industrials	–	–	–	–	–	–	·
A. O. Smith	Industrials	✓	✓	✓	✓	✓	✓	·
Abbott Laboratories	Health Care	✓	✓	✓	✓	✓	✓	·
AbbVie	Health Care	✓	✓	✓	✓	✓	✓	·
Accenture	Information Technology	✓	✓	✓	✓	✓	✓	·
Adobe Inc.	Information Technology	✓	✓	✓	✓	✓	✓	✓
Advanced Micro Devices	Information Technology	–	–	–	–	–	–	·
AES Corporation	Utilities	✓	✓	✓	✓	✓	✓	·
Aflac	Financials	✓	✓	✓	✓	✓	✓	✓
Agilent Technologies	Health Care	✓	✓	✓	✓	✓	✓	·
Air Products	Materials	–	–	–	–	–	–	·
Airbnb	Consumer Discretionary	✕	✓	✕	✓	✓	✓	·
Akamai Technologies	Information Technology	–	–	–	–	–	–	·
Albemarle Corporation	Materials	✓	✓	✓	✓	✓	✓	✓
Alexandria Real Estate Equities	Real Estate	✓	✓	✓	✓	✓	✓	·
Align Technology	Health Care	✓	✓	✓	✓	✓	✓	·
Allegion	Industrials	✓	✓	✓	✓	✓	✓	·
Alliant Energy	Utilities	–	–	–	–	–	–	·
Allstate	Financials	–	–	–	–	–	–	·
Alphabet Inc.	Communication Services	✓	✓	✓	✓	✓	✓	·
Altria	Consumer Staples	✓	✓	✓	✓	✓	✓	·
Amazon	Consumer Discretionary	✕	✕	✕	✕	✕	✕	·
Amcor	Materials	✓	✓	✓	✓	✓	✓	✓
Ameren	Utilities	✓	✓	✓	✓	✓	✓	·
American Electric Power	Utilities	✓	✓	✓	✓	✓	✓	·
American Express	Financials	✓	✓	✓	✓	✓	✓	✓
American International Group	Financials	✓	✓	✓	✓	✓	✓	·
American Tower	Real Estate	–	–	–	–	–	–	·
American Water Works	Utilities	–	–	–	–	–	–	·
Ameriprise Financial	Financials	✓	✓	✓	✓	✓	✓	✓
Ametek	Industrials	✓	✓	✓	✓	✓	✓	·
Amgen	Health Care	✓	✓	✓	✓	✓	✓	·
Amphenol	Information Technology	–	–	–	–	–	–	·
Analog Devices	Information Technology	–	–	–	–	–	–	·
Aon plc	Financials	–	–	–	–	–	–	·
APA Corporation	Energy	✓	✓	✓	✓	✓	✓	·
Apollo Global Management	Financials	✓	✓	✓	✓	✓	✓	·
Apple Inc.	Information Technology	✓	✓	✓	✓	✓	✓	·
Applied Materials	Information Technology	–	–	–	–	–	–	·
AppLovin	Communication Services	✓	✓	✓	✓	✓	✓	·
Aptiv	Consumer Discretionary	✓	✓	✓	✓	✓	✓	·
Arch Capital Group	Financials	✓	✓	✓	✓	✓	✓	·
Archer Daniels Midland	Consumer Staples	✓	✓	✓	✓	✓	✓	✓
Ares Management	Financials	✕	✓	✕	✓	✕	✕	·
Arista Networks	Information Technology	✓	✓	✓	✓	✓	✓	·
Arthur J. Gallagher & Co.	Financials	✓	✓	✓	✓	✓	✓	·
Assurant	Financials	✓	✓	✓	✓	✓	✓	·
AT&T	Communication Services	✓	✓	✓	✓	✓	✓	✓
Atmos Energy	Utilities	✓	✓	✓	✓	✓	✓	·
Autodesk	Information Technology	–	–	–	–	–	–	·
Automatic Data Processing	Industrials	✓	✓	✓	✓	✓	✓	✓
AutoZone	Consumer Discretionary	✓	✓	✓	✓	✓	✓	✓
Avery Dennison	Materials	✓	✓	✓	✓	✓	✓	·
Axon Enterprise	Industrials	✓	✓	✓	✓	✓	✓	✓
Baker Hughes	Energy	✓	✓	✓	✓	✓	✓	·
Ball Corporation	Materials	–	–	–	–	–	–	·
Bank of America	Financials	✓	✓	✓	✓	✓	✓	·
Baxter International	Health Care	✓	✓	✓	✓	✓	✓	·
Becton Dickinson	Health Care	✓	✓	✓	✓	✓	✓	✓
Berkshire Hathaway	Financials	✓	✓	✓	✓	✓	✓	·
Best Buy	Consumer Discretionary	✓	✓	✓	✓	✓	✓	·
Bio-Techne	Health Care	–	–	–	–	–	–	·
Biogen	Health Care	✓	✓	✓	✓	✓	✓	✓
BlackRock	Financials	✓	✓	✓	✓	✓	✓	·
Blackstone Inc.	Financials	–	–	–	–	–	–	·
Block, Inc.	Financials	✓	✓	✓	✓	✓	✓	·
Bloom Energy	Industrials	✓	✓	✓	✓	✓	✓	✓
BNY Mellon	Financials	✓	✓	✓	✓	✓	✓	·
Boeing	Industrials	✓	✓	✓	✓	✓	✓	·
Booking Holdings	Consumer Discretionary	✓	✓	✓	✓	✓	✓	✓
Boston Scientific	Health Care	✓	✓	✓	✓	✓	✓	·
Bristol Myers Squibb	Health Care	✓	✓	✓	✓	✓	✓	·
Broadcom	Information Technology	✓	✓	✓	✓	✓	✓	·
Broadridge Financial Solutions	Industrials	✓	✓	✓	✓	✓	✓	✓
Brown & Brown	Financials	✓	✓	✓	✓	✓	✓	·
Brown–Forman	Consumer Staples	–	–	–	–	–	–	·
Bunge Global	Consumer Staples	✓	✓	✓	✓	✓	✓	·
BXP, Inc.	Real Estate	✓	✓	✓	✓	✓	✓	·
C.H. Robinson	Industrials	✓	✓	✓	✓	✕	✓	·
Cadence Design Systems	Information Technology	–	–	–	–	–	–	·
Camden Property Trust	Real Estate	✓	✓	✓	✓	✓	✓	·
Capital One	Financials	✓	✓	✓	✓	✓	✓	·
Cardinal Health	Health Care	–	–	–	–	–	–	·
Carnival Corporation	Consumer Discretionary	✓	✓	✓	✓	✓	✓	·
Carrier Global	Industrials	✓	✓	✓	✓	✓	✓	·
Carvana	Consumer Discretionary	✓	✓	✓	✓	✓	✓	✓
Casey's	Consumer Staples	–	–	–	–	–	–	·
Caterpillar Inc.	Industrials	–	–	–	–	–	–	·
Cboe Global Markets	Financials	✓	✓	✓	✓	✓	✓	·
CBRE Group	Real Estate	–	–	–	–	–	–	·
CDW Corporation	Information Technology	✓	✓	✓	✓	✓	✓	·
Cencora	Health Care	✓	✓	✓	✓	✓	✓	✓
Centene Corporation	Health Care	✓	✓	✓	✓	✓	✓	·
CenterPoint Energy	Utilities	✓	✓	✓	✓	✓	✓	·
CF Industries	Materials	–	–	–	–	–	–	·
Charles River Laboratories	Health Care	–	–	–	–	–	–	·
Charles Schwab Corporation	Financials	–	–	–	–	–	–	·
Charter Communications	Communication Services	–	–	–	–	–	–	·
Chevron Corporation	Energy	–	–	–	–	–	–	·
Chipotle Mexican Grill	Consumer Discretionary	✓	✓	✓	✓	✓	✓	·
Chubb Limited	Financials	✓	✓	✓	✓	✓	✓	·
Church & Dwight	Consumer Staples	✓	✓	✓	✓	✓	✓	·
Ciena	Information Technology	–	–	–	–	–	–	·
Cigna	Health Care	✓	✓	✓	✓	✓	✓	✓
Cincinnati Financial	Financials	✓	✓	✓	✓	✓	✓	·
Cintas	Industrials	✓	✓	✓	✓	✓	✓	·
Cisco	Information Technology	✓	✓	✓	✓	✓	✓	·
Citigroup	Financials	✓	✓	✓	✓	✓	✓	·
Citizens Financial Group	Financials	–	–	–	–	–	–	·
Clorox	Consumer Staples	✓	✓	✓	✓	✓	✓	·
CME Group	Financials	–	–	–	–	–	–	·
CMS Energy	Utilities	✓	✓	✓	✓	✓	✓	·
Coca-Cola Company (The)	Consumer Staples	✓	✓	✓	✓	✓	✓	·
Cognizant	Information Technology	✓	✓	✓	✓	✓	✓	✓
Coherent Corp.	Information Technology	✓	✓	✓	✓	✓	✓	·
Coinbase	Financials	✓	✓	✓	✓	✓	✓	✓
Colgate-Palmolive	Consumer Staples	✓	✓	✓	✓	✓	✓	·
Comcast	Communication Services	✓	✓	✓	✓	✓	✓	·
Comfort Systems USA	Industrials	–	–	–	–	–	–	·
ConocoPhillips	Energy	✓	✓	✓	✓	✓	✓	·
Consolidated Edison	Utilities	✓	✓	✓	✓	✓	✓	·
Constellation Brands	Consumer Staples	✓	✓	✓	✓	✓	✓	✓
Constellation Energy	Utilities	✓	✓	✓	✓	✓	✓	·
Cooper Companies (The)	Health Care	✓	✓	✓	✓	✓	✓	·
Copart	Industrials	✓	✓	✓	✓	✓	✓	·
Corning Inc.	Information Technology	–	–	–	–	–	–	·
Corpay	Financials	✓	✓	✓	✓	✓	✓	✓
CoStar Group	Real Estate	–	–	–	–	–	–	·
Costco	Consumer Staples	✓	✓	✓	✓	✓	✓	·
CRH plc	Materials	✓	✓	✓	✓	✓	✓	·
CrowdStrike	Information Technology	✓	✓	✓	✓	✓	✓	✓
Crown Castle	Real Estate	✓	✓	✓	✓	✓	✓	·
CSX Corporation	Industrials	–	–	–	–	–	–	·
Cummins	Industrials	✓	✓	✓	✓	✓	✓	·
CVS Health	Health Care	✓	✓	✓	✓	✓	✓	✓
D. R. Horton	Consumer Discretionary	✓	✓	✓	✓	✓	✓	·
Danaher Corporation	Health Care	✓	✓	✓	✓	✓	✓	·
Darden Restaurants	Consumer Discretionary	✓	✓	✓	✓	✓	✓	·
Datadog	Information Technology	✓	✓	✓	✓	✓	✓	✓
DaVita	Health Care	✓	✓	✓	✓	✓	✓	·
Deckers Brands	Consumer Discretionary	✓	✓	✓	✓	✓	✓	·
Deere & Company	Industrials	✓	✓	✓	✓	✓	✓	·
Dell Technologies	Information Technology	✓	✓	✓	✓	✓	✓	✓
Delta Air Lines	Industrials	✓	✓	✓	✓	✓	✓	·
Devon Energy	Energy	✓	✓	✓	✓	✓	✓	·
Dexcom	Health Care	✓	✓	✓	✓	✓	✓	·
Diamondback Energy	Energy	–	–	–	–	–	–	·
Digital Realty	Real Estate	✓	✓	✓	✓	✓	✓	·
Dollar General	Consumer Staples	✓	✓	✓	✓	✓	✓	·
Dollar Tree	Consumer Staples	✓	✓	✓	✓	✓	✓	·
Dominion Energy	Utilities	✓	✓	✓	✓	✓	✓	·
Domino's	Consumer Discretionary	✓	✓	✓	✓	✓	✓	·
DoorDash	Consumer Discretionary	✓	✓	✓	✓	✓	✓	·
Dover Corporation	Industrials	–	–	–	–	–	–	·
Dow Inc.	Materials	✓	✓	✓	✓	✓	✓	·
DTE Energy	Utilities	✓	✓	✓	✓	✓	✓	·
Duke Energy	Utilities	–	–	–	–	–	–	·
DuPont	Industrials	✓	✓	✓	✓	✓	✓	·
Eaton Corporation	Industrials	–	–	–	–	–	–	·
eBay Inc.	Consumer Discretionary	✕	✓	✕	✕	✓	✕	·
EchoStar	Communication Services	✓	✓	✓	✓	✓	✓	·
Ecolab	Materials	✓	✓	✓	✓	✓	✓	·
Edison International	Utilities	✓	✓	✓	✓	✓	✓	·
Edwards Lifesciences	Health Care	✓	✓	✓	✓	✓	✓	·
Elevance Health	Health Care	✓	✓	✓	✓	✓	✓	·
Emcor	Industrials	✓	✓	✓	✓	✓	✓	✓
Emerson Electric	Industrials	✓	✓	✓	✓	✓	✓	·
Entergy	Utilities	✓	✓	✓	✓	✓	✓	·
EOG Resources	Energy	✓	✓	✓	✓	✓	✓	·
EQT Corporation	Energy	✓	✓	✓	✓	✓	✓	·
Equifax	Industrials	✓	✓	✓	✓	✓	✓	✓
Equinix	Real Estate	–	–	–	–	–	–	·
Erie Indemnity	Financials	✓	✓	✓	✓	✓	✓	·
Essex Property Trust	Real Estate	–	–	–	–	–	–	·
Estée Lauder Companies (The)	Consumer Staples	✓	✓	✓	✓	✓	✓	·
Everest Group	Financials	✓	✓	✓	✓	✓	✓	·
Evergy	Utilities	✓	✓	✓	✓	✓	✓	·
Everpure	Information Technology	✓	✓	✓	✓	✓	✓	·
Eversource Energy	Utilities	✓	✓	✓	✓	✓	✓	·
Exelon	Utilities	✓	✓	✓	✓	✓	✓	·
Expand Energy	Energy	✓	✓	✓	✓	✓	✓	·
Expedia Group	Consumer Discretionary	✓	✓	✓	✓	✓	✓	·
Expeditors International	Industrials	✓	✓	✓	✓	✓	✓	·
Extra Space Storage	Real Estate	✓	✓	✓	✓	✓	✓	·
ExxonMobil	Energy	✓	✓	✓	✓	✓	✓	·
F5, Inc.	Information Technology	✓	✓	✓	✓	✓	✓	·
FactSet	Financials	✓	✓	✓	✓	✓	✓	·
Fair Isaac	Information Technology	✓	✓	✓	✓	✓	✓	·
Fastenal	Industrials	–	–	–	–	–	–	·
Federal Realty Investment Trust	Real Estate	✓	✓	✓	✓	✓	✓	·
FedEx	Industrials	✓	✓	✓	✓	✓	✓	·
FedEx Freight	Industrials	–	–	–	–	–	–	·
Ferguson Enterprises	Industrials	✓	✓	✓	✓	✓	✓	·
Fidelity National Information Services	Financials	✓	✓	✓	✓	✓	✓	✓
Fifth Third Bancorp	Financials	–	–	–	–	–	–	·
First Solar	Information Technology	✓	✓	✓	✓	✓	✓	·
FirstEnergy	Utilities	✓	✓	✓	✓	✓	✓	·
Fiserv	Financials	✓	✓	✓	✓	✓	✓	·
Flex Ltd.	Information Technology	✓	✓	✓	✓	✓	✓	✓
Ford Motor Company	Consumer Discretionary	✓	✓	✓	✓	✓	✓	✓
Fortinet	Information Technology	✓	✓	✕	✓	✓	✕	·
Fortive	Industrials	✓	✓	✓	✓	✓	✓	·
Fox Corporation	Communication Services	✓	✓	✓	✓	✓	✓	·
Franklin Resources	Financials	–	–	–	–	–	–	·
Freeport-McMoRan	Materials	✓	✓	✓	✓	✓	✓	·
Garmin	Consumer Discretionary	✓	✓	✓	✕	✓	✓	·
Gartner	Information Technology	✕	✕	✓	✓	✕	✕	·
GE Aerospace	Industrials	✓	✓	✓	✓	✓	✓	✓
GE HealthCare	Health Care	✓	✓	✓	✓	✓	✓	·
GE Vernova	Industrials	✓	✓	✓	✓	✓	✓	·
Gen Digital	Information Technology	✓	✓	✓	✓	✓	✓	·
Generac	Industrials	✓	✓	✓	✓	✓	✓	·
General Dynamics	Industrials	–	–	–	–	–	–	·
General Mills	Consumer Staples	✓	✓	✓	✓	✓	✓	·
General Motors	Consumer Discretionary	–	–	–	–	–	–	·
Genuine Parts Company	Consumer Discretionary	✓	✓	✓	✓	✓	✓	·
Gilead Sciences	Health Care	✓	✓	✓	✓	✓	✓	·
Global Payments	Financials	✓	✓	✓	✓	✓	✓	·
Globe Life	Financials	✓	✓	✓	✓	✓	✓	·
GoDaddy	Information Technology	–	–	–	–	–	–	·
Goldman Sachs	Financials	–	–	–	–	–	–	·
Halliburton	Energy	✓	✓	✓	✓	✓	✓	·
Hartford (The)	Financials	✓	✓	✓	✓	✓	✓	·
Hasbro	Consumer Discretionary	–	–	–	–	–	–	·
HCA Healthcare	Health Care	✓	✓	✓	✓	✓	✓	·
Healthpeak Properties	Real Estate	✓	✓	✓	✓	✓	✓	·
Henry Schein	Health Care	✓	✓	✓	✓	✓	✓	·
Hershey Company (The)	Consumer Staples	✓	✓	✓	✓	✓	✓	·
Hewlett Packard Enterprise	Information Technology	–	–	–	–	–	–	·
Hilton Worldwide	Consumer Discretionary	✓	✓	✓	✓	✓	✓	·
Home Depot (The)	Consumer Discretionary	✓	✓	✓	✓	✓	✓	·
Honeywell Aerospace	Industrials	✓	✓	✓	✓	✓	✓	·
Honeywell Technologies	Industrials	✓	✓	✓	✓	✓	✓	·
Hormel Foods	Consumer Staples	✓	✓	✓	✓	✓	✓	·
Host Hotels & Resorts	Real Estate	–	–	–	–	–	–	·
Howmet Aerospace	Industrials	✓	✓	✓	✓	✓	✓	·
HP Inc.	Information Technology	✓	✓	✓	✓	✓	✓	·
Hubbell Incorporated	Industrials	✓	✓	✓	✓	✓	✓	·
Humana	Health Care	✓	✓	✓	✓	✓	✓	·
Huntington Bancshares	Financials	–	–	–	–	–	–	·
Huntington Ingalls Industries	Industrials	✓	✓	✓	✓	✓	✓	✓
IBM	Information Technology	✓	✓	✓	✓	✓	✓	·
IDEX Corporation	Industrials	✓	✓	✓	✓	✓	✓	·
Idexx Laboratories	Health Care	✓	✓	✓	✓	✓	✓	·
Illinois Tool Works	Industrials	✓	✓	✓	✓	✓	✓	·
Illumina, Inc.	Health Care	✓	✓	✓	✓	✓	✓	·
Incyte	Health Care	✓	✓	✓	✓	✓	✓	·
Ingersoll Rand	Industrials	–	–	–	–	–	–	·
Insulet Corporation	Health Care	✓	✓	✓	✓	✓	✓	·
Intel	Information Technology	✓	✓	✓	✓	✓	✓	✓
Interactive Brokers	Financials	✓	✓	✓	✓	✓	✓	✓
Intercontinental Exchange	Financials	✓	✓	✓	✓	✓	✓	·
International Flavors & Fragrances	Materials	✓	✓	✓	✓	✓	✓	·
International Paper	Materials	✓	✓	✓	✓	✓	✓	·
Intuit	Information Technology	✓	✓	✓	✓	✓	✓	✓
Intuitive Surgical	Health Care	✓	✓	✓	✓	✓	✓	·
Invesco	Financials	✓	✓	✓	✓	✓	✓	·
Invitation Homes	Real Estate	✓	✓	✓	✓	✓	✓	·
IQVIA	Health Care	✓	✓	✓	✓	✓	✓	·
Iron Mountain	Real Estate	–	–	–	–	–	–	·
J.B. Hunt	Industrials	✓	✓	✓	✓	✓	✓	✓
J.M. Smucker Company (The)	Consumer Staples	✓	✓	✓	✓	✓	✓	·
Jabil	Information Technology	✓	✓	✓	✓	✓	✓	✓
Jack Henry & Associates	Financials	✓	✓	✓	✓	✓	✓	·
Jacobs Solutions	Industrials	✓	✓	✓	✓	✓	✓	·
Johnson & Johnson	Health Care	✓	✓	✓	✓	✓	✓	·
Johnson Controls	Industrials	✓	✓	✓	✓	✓	✓	✓
JPMorgan Chase	Financials	✓	✓	✓	✓	✓	✓	·
Kenvue	Consumer Staples	✓	✓	✓	✓	✓	✓	·
Keurig Dr Pepper	Consumer Staples	✓	✓	✓	✓	✓	✓	·
KeyCorp	Financials	–	–	–	–	–	–	·
Keysight Technologies	Information Technology	✓	✓	✓	✓	✓	✓	·
Kimberly-Clark	Consumer Staples	✕	✓	✕	✓	✕	✕	·
Kimco Realty	Real Estate	–	–	–	–	–	–	·
Kinder Morgan	Energy	✓	✓	✓	✓	✓	✓	·
KKR & Co.	Financials	✓	✓	✓	✓	✓	✓	·
KLA Corporation	Information Technology	✓	✓	✓	✓	✓	✓	·
Kraft Heinz	Consumer Staples	✓	✓	✓	✓	✓	✓	·
Kroger	Consumer Staples	–	–	–	–	–	–	·
L3Harris	Industrials	✓	✓	✓	✓	✓	✓	·
Labcorp	Health Care	✓	✓	✓	✓	✓	✓	✓
Lam Research	Information Technology	✓	✓	✓	✓	✓	✓	·
Las Vegas Sands	Consumer Discretionary	–	–	–	–	–	–	·
Leidos	Industrials	✓	✓	✓	✓	✓	✓	✓
Lennar	Consumer Discretionary	✓	✓	✓	✓	✓	✓	✓
Lennox International	Industrials	✓	✓	✓	✓	✓	✓	·
Lilly (Eli)	Health Care	✓	✓	✓	✓	✓	✓	·
Linde plc	Materials	✓	✓	✓	✓	✓	✓	·
Live Nation Entertainment	Communication Services	✓	✓	✓	✓	✓	✓	·
Lockheed Martin	Industrials	✓	✓	✓	✓	✓	✓	·
Loews Corporation	Financials	✓	✓	✓	✓	✓	✓	·
Lowe's	Consumer Discretionary	–	–	–	–	–	–	·
Lululemon Athletica	Consumer Discretionary	–	–	–	–	–	–	·
Lumentum	Information Technology	✓	✓	✓	✓	✓	✓	·
LyondellBasell	Materials	✓	✓	✓	✓	✓	✓	·
M&T Bank	Financials	✓	✓	✓	✓	✓	✓	·
Marathon Petroleum	Energy	–	–	–	–	–	–	·
Marriott International	Consumer Discretionary	✓	✓	✓	✓	✓	✓	·
Marsh McLennan	Financials	✓	✓	✓	✓	✓	✓	·
Martin Marietta Materials	Materials	✓	✓	✓	✓	✓	✓	·
Marvell Technology	Information Technology	–	–	–	–	–	–	·
Masco	Industrials	✓	✓	✓	✓	✓	✓	·
Mastercard	Financials	–	–	–	–	–	–	·
McCormick & Company	Consumer Staples	✓	✓	✓	✓	✓	✓	✓
McDonald's	Consumer Discretionary	–	–	–	–	–	–	·
McKesson	Health Care	✓	✓	✓	✓	✓	✓	·
Medtronic	Health Care	✓	✓	✓	✓	✓	✓	·
Merck & Co.	Health Care	✓	✓	✓	✓	✓	✓	·
Meta Platforms	Communication Services	✓	✓	✓	✓	✓	✓	·
MetLife	Financials	✓	✓	✓	✓	✓	✓	✓
Mettler Toledo	Health Care	✕	✓	✓	✓	✓	✓	·
MGM Resorts	Consumer Discretionary	–	–	–	–	–	–	·
Microchip Technology	Information Technology	–	–	–	–	–	–	·
Micron Technology	Information Technology	✓	✓	✓	✓	✓	✓	·
Microsoft	Information Technology	✓	✓	✓	✓	✓	✓	·
Mid-America Apartment Communities	Real Estate	✓	✓	✓	✓	✓	✓	·
Moderna	Health Care	✓	✓	✓	✓	✓	✓	·
Mondelez International	Consumer Staples	✓	✓	✓	✓	✓	✓	·
Monolithic Power Systems	Information Technology	–	–	–	–	–	–	·
Monster Beverage	Consumer Staples	✓	✓	✓	✓	✓	✓	·
Moody's Corporation	Financials	✓	✓	✓	✓	✓	✓	·
Morgan Stanley	Financials	–	–	–	–	–	–	·
Mosaic Company (The)	Materials	✓	✓	✓	✓	✓	✓	·
Motorola Solutions	Information Technology	✓	✓	✓	✓	✓	✓	·
MSCI	Financials	✓	✓	✓	✓	✓	✓	·
Nasdaq, Inc.	Financials	–	–	–	–	–	–	·
NetApp	Information Technology	–	–	–	–	–	–	·
Netflix	Communication Services	✓	✓	✓	✓	✓	✕	·
Newmont	Materials	✓	✓	✓	✓	✓	✓	·
News Corp	Communication Services	✓	✓	✓	✓	✓	✓	·
NextEra Energy	Utilities	✓	✓	✓	✓	✓	✓	·
Nike, Inc.	Consumer Discretionary	✓	✓	✓	✓	✓	✓	·
NiSource	Utilities	✓	✓	✓	✓	✓	✓	·
Nordson Corporation	Industrials	–	–	–	–	–	–	·
Norfolk Southern	Industrials	✓	✓	✓	✓	✓	✓	·
Northern Trust	Financials	✓	✓	✓	✓	✓	✓	·
Northrop Grumman	Industrials	✕	✓	✕	✓	✕	✕	·
Norwegian Cruise Line Holdings	Consumer Discretionary	✓	✓	✓	✓	✓	✓	·
NRG Energy	Utilities	✓	✓	✓	✓	✓	✓	·
Nucor	Materials	✓	✓	✓	✓	✓	✓	✓
Nvidia	Information Technology	✓	✓	✓	✓	✓	✓	✓
NVR, Inc.	Consumer Discretionary	✓	✓	✓	✓	✓	✓	·
NXP Semiconductors	Information Technology	✓	✓	✓	✓	✓	✓	·
O'Reilly Automotive	Consumer Discretionary	–	–	–	–	–	–	·
Occidental Petroleum	Energy	–	–	–	–	–	–	·
Old Dominion	Industrials	✓	✓	✓	✓	✓	✓	·
Omnicom Group	Communication Services	✓	✓	✓	✓	✓	✓	·
ON Semiconductor	Information Technology	✓	✓	✓	✓	✓	✓	·
Oneok	Energy	✓	✓	✓	✓	✓	✓	·
Oracle Corporation	Information Technology	–	–	–	–	–	–	·
Otis Worldwide	Industrials	✓	✓	✓	✓	✓	✓	·
Paccar	Industrials	–	–	–	–	–	–	·
Packaging Corporation of America	Materials	–	–	–	–	–	–	·
Palantir Technologies	Information Technology	✓	✓	✓	✓	✓	✓	·
Palo Alto Networks	Information Technology	✓	✓	✓	✓	✓	✓	✓
Parker Hannifin	Industrials	–	–	–	–	–	–	·
Paychex	Industrials	✓	✓	✓	✓	✓	✓	✓
PayPal	Financials	✓	✓	✓	✓	✓	✓	✓
Pentair	Industrials	✓	✓	✓	✓	✓	✓	·
PepsiCo	Consumer Staples	✓	✓	✓	✓	✓	✓	·
Pfizer	Health Care	–	–	–	–	–	–	·
PG&E Corporation	Utilities	✕	✓	✓	✓	✓	✓	·
Philip Morris International	Consumer Staples	✓	✓	✓	✓	✓	✓	·
Phillips 66	Energy	✓	✓	✓	✓	✓	✓	·
Pinnacle West Capital	Utilities	✓	✓	✓	✓	✓	✓	·
PNC Financial Services	Financials	✓	✓	✓	✓	✓	✓	·
PPG Industries	Materials	✓	✓	✓	✓	✓	✓	·
PPL Corporation	Utilities	✓	✓	✓	✓	✓	✓	·
Principal Financial Group	Financials	✓	✓	✓	✓	✓	✓	·
Procter & Gamble	Consumer Staples	✓	✓	✓	✓	✓	✓	·
Progressive Corporation	Financials	✓	✓	✓	✓	✓	✓	·
Prologis	Real Estate	✓	✓	✓	✓	✓	✓	·
Prudential Financial	Financials	✕	✓	✓	✓	✕	✕	·
PTC Inc.	Information Technology	–	–	–	–	–	–	·
Public Service Enterprise Group	Utilities	✓	✓	✓	✓	✓	✓	·
Public Storage	Real Estate	✓	✓	✓	✓	✓	✓	✓
PulteGroup	Consumer Discretionary	✓	✓	✓	✓	✓	✓	·
Qnity Electronics	Information Technology	✓	✓	✓	✓	✓	✓	✓
Qualcomm	Information Technology	✓	✓	✓	✓	✓	✓	✓
Quanta Services	Industrials	✓	✓	✓	✓	✓	✓	·
Quest Diagnostics	Health Care	✓	✓	✓	✓	✓	✓	·
Ralph Lauren Corporation	Consumer Discretionary	✓	✓	✓	✓	✓	✓	·
Raymond James Financial	Financials	–	–	–	–	–	–	·
Realty Income	Real Estate	✓	✓	✓	✓	✓	✓	·
Reddit	Communication Services	✓	✓	✓	✓	✓	✓	·
Regency Centers	Real Estate	✓	✓	✓	✓	✓	✓	·
Regeneron Pharmaceuticals	Health Care	✓	✓	✓	✓	✓	✓	·
Regions Financial Corporation	Financials	✓	✓	✓	✓	✓	✓	·
Republic Services	Industrials	✓	✓	✓	✓	✓	✓	·
ResMed\|	Health Care	–	–	–	–	–	–	·
Revvity	Health Care	–	–	–	–	–	–	·
Robinhood Markets	Financials	✓	✓	✓	✓	✓	✓	·
Rockwell Automation	Industrials	✓	✓	✓	✓	✓	✓	✓
Rollins, Inc.	Industrials	✓	✓	✓	✓	✓	✓	·
Roper Technologies	Information Technology	–	–	–	–	–	–	·
Ross Stores	Consumer Discretionary	✓	✓	✓	✓	✓	✓	·
Royal Caribbean Group	Consumer Discretionary	✓	✓	✓	✓	✓	✓	·
RTX Corporation	Industrials	–	–	–	–	–	–	·
S&P Global	Financials	–	–	–	–	–	–	·
Salesforce	Information Technology	✓	✓	✓	✓	✓	✓	✓
Sandisk	Information Technology	✓	✓	✓	✓	✓	✓	✓
SBA Communications	Real Estate	✓	✓	✓	✓	✓	✓	·
Schlumberger	Energy	✓	✓	✓	✓	✓	✓	·
Seagate Technology	Information Technology	✓	✓	✓	✓	✓	✓	·
Sempra	Utilities	✓	✓	✓	✓	✓	✓	·
ServiceNow	Information Technology	–	–	–	–	–	–	·
Sherwin-Williams	Materials	✓	✓	✓	✓	✓	✓	·
Simon Property Group	Real Estate	✕	✓	✕	✕	✕	✕	·
Skydance Corporation	Communication Services	✓	✓	✓	✓	✓	✓	·
Skyworks Solutions	Information Technology	✓	✓	✓	✓	✓	✓	·
Smurfit Westrock	Materials	✓	✓	✓	✓	✓	✓	·
Snap-on	Industrials	✓	✓	✓	✓	✓	✓	·
Solventum	Health Care	✓	✓	✓	✓	✓	✓	✓
Southern Company	Utilities	✓	✓	✓	✓	✓	✓	·
Southwest Airlines	Industrials	✓	✓	✓	✓	✓	✓	·
Stanley Black & Decker	Industrials	✓	✓	✓	✓	✓	✓	·
Starbucks	Consumer Discretionary	✓	✓	✓	✓	✓	✓	·
State Street Corporation	Financials	✓	✓	✓	✓	✓	✓	·
Steel Dynamics	Materials	✓	✓	✓	✓	✓	✓	·
Steris	Health Care	✓	✓	✓	✓	✓	✓	·
Stryker Corporation	Health Care	✓	✓	✓	✓	✓	✓	·
Supermicro	Information Technology	✓	✓	✓	✓	✓	✓	·
Synchrony Financial	Financials	✓	✓	✓	✓	✓	✓	·
Synopsys	Information Technology	✓	✓	✓	✓	✓	✓	✓
Sysco	Consumer Staples	✓	✓	✓	✓	✓	✓	·
T-Mobile US	Communication Services	–	–	–	–	–	–	·
T. Rowe Price	Financials	✓	✓	✓	✓	✓	✓	·
Take-Two Interactive	Communication Services	✓	✓	✓	✓	✓	✓	·
Tapestry, Inc.	Consumer Discretionary	✓	✓	✓	✓	✓	✓	·
Targa Resources	Energy	–	–	–	–	–	–	·
Target Corporation	Consumer Staples	✓	✓	✓	✓	✓	✓	✓
TE Connectivity	Information Technology	✓	✓	✓	✓	✓	✓	·
Teledyne Technologies	Information Technology	✓	✓	✓	✓	✓	✓	·
Teradyne	Information Technology	✓	✓	✓	✓	✓	✓	·
Tesla, Inc.	Consumer Discretionary	–	–	–	–	–	–	·
Texas Instruments	Information Technology	✓	✓	✓	✓	✓	✓	·
Texas Pacific Land Corporation	Energy	✓	✓	✓	✓	✓	✓	·
Textron	Industrials	✓	✓	✓	✓	✓	✓	·
Thermo Fisher Scientific	Health Care	✓	✓	✓	✓	✓	✓	✓
TJX Companies	Consumer Discretionary	–	–	–	–	–	–	·
TKO Group Holdings	Communication Services	✓	✓	✓	✓	✓	✓	·
Tractor Supply	Consumer Discretionary	–	–	–	–	–	–	·
Trane Technologies	Industrials	–	–	–	–	–	–	·
TransDigm Group	Industrials	✓	✓	✓	✓	✓	✓	·
Travelers Companies (The)	Financials	✓	✓	✓	✓	✓	✓	·
Trimble Inc.	Information Technology	✓	✓	✓	✓	✓	✓	✓
Truist Financial	Financials	✓	✓	✓	✓	✓	✓	·
Twilio	Information Technology	✓	✓	✓	✓	✓	✓	✓
Tyler Technologies	Information Technology	✓	✓	✓	✓	✓	✓	·
Tyson Foods	Consumer Staples	✓	✓	✓	✓	✓	✓	·
U.S. Bancorp	Financials	✓	✓	✓	✓	✓	✓	✓
Uber	Industrials	✓	✓	✓	✓	✓	✓	·
UDR, Inc.	Real Estate	✓	✓	✓	✓	✓	✓	✓
Ulta Beauty	Consumer Discretionary	✓	✓	✓	✓	✓	✓	·
Union Pacific Corporation	Industrials	✓	✓	✓	✓	✓	✓	·
United Airlines Holdings	Industrials	–	–	–	–	–	–	·
United Parcel Service	Industrials	–	–	–	–	–	–	·
United Rentals	Industrials	✓	✓	✓	✓	✓	✓	·
UnitedHealth Group	Health Care	✓	✓	✓	✓	✓	✓	·
Universal Health Services	Health Care	✓	✓	✓	✓	✓	✓	✓
Valero Energy	Energy	✓	✓	✓	✓	✓	✓	·
Veeva Systems	Health Care	✓	✓	✓	✓	✓	✓	✓
Ventas	Real Estate	✓	✓	✓	✓	✓	✓	·
Veralto	Industrials	✓	✓	✓	✓	✓	✓	·
Verisign	Information Technology	✓	✓	✓	✓	✓	✓	·
Verisk Analytics	Industrials	✓	✓	✓	✓	✓	✓	·
Verizon	Communication Services	✓	✓	✓	✓	✓	✓	·
Vertex Pharmaceuticals	Health Care	–	–	–	–	–	–	·
Vertiv	Industrials	✓	✓	✓	✓	✓	✓	✓
Viatris	Health Care	✓	✓	✓	✓	✓	✓	·
Vici Properties	Real Estate	✓	✓	✓	✓	✓	✓	·
Visa Inc.	Financials	✓	✓	✓	✓	✓	✓	·
Vistra Corp.	Utilities	✓	✓	✓	✓	✓	✓	·
Vivmark Residential	Real Estate	✓	✓	✓	✓	✓	✓	·
Vulcan Materials Company	Materials	✓	✓	✓	✓	✓	✓	·
Vylor	Consumer Staples	✓	✓	✓	✓	✓	✓	·
W. R. Berkley Corporation	Financials	✓	✓	✓	✓	✓	✓	·
W. W. Grainger	Industrials	–	–	–	–	–	–	·
Wabtec	Industrials	✓	✓	✓	✓	✓	✓	·
Walmart	Consumer Staples	✓	✓	✓	✓	✓	✓	·
Walt Disney Company (The)	Communication Services	✓	✓	✓	✓	✓	✓	·
Waste Management	Industrials	✓	✓	✓	✓	✓	✓	·
Waters Corporation	Health Care	–	–	–	–	–	–	·
WEC Energy Group	Utilities	✓	✓	✓	✓	✓	✓	·
Wells Fargo	Financials	✓	✓	✓	✓	✓	✓	·
Welltower	Real Estate	✓	✓	✓	✓	✓	✓	·
West Pharmaceutical Services	Health Care	✓	✓	✓	✓	✓	✓	✓
Western Digital	Information Technology	✓	✓	✓	✓	✓	✓	✓
Weyerhaeuser	Real Estate	✓	✓	✓	✓	✓	✓	·
Williams Companies	Energy	✓	✓	✓	✓	✓	✓	·
Williams-Sonoma, Inc.	Consumer Discretionary	–	–	–	–	–	–	·
Willis Towers Watson	Financials	–	–	–	–	–	–	·
Workday, Inc.	Information Technology	✓	✓	✓	✓	✓	✓	·
Wynn Resorts	Consumer Discretionary	✓	✓	✓	✓	✓	✓	·
Xcel Energy	Utilities	✓	✓	✓	✓	✓	✓	·
Xylem Inc.	Industrials	✓	✓	✓	✓	✓	✓	·
Yum! Brands	Consumer Discretionary	–	–	–	–	–	–	·
Zebra Technologies	Information Technology	✓	✓	✓	✓	✓	✓	✓
Zimmer Biomet	Health Care	✓	✓	✓	✓	✓	✓	·
Zoetis	Health Care	✓	✓	✓	✓	✓	✓	·

Checked on 17 October 2026.

Methodology

How we checked

On 17 October 2026 we read the robots.txt file on the main website of every S&P 500 company. We looked at six AI crawlers: GPTBot (used to train ChatGPT), OAI-SearchBot (ChatGPT search), ClaudeBot (Claude), PerplexityBot (Perplexity), Google-Extended (Gemini) and CCBot (Common Crawl, whose data trains many AI models).

A crawler counts as blocked only when the file tells it not to read any page. Blocks on parts of a website are not counted. Some websites turned away our automated request, so we could not read their rules; they are marked with a dash and left out of the percentages. We also checked each website for a file at /llms.txt; websites that turned us away may have one we could not see. The company list and sectors come from the S&P 500 constituents list (GICS sectors).

We checked every result twice. On 18 October 2026 we re-read each rules file with a stricter method, which confirmed all blocking results, and we opened each llms.txt file by hand, removing one that was an ordinary web page.