AI Search Can't See Your Instagram: Six Platforms, Tested

You have poured your brand's work into Instagram and Threads.

So when someone asks ChatGPT "which brands in Taiwan do social media growth", can it read you?

In August 2026 we went and fetched the robots.txt of six major platforms. The answers diverge more than you would expect — and one platform does the exact opposite of what most people assume.

First, two layers people keep conflating

Discussions about "will AI see my content" routinely collapse two different questions into one:

Layer one: can the crawler fetch it? Every site has a plain-text file at its root called robots.txt telling each crawler which paths it may and may not fetch. This is a rule the platform declares unilaterally, and anyone can open it and read it.

Layer two: once fetched, does it get used? Being fetchable does not mean the content lands in an index, and landing in an index does not mean an AI will cite it in an answer.

This article deals only with layer one, because that is the only part that can be publicly verified. Layer two depends on each vendor's internal ranking logic, and no first-party data exists for it — so you will not find a "X% of AI citations come from social" figure here. We could not trace any of the numbers circulating online back to a source.

Layer one is only a threshold, but it is a hard one: if robots.txt says no, a compliant crawler never arrives, and the other two layers never come up.

Results: how six platforms treat AI crawlers (August 2026)

Verifying this yourself takes seconds: open https://www.instagram.com/robots.txt in a browser. No tooling required.

Instagram and Threads: search engines and AI crawlers, handled separately

These two platforms ship identical robots.txt groupings, which makes it plainly one policy. Across 36 crawler sections, they split cleanly into two piles.

The 15 written off with Disallow: / (fully blocked):

Amazonbot, Applebot-Extended, Brightbot, ClaudeBot, Google-Extended,
GPTBot, PerplexityBot, PetalBot, Scrapy, uptimerobot, viberbot,
YaK, Yandex, Yeti, and the catch-all *

The 21 that only lose specific paths (main content stays reachable):

Applebot, Bingbot, deepcrawl, Discordbot, DuckDuckBot,
facebookexternalhit, Googlebot, Googlebot-Image, Googlebot-Video,
Googlebot-News, Google-InspectionTool, LinkedInBot, lumar, msnbot,
Pinterestbot, Screaming Frog SEO Spider, seznambot, Slurp, teoma,
TelegramBot, Twitterbot

See the pattern? The second pile is entirely conventional search engines, link-preview bots and SEO auditing crawlers. The first pile is, almost without exception, crawlers that train AI models or feed AI search.

Apple appearing on both sides is the giveaway: Applebot (Apple's ordinary search crawler) gets partial access, while Applebot-Extended (the one Apple uses for AI training) is fully blocked. Google is treated the same way — Googlebot allowed, Google-Extended (which feeds Gemini and other generative products) blocked. Same company, two crawlers, two verdicts. That is not a misconfiguration; it is a deliberate split.

Here is the list Instagram hands Googlebot:

Disallow: /*/c/
Disallow: /*/comments/
Disallow: /*/liked_by/
Disallow: /accounts/login/*?next=
Disallow: /ajax/
Disallow: /client_error/
Disallow: /direct/
Disallow: /logging/
Disallow: /publicapi/
Disallow: /qp/batch_fetch_web/
Disallow: /query/
Allow: /places/c/

What is blocked: comments, like lists, direct messages, internal APIs. Profile pages and post pages are not on the list.

To be precise: robots.txt permitting a fetch does not mean those pages get indexed by Google or rank well. Fetching is the first gate; indexing and ranking are two more. This article has no indexing data, so it will not draw that conclusion for you.

The reverse inference, however, holds firmly: a crawler written off with Disallow: / never clears the first gate at all.

Facebook runs it backwards (the counter-intuitive part)

If you assumed all three Meta properties share one policy, Facebook will surprise you.

• User-agent: * → Disallow: / (generic crawlers fully blocked)
• GPTBot, PerplexityBot, ClaudeBot, Google-Extended → each receives the same list, blocking only facebook.com/*/plugins/*, facebook.com/ajax/, facebook.com/login*, facebook.com/share*, facebook.com/photos.php, facebook.com/meta-ai-answers/ and similar, and even spelling out Allow: /careers/ and Allow: /safetycheck/

In other words, on Facebook a named AI crawler has broader access than an anonymous one.

We are not going to explain why Instagram and Threads slam the door while Facebook holds it open — Meta has published no reasoning, and any motive we offered would be speculation. The verifiable fact is simply this: the three platforms point in different directions, so do not treat them as one.

TikTok and YouTube: one blocks everything, one really has no dedicated rules

These two get lumped together constantly, but they actually run opposite policies.

TikTok is the strictest of the six platforms towards AI crawlers. Its robots.txt has only three sections, and the first one puts 25 user agents together under a single Disallow: /:

Baiduspider, 360Spider, Sogouspider, Yisouspider, PetalBot,
GPTBot, OAI-SearchBot, ChatGPT-User, anthropic-ai, ClaudeBot,
Claude-User, Claude-SearchBot, PerplexityBot, Perplexity-User,
Google-Extended, Applebot-Extended, meta-externalagent, CCBot,
Bytespider, DuckAssistBot, Gemini-Deep-Research,
GoogleAgent-Mariner, Google-NotebookLM, AI2Bot, MistralAI-User

Twenty of those are AI crawlers; the other five are Chinese search engine crawlers. The User-agent: * section that generic crawlers fall back to is comparatively permissive: TikTok explicitly allows tiktok.com/foryou, tiktok.com/discover, tiktok.com/tag, tiktok.com/music and tiktok.com/share, blocking mainly on-site search (tiktok.com/search?), shop pages and embed endpoints. Same direction as Instagram: search and previews open, AI shut.

YouTube is the one that really has no dedicated rules. Its entire robots.txt contains just two sections, Mediapartners-Google* and *, with no mention of GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, ClaudeBot or Google-Extended. The robots.txt rule is that a crawler with no dedicated section falls back to User-agent: * — and YouTube's * blocks youtube.com/api/, youtube.com/results (search results), youtube.com/login and youtube.com/comment, all utility paths. youtube.com/watch video pages are not on the blocklist.

In other words, YouTube is the only one of the six that currently gives AI crawlers no different treatment from anyone else.

"AI crawler" is not one thing: four crawlers, four jobs

This is the most misunderstood part. People often object: "but I paste an Instagram link into ChatGPT and it reads it just fine." That does not contradict anything above, because OpenAI operates several separate crawlers.

Per OpenAI's official crawler documentation (read August 2026):

The last one is the key. OpenAI classifies ChatGPT-User as user-initiated activity, and the documentation states that robots.txt rules may not apply to it. Note also that OAI-AdsBot only visits pages submitted as ads, which is a separate question from whether your content gets cited.

So the seemingly contradictory experience makes perfect sense:

• You paste an Instagram link into ChatGPT → ChatGPT-User fetches it → it works
• Someone asks ChatGPT a question without naming a link, so it searches its index → that index was built by OAI-SearchBot → which has no dedicated section on Instagram and inherits *'s Disallow: / → your content is not in there

The first scenario requires someone to already know you and to paste your link by hand. Only the second is discovery. You want the second, and on Instagram and Threads the second is currently closed.

What this means if you run social accounts

First, what it does not mean: social is not pointless. Algorithmic distribution, engagement and conversion on Instagram and Threads all still work exactly as before. If you should be posting, keep posting — the hundred-plus guides on this site are about doing precisely that, and this article overturns none of them.

What it does mean is narrower and more specific: the content you have built up on social platforms is currently absent from the path where someone asks an AI and the AI goes looking.

If your brand exists only on Instagram and Threads, then at the AI-search layer you have no citable asset at all. And user behaviour keeps drifting that way — we covered the traffic-structure side of this in the piece on blogger KPI pressure and website traffic.

The practical response is not to keep all your eggs in one basket: let social handle reach and engagement, and let content you own (your site, your blog) handle being found and being cited. They solve different stages of the same problem.

As for how to write that owned content so an AI will actually quote it — how to chunk paragraphs, whether to front-load the answer, which sentence your brand name belongs in — that is a separate discipline. Our sister brand AI SEO Hacker, run by the same team as Lion Fans, works on exactly that and has broken down what makes content quotable verbatim by an AI. The social-platform layer is where our information ends; the other half is their territory.

Three things you can do now

1. Verify it yourself instead of trusting this article. Open platform-url/robots.txt in a browser and Ctrl+F for GPTBot. A Disallow: / means a full block. These policies change without notice; everything here reflects August 2026.

2. Do not let important content exist only as a social post. Product details, service processes, professional positions, FAQs — anything you want people (and AI) to be able to look up should also live somewhere you control. A social post can be the entry point; it should not be the only copy.

3. Confirm your own content is actually readable. Plenty of people configure robots.txt and never test it. Use curl with different user agents against your own pages and compare:

# Fetch your own page as GPTBot
curl -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot" \
     -o /dev/null -w "%{http_code} %{size_download}\n" https://your-domain/

# Control: an ordinary browser
curl -A "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 Chrome/131.0.0.0 Safari/537.36" \
     -o /dev/null -w "%{http_code} %{size_download}\n" https://your-domain/

Both lines should return the same status code and the same response size. If the AI crawler gets a 403, a 429, or a noticeably smaller body, your host or CDN is blocking it independently of robots.txt — and a permissive robots.txt will not save you.

(We ran the same test against this site: GPTBot, OAI-SearchBot and an ordinary browser all received identical responses.)

Frequently Asked Questions

My Instagram posts show up in Google. Doesn't that mean ChatGPT can read them?

No. These are two separate rule sets. Instagram opens its main content paths to Googlebot while writing a flat Disallow: / for GPTBot, ClaudeBot, PerplexityBot and Google-Extended. Even Google's own two crawlers are split: Googlebot allowed, Google-Extended — the one feeding generative products — blocked.

Why does ChatGPT read the content when I paste an Instagram link?

Because that is ChatGPT-User doing the work. OpenAI's documentation classifies it as a user-initiated live fetch and notes that robots.txt rules may not apply. It is a different crawler from OAI-SearchBot, which builds the search index. Being readable when you paste a link does not mean an AI can find you when someone else asks a question.

Facebook is friendlier to AI crawlers. Should I invest more there?

This article does not suggest choosing platforms based on crawler policy. Platform choice should follow where your audience is, whether the content format fits, and how well the conversion path works — all of which outweigh robots.txt by a wide margin. For that decision, see the guide to choosing a social platform. Platform policies also change at will, which makes them shaky ground for long-term strategy.

How often do these policies change, and how do I keep track?

There is no fixed cycle; platforms can revise robots.txt at any time and rarely announce it. The most reliable method is to open the file yourself periodically — it takes thirty seconds. Everything stated here reflects testing done in August 2026.

Is my Threads content blocked the same way?

Yes. Threads' blocklist is identical to Instagram's — the same 15 fully blocked, the same 21 partially open. For Threads growth itself, see the introduction to Threads marketing.

Closing

The conclusion here is not "social is useless" — it is "social and owned content solve different problems, so don't let one replace the other."

Social platforms get you reach, engagement and conversions, and none of that value has changed. But on the increasingly common path where someone asks an AI and the AI goes looking for an answer, Instagram and Threads currently keep the door shut, and that door is not yours to open.

The door on your own side is.