# Meta's new terms let it crawl your website from 20 November 2026

> Meta's updated ad terms let it crawl advertisers' websites from 20 November 2026. Which Meta crawlers do what, and which ones to block or keep.
- **Author**: Marxx
- **Published**: 2026-10-25
- **Category**: Meta
- **URL**: https://marxx.ai/library-news/meta-website-crawling-terms-robots-txt

---

Meta's updated Commercial Terms let it crawl websites linked to your business for its advertising and business products from **20 November 2026**. You can steer most of Meta's crawlers with your robots.txt file, but blocking the wrong one breaks your ad link previews.

## What we know

- **Confirmed by Meta:** its [web crawlers page](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/) lists five crawlers and says which follow robots.txt (checked 25 Oct 2026).
- **Reported:** by 23 October Meta removed a sentence about training AI from the crawling clause and moved the start date to 20 November, as [PPC Land](https://ppc.land/meta-delays-its-ai-terms-to-november-20-the-trust-problem-does-not-move/) reported. Our [page on the delay](/news/meta-commercial-terms-delayed-november-20) has the timeline.
- **Reported:** on 23 October the European Publishers Council asked the European Commission to pause the clause before 20 November, per [PPC Land](https://ppc.land/european-publishers-ask-eu-to-halt-metas-crawling-terms-before-nov-20/). Meta has not replied in public.

## What it means for your ad account

Your landing pages are now inside the terms you accept by running ads. The robots.txt file on your site is your main control.

| Crawler | What Meta uses it for | Follows robots.txt |
|---|---|---|
| FacebookExternalHit | Title, text and picture for shared links | Mostly |
| Meta-ExternalAds | Ads and business products | Yes |
| Meta-ExternalAgent | Training AI models and indexing | Yes |
| Meta-WebIndexer | Meta AI search answers | Yes |
| Meta-ExternalFetcher | Links a user asks Meta AI to open | No |

## What to do this week

1. **Open yoursite.com/robots.txt** and look for any Meta crawler names.
2. **Keep FacebookExternalHit allowed.** It builds the preview of your link in ads and posts.
3. **Disallow Meta-ExternalAgent** if you do not want your pages used to train AI.
4. **Decide on Meta-ExternalAds** with whoever runs your ads. Our [AI features terms page](/news/meta-ai-features-terms-october-30) explains the related AI edits clause.
5. **Leave your pixel and server events alone.** Our [Pixel vs Conversions API](/compare/pixel-vs-conversions-api) page explains why they are separate from crawling.

## FAQ

### Does blocking Meta-ExternalAgent stop my ads?

Meta's page does not say it affects ads. It describes that crawler as used for AI training and indexing.

### Is the training sentence really gone?

Reports say it was removed by 23 October. Publishers argue the remaining wording could still cover AI use.

### Does this apply in India and the Gulf?

Meta named no country exceptions for the new terms.

---
- [More Meta articles](https://marxx.ai/library-news/category/meta)
- [All articles](https://marxx.ai/library-news)