Your Link Preview Is Broken Because the Crawler Never Saw the Page

Our image pipeline started emitting WebP last spring. Smaller files, better scores, everyone happy — and every link we shared on LinkedIn for the next three weeks rendered as a grey box. The Open Graph tags were perfect. I checked them a dozen times, in the page source, in the analyser, in a colleague's browser. The tags were never the problem. LinkedIn's crawler had fetched a WebP file, decided it could not use it, and rendered nothing rather than telling anyone why.

That is the shape of nearly every broken preview I have debugged since. The tags are fine. What differs is what the crawler received — which is frequently not what you see when you open the page yourself. Before you touch a single <meta> tag, find out what the bot actually got.

The Crawler Is Not a Browser

A preview crawler is a small HTTP client with a short attention span. It does not run JavaScript. It does not have your session cookie. It follows a couple of redirects, waits a few seconds, reads the first chunk of HTML, and leaves. It never scrolls, never hydrates, and never sees anything your framework renders after the initial response.

Everything that breaks previews follows from that:

• Tags injected client-side by a head manager do not exist as far as the bot is concerned.
• A page that requires a login returns your login screen's tags, not the article's.
• A relative og:image path has no origin to resolve against, so it is dropped.
• An image endpoint that needs a signed query parameter or a cookie returns 403 to the bot.

The Open Graph tag reference covers what to write and how each platform caches it. This article is about the layer underneath: proving what the crawler received.

Reproduce It in One Command

Every diagnosis starts here. Pretend to be the bot:

# What Facebook, Instagram and Messenger see
curl -sSL -A 'facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)' \
     -D /tmp/headers.txt https://example.com/your-page | grep -iE '<meta[^>]+og:'

# What LinkedIn sees
curl -sSL -A 'LinkedInBot/1.0 (compatible; Mozilla/5.0; +http://www.linkedin.com)' \
     https://example.com/your-page | grep -iE '<meta[^>]+og:'

# What Slack sees
curl -sSL -A 'Slackbot-LinkExpanding 1.0 (+https://api.slack.com/robots)' \
     https://example.com/your-page | grep -iE '<meta[^>]+og:'

Then read /tmp/headers.txt for the status code and content type. Three outcomes, three completely different bugs:

Run the same command with a normal browser user agent. If the browser UA gets tags and the bot UA does not, you have found your culprit without touching the application code at all.

Bot Protection Is the Most Common Cause

Cloudflare's Bot Fight Mode, AWS WAF's default rules, and most "block suspicious traffic" toggles treat a headless HTTP client with an unusual user agent exactly like a scraper. The page loads perfectly for you and returns a challenge to facebookexternalhit. Nothing in your application logs looks wrong, because the request never reached your application.

Check, in this order: the edge/WAF rules for user-agent filtering, rate limiting on the page and on the image path, robots.txt for a blanket Disallow that these bots respect, and any basic-auth or IP allowlist on a staging domain. Preview bots must be allowed through explicitly — they are unavoidably indistinguishable from the bots you are trying to stop.

There is a second reason to care about this beyond social sharing: the crawlers that build AI answers behave the same way, and blocking them removes you from a discovery channel you probably want. I went through that trade-off in optimising for AI answer engines.

Client-Side Rendering Means No Tags At All

Open your page, view source — the actual source, not the inspector's rendered DOM — and search for og:title. The inspector shows the DOM after JavaScript has run. The crawler sees the source.

If the tags are missing from the source, the fix is server-side rendering, static prerendering for the routes that get shared, or an edge worker that injects the tags. There is no client-side workaround: no amount of correctness in a useEffect reaches a client that never executes it.

This trips up single-page applications most often, but I have also seen it from caching layers that serve a shell for anonymous users and hydrate the real content afterwards. Same result, same diagnosis: the source is the contract.

Image Rules That Fail Silently

Once the tags are confirmed present, the image is the next suspect, and platforms are unhelpfully quiet when they reject one.

• Format. Use JPEG or PNG. LinkedIn wants exactly that; WebP, AVIF, SVG and animated GIFs are rejected or inconsistently supported elsewhere. If your build pipeline rewrites images to modern formats, exempt the OG image from it. This is the one place where the minification and asset-optimisation reflex actively costs you.
• Size. 1200×630 is the safe default; keep it above 1200×627 for LinkedIn and above 300×157 for the large X card. Below the minimum, platforms silently downgrade to a small square card or drop the image.
• Absolute HTTPS URL. /img/og.jpg is ignored everywhere. So is a mixed-content http:// URL on an HTTPS page.
• Publicly fetchable. No signed URLs, no cookie requirement, no hotlink protection, no Vary: User-Agent weirdness on the image path. Test the image URL with the same curl -A trick you used on the page.
• File size. Stay under about 5 MB; some crawlers stop reading before then.

A useful sanity check before publishing is to look at the card as each platform composes it, which is what the Social Media Preview tool renders, alongside the Meta Tag Analyser for confirming the tags are all present and well-formed.

When It Is Wrong on Exactly One Platform

If the crawler test passes and one platform still shows something stale or odd, you are looking at a cache or a platform quirk, not a bug in your page. Two things worth knowing, because they are the ones that waste the most time:

X retired its Card Validator in 2022 and never replaced it. The only way to preview a card now is to paste the URL into the compose window and look at what appears — you do not have to post it. Any third-party "X card validator" is inferring the result from your tags, not from X.

LinkedIn's Post Inspector refreshes the cache for future shares only. A post you have already published keeps its original preview permanently. If a launch announcement went out with the wrong image, editing the page will not fix that post; you delete and repost, or you live with it.

The refresh routes for each platform are tabulated in the Open Graph guide; the point here is diagnostic. Wrong on every platform means your page. Wrong on one means their cache.

Check It Before the First Share

The asymmetry that makes this expensive is simple: the first share is what gets cached, and the cache outlives your patience. Get into the habit of running the curl -A check as the last step of publishing, before the link goes anywhere — status code, content type, og: tags present in source, image URL fetchable with a bot user agent. Sixty seconds of that beats three weeks of grey boxes on LinkedIn, which is precisely what my WebP change cost me.