Ever wondered why some links you post to Mastodon show a nicely formatted preview card, but others don’t? It can seem random, even showing differently from app to app.

Rich interpretations of links (audio, video, and other media) is one of the few things I miss from more mainstream social media feeds.

I’ve been examining how Mastodon handles the generation of preview cards as I continue work on a little side project, Mastodon Starter Kits.

Tho I still find some things cloaked in Mastodon mysteries, it’s enough to share and hopefully the community can help fill in the blanks. 💪🏼

Post Summary

Mastodon relies on the following to generate preview cards:

The linked page must have a valid oEmbed link that isn’t identified as a “rich” type -OR-
The linked page must have valid OpenGraph meta data
Even if these two fail, if a “twitter:player” meta tag exists, it will be rendered with a “video” type regardless of the oEmbed values.
The image attached to the link must be less than 2 MB in size (the default config from Mastodon)
If none of these are true, you just get a plain old link.
If something goes wrong during the preview card generation process on the server, you just get a plain old link.
Even when ONE Mastodon instance generates the preview card, it could still fail on a federated instance that is storing your post.
By default, the cached preview card won’t be refreshed for 14 days (in docs, but not confirmed).
Even when Mastodon generates rich embed for media, it’s up to each individual app to determine if they are used. Most apps other than the official Mastodon app don’t show them. Your best bet for viewing them is the official Mastodon apps or website.

If the ever chaotic Twitter decides to strip the twitter:player functionality from their platform, it would definitely affect Mastodon, as this is quite often used as the fallback for oEmbeds views.

If you’re interested in seeing how things appear in your own app, search for the user “@[email protected]” from your instance and look at the timeline for plenty of preview card content. This is an account I created to use for testing out the starter kit I mentioned earlier.

Want to test links before posting to see if they have the appropriate data to generate preview cards? iFramely provides a great tool for that. I also use Opengraph.xyz.

Alright - for those that want to get into the details, continue on.

The Details

You might notice that immediately after you submit your post to a server, only the link appears. Then, after a few minutes, the richer preview card displays. This is because the process that creates the cards are queued to run AFTER your post is submitted. Depending on how busy the server is, or if the card has already been cached recently, it could take a few minutes to generate.

Mastodon isn’t a centralized social media service. So when you submit your new post, it gets distributed across other instances that know about you (someone on another instance follows you, boosts you, etc.). So while you may see your post and preview card get generated locally, it will take awhile to get pushed to other instances and THEN it has to get picked up in THEIR queues for processing the preview card requests.

The PreviewCard data is not part of the payload that goes to these other servers as it’s not part of the ActivityPub protocol - it’s regenerated on each instance per that specific instance’s configuration which may vary.

A SideKiq queue is pinged when a new post is created, or an existing one updated. I believe it’s also kicked off if someone boosts a post for the first time from another server. When picked up, it gets analyzed to determine if the first link in the status has any oEmbeds or Open Graph Card identifiers. If it does, that information is included and a preview card is created.

The Magic Sauce 🍜

If the link isn’t found in the existing local cache, or more than two weeks old, or the cached version doesn’t include an image, it begins looking for oEmbed links on the submitted page

This means that if an OpenGraph representation of a preview card doesn’t have an image, it’s going to just keep trying to pull that link data over and over again as people display it on their timeline..right? Hmmm…keep this in mind as we continue.

If the oEmbed link is a “rich” type, it’s ignored. The Mastodon code calls this out officially as “No No” due to security risks.

I’m curious about this decision. As far as I can see, all of the eEmbed link types include scripts in the returned HTML (example, PeerTube returned as video type), so how is this rich type any more of a risk? Opening this up would return a lot more oEmbeds, including Spotify.

If the oEmbed fails or is not found, a search for OpenGraph meta data occurs.

Theoretically, this creates the usual image + description card data that we are all familiar with.

But - many times I have found that isn’t the case!

Example: FunkWhale returns “rich” type oEmbeds from a public call, so they should be ignored, but they are actually stored as “video” type oEmbeds on Mastodon.

After some digging, I found the logic associated to this. So even if your oEmbed fails because it’s of “rich” type, there’s an ADDITIONAL check. Mastodon looks for a meta tag called “twitter:player” on the page. If it exists, a fabricated oEmbed is generated with a “type:video” attribute instead of rich. Twitter-player requires a specific format outlined on the developer platform.

Wonder how long twitter-player will continue to work? It’s used quite extensively across audio and video sharing sites, it seems. And for Mastodon, it seems to be used quite often as the fallback right now.

This is why Spotify doesn’t work, but SoundCloud does. Spotify doesn’t have the twitter:player attribute and the oEmbed is of type “rich” so that is ignored as well.

If none of the meta data mentioned above is found, you just have your undecorated, sad little link. 🥱

Otherwise, you get a nice representation as a preview card or embedded object! Depending on what the service logic was able to identify, the Preview Card could look like something in this collection.

)

Mastodon does all this work to create the Preview Card and/or oEmbed HTML for you, stored in the status.card, and status.card.html properties.

But…sometimes my links to the same site, different video or song, don’t end up showing an embed. Why?

More Instances, More Problems

So remember I told you each instance has their own SideKiq worker to handle these requests coming in?

Well, yeah. I’ve noticed that each instance seems to end up handling the creation of preview cards differently. One server will create the OpenCard representation, the next will just show the link.

Inconsistent Results - Why?

Maybe all the incoming requests for preview card data causes the domain to stop accepting requests, and then the worker falls back to just displaying the link. Or maybe each instance has it’s own rules about which domains to allow creation of preview cards. Or maybe the SideKiq queue just fails for some reason.

Here’s a good example. NPR doesn’t currently display preview cards from Mastodon instance. Why? Because the requests coming from the preview card generation service sends a user-agent string that is not allowed by the NPR servers. The service includes https://example.com/ in the user-agent string, which seems to cause the failure. Taking it out allows the request. 🤷🏽‍♀️

User-Agent: http.rb/5.1.1 (Mastodon/4.1.2; +https://{instance.url}/) Bot

Remember I said the Mastodon PreviewCard service keeps trying over and over again if an image isn’t found.

This is a particular instance where an OpenGraph representation was found, but no image was included. In these cases, Mastodon is going ahead and creating a preview card. But, whenever this link is boosted or mentioned in another post, it’s going to reach out again to see if an image exists again, circumventing the cache.

Welp…yeah. I think it’s just going to keep hitting that server like a DDOS attack for awhile - I’m not sure how or when the Mastodon service finally gives up on a link and just lets it be a link.

There IS a cache on the oEmbed service which defaults to 24 hours, so perhaps that alleviates the issue somewhat. But if an admin had to restart that service…wouldn’t it try picking up all those links again as people pull the posts down?

I’m not seeing the same caching declaration on the OpenGraph service.

NEW INFO Recently ran across this CLI option to remove preview cards. It states that once a preview card has been added for a specific URL, it’s not going to be refreshed for 14 days. I haven’t thoroughly tested if this is true or not. I know a few people that have had similar issues.

A workaround would be to slightly alter the URL in some way and then re-submit the link in a post.

How can these issues be resolved? Perhaps Mastodon should reach out to larger organizations and ask them to whitelist the user-agent string. But where does that end. Or maybe Mastodon should re-architect the preview card service itself to better manage the requests.

Because of the federated way posts are shared between instances, a highly followed account like Linux Torvalds could cause a lot of requests from tons of servers, all at the same time. I wouldn’t blame some of these sites for blocking the requests with a Mastodon user agent!

I had my own experience with this when I started sharing FunkWhale links. I brought down a home server for a few hours. 😬

Another option I heard discussed in the official Discord chat was pushing all requests to a centralized service, like Embedly, but not embedly of course - something open source. But that would be a huge undertaking, require cooperation and usage by a majority of servers to be of use, and I’m not sure how well it would work with the federated nature of Mastodon.

It’s all for nothing, mostly 😭

But, guess what? Most third party apps ignore these embeds, and just display the common picture + metadata text description preview card instead. Why? 🤯

Mastodon’s official web app and native apps do a good job of showing how these embeds can be displayed. But so far, not many others are following suit. I only found one known third party app so far, Cuckoo.social, to have them.

There are good reasons, but I’m still disappointed that more apps don’t do the additional work to include the oEmbeds. From personal experience, I do see that it takes a good amount of effort, even in a simple web app.

A great discussion thread led me to the considerations below as being the most probable reasons for exclusion.

I would love to hear more from the official app developers on this topic!

Unpredicatbility of Embeded HTML Code

The oEmbed returns HTML code. It can include CSS, Javascript, just about anything. Most sites will do their very best to return valid, safe oEmbed data, but you can’t trust that EVERY site will do that. So your web app could end up looking odd, having the oEmbed affecting the display or even functionality of your app! Not good.

Limitations of the technology stack

If you’re running a native app (MacOS, iOS, Android, Windows, etc.) - there may be limitations on how you can represent the HTML / Script returned for the oEmbeds. Because you have little control over what content gets pulled down from those oEmbeds, I suspect it may be a blocker for some of the pickier App Store reviewers as well.

I have never created a native app so this is all conjecture - my question is, if this is true - how do mainstream social sites handle it, then? I think that’s why we see twitter and facebook specific open graph tags? You are forced to present a specific format to them to be visible in their feeds.

Accessibility Issues

How do you represent this external object in a way that is recognizable and functional to those that utilize screen readers, for example? How do you create an understanding of how to interact with it? Most of that is left up to the creator of the oEmbed (Spotify, YouTube, etc.) and you have little control over it.

Unapproved Tracking from oEmbeds

Depending on how the oEmbed is presented on the timeline, it can including tracking analytics. Let’s say I post a YouTube link. My little Starter Kit web app isn’t very complex, so it just displays the direct oEmbed to YouTube. It’s possible that data you weren’t expecting to be shared (IP address for example) gets sent to YouTube for tracking on EVERY YouTube post you see on your timeline - even if you don’t interact with it!

Note that the official Mastodon app handles this gracefully. The oEmbeds are NOT loaded by default. Instead, you get an image representation with a “play” button. Once you click on the image, THEN the oEmbed is loaded, so at least the user has to interact with it, showing interest.

An interesting solution to this issue can be found in use at Cohost. It uses a service from iframely.net, which allows the end user to consent to specific embeds they want to allow. Are you comfortable with Instagram, but not TikTok? There’s a way for YOU to configure that for your self. Nice.

The preview option you get in Cohost when posting something new is top notch. It allows you to immediately see if the link you’re including will work as expected.

For Mastodon, similar functionality would require the app to include all of the logic we just walked through on the client side to make a best guess at what the representation will look like. But even then, most apps will ignore it anyway and just display the simple preview card. 😩

Conclusion

Preview Cards are unreliable right now. You can’t be assured they will work, even if your link has all the right pieces and follows the rules Mastodon sets up for inclusion.

I think it’s worth a discussion within the community on why these inconsistencies exist and how it can be improved. The inclusion of oEmbeds specifically, which allow you to stay present in the app without having to jump around to popups or external browsers.

References

https://mastodon.social/@box464/110194668526931039
https://docs.joinmastodon.org/entities/PreviewCard/
https://docs.joinmastodon.org/admin/tootctl/#preview_cards-remove
https://github.com/mastodon/mastodon/blob/main/app/services/fetch_link_card_service.rb
https://github.com/mastodon/mastodon/blob/main/app/services/fetch_oembed_service.rb
https://github.com/search?q=repo%3Amastodon%2Fmastodon%20preview_card&type=code
https://github.com/mastodon/mastodon/blob/main/app/controllers/api/oembed_controller.rb
https://github.com/mastodon/mastodon/blob/main/app/workers/link_crawl_worker.rb#L3
https://mastodon.social/@box464/110194668526931039

https://oembed.com/
https://ogp.me/
http://debug.iframely.com/
https://www.opengraph.xyz/

https://github.com/mastodon/mastodon/blob/main/spec/views/statuses/show.html.haml_spec.rb
https://github.com/mastodon/mastodon/blob/main/app/lib/link_details_extractor.rb
https://developer.twitter.com/en/docs/twitter-for-websites/cards/overview/player-card

Special Thanks

Appreciate the insights I received from @[email protected], which helped me focus in on some topics, especially around accessibility and security. And, now you know why Spotify links don’t show an embed! 🎶