julian (@julian.community.nodebb.org.ap.brid.gy)

Backfilling Conversations: Two Major Approaches In February 2025, I presented a topic at FOSDEM in Brussels entitled <a href="https://spectra.video/w/xwCSYfZh1mJY64zJ9GngbE" rel="nofollow ugc">The Fediverse is Quiet — Let's Fix That!</a> In it, I outlined several "hard problems" endemic to the fediverse, focusing on one particular complaint that is often voiced by newcomers and oldtimers alike; that the fediverse is quiet because you don't ever see the full conversation due to some design considerations made at the protocol level. Since then there have been a number of approaches toward solving this problem, and it is worth spending the time to review the two main approaches and their pros and cons. N.B. I have a conflict of interest in this subject as I am a proponent of one of the approaches (FEP 7888/f228) outlined below. This article should be considered an opinion piece. <hr /> <h2>Crawling of the reply tree</h2> First discussed 15 April 2024 and merged into Mastodon core on 12 Mar 2025, <a href="https://neuromatch.social/@jonny">@<bdi>[email protected]</bdi></a> pioneered this approach to "fetch all replies" by crawling the entirety of the reply tree. When presented with an object, the Mastodon service would make a call to the <code>context</code> endpoint, and if supported(?) would start to crawl the reply tree via the <code>replies</code> collection, generating a list of statuses to ingest. This approach is advantageous for a number of reasons, most notably that <code>inReplyTo</code> and <code>replies</code> are properties that are ubiquitous among nearly all implementations and their usage tends not to differ markedly from one another. N.B. I am not certain whether the service would crawl up the <code>inReplyTo</code> chain first, before expanding downwards, or whether <code>context</code> is set in intermediate and leaf nodes that point to the root-level object. One disadvantage is this approach's susceptibility to network fragility. If a single node in the reply tree is temporarily or permanently inaccessible, then every branch of the reply tree emanating from that node is inaccessible as well. Another disadvantage is the reliance on intermediate nodes for indexing the reply tree. The amount of work (CPU time, network requests, etc.) scales linearly with the size of the reply tree, and more importantly discoverability of new branches of the reply tree necessitate a re-crawl of the entire reply tree. For fast-growing trees, this may not net you a complete tree depending on when you begin crawling. Lastly, in the ideal case, a full tree crawl would net you a complete tree with all branches and leaves. Great! Mastodon is the sole implementor of this approach, although it is not proprietary or special to Mastodon by any means. <h2>FEP 7888/f228, or FEP 171b/f228</h2> Summarized by <a href="https://mitra.social/users/silverpill">@<bdi>[email protected]</bdi></a> in <a href="https://community.nodebb.org//w3id.org/fep/f228" rel="nofollow ugc">FEP f228</a> (as an extension of FEPs <a href="https://community.nodebb.org//w3id.org/fep/7888" rel="nofollow ugc">7888</a> by <a href="https://mastodon.social/@trwnh">@<bdi>[email protected]</bdi></a> and <a href="https://community.nodebb.org//w3id.org/fep/171b" rel="nofollow ugc">171b</a> by <a href="https://fediversity.site/channel/mikedev">@<bdi>[email protected]</bdi></a>), this conversational backfill approach defines the concept of a "context owner" as referenced by compatible nodes in the tree. This context owner returns an <code>OrderedCollection</code> containing all members of the context. A major advantage of this approach centers around the pseudo-centralization provided by the context owner. This "single source of truth" maintains the index of objects (or activities) and supplies their IDs (or signed full activities) on request. Individual implementations then retrieve the objects (or activities). It is important to note that should the context owner become inaccessible, then backfill is no longer possible to achieve. On the other hand, a dead or unresponsive intermediate node will not affect the ability of the downstream nodes to be processed. The context owner is only able to respond with a list of objects/activities that it knows about. This does mean that downstream branches that do not propagate upwards back to the root will not be known to the context owner. Additionally, consumers are also able to query the context owner for an index without needing to crawl the entire reply tree. The ability to de-duplicate objects at this level reduces the overall number of network requests (and CPU time from parsing retrieved objects) required, making this approach relatively more efficient. Additional synchronization methods (via id hashsums) could be leveraged to reduce the number of network calls further. A number of implementors follow this approach to backfill, including NodeBB, Discourse, WordPress, Frequency, Mitra, and Streams. Additional implementors like Lemmy and Piefed have expressed interest. One technical hurdle with this approach is technical buy-in from implementors themselves. Unlike crawling a reply tree, this approach only works when the context owner supports it, and thus should be combined with various other backfill strategies as part of an overall conversational backfill solution. <h2>Conclusion</h2> 2025 is shaping up to be an exciting year for resolving some of the harder technical and social problems endemic to the open social web/fediverse. It is this author's opinion that we may be able to make good headway towards resolving the "quiet fedi" problem with these two approaches. It is important to note that neither approach conflicts with the other. Implementations are free to utilise multiple approaches to backfill a conversation. Both methods presented here have pros and cons, and a combination of both (or more) could be key. Feel free to use this as a starting point for discussions regarding either approach. Does one speak to you more than the other? Are the cons of either approach significant enough for you to disregard it? What other approaches or changes could you recommend? https://community.nodebb.org/post/104809