Just released a small update to 23.3 of the @screamingfrog.co.uk SEO Spider - www.screamingfrog.co.uk/blog/seo-spi.... Includes an update to issue 'HTML Document Over 2MB' with Google's update on file size limits, alongside small improvements and bug fixes.

Little tip - Under ‘Config > Spider > Limits’ you can limit the number of URLs crawled per subdomain. This little-known config can be useful when performing research, such as crawling a sample number of URLs per subdomain in list mode.

Little tip - If you’re using vector embeddings for redirect mapping, you can add a rule to define the URL patterns used for semantic similarity analysis via ‘Config > Content > Embeddings', such as from a staging to a live website. More here - www.screamingfrog.co.uk/seo-spider/t...

Little tip - If you don't want to keep crawls you can adjust the crawl retention settings via ‘File > Settings > Crawl Retention’ to auto delete after a period of time. You can also prevent projects and crawls from being cleaned up via right click 'Lock' in 'File > Crawls'.

Hat on

Just released a small update to 23.1 of the @screamingfrog.co.uk SEO Spider - www.screamingfrog.co.uk/blog/seo-spi.... Mostly small improvements, and bug fixes to the latest release.

Little tip - You can right click and 'Show Inlinks', 'Show Inlinks to Children', or perform the same for 'Outlinks' to see all inlink and outlink relationships in our site visualisations.

The @brightonseo.com Crawling Clinic October ’25 - screamingfrog.co.uk/blog/brighto... Come and chat about any crawling issues you're experiencing, or grab some new SF swag!

Just released @screamingfrog.co.uk SEO Spider v.23: screamingfrog.co.uk/blog/seo-spi... Includes - 💡Lighthouse & PSI Updated to Insight Audits ⚙️Crawl Retention Config 🧑‍🤝‍🧑Semantic Similarity Embedding Rules 📊Display All Links in Visualisations 🍇Links in Content Cluster Diagram & more!

🌿 Less than a month to go until our next GreenSEO event! Join us in Brighton on Thursday 22nd Oct for an evening of sessions on SEO and the environment. 🎟️ Grab your free ticket here: greenseo.org/brightonseo Part of the @brightonseo.com Fringe. Sponsored by Climb Trees and @screamingfrog.co.uk.

Little tip - You can fetch API data without any crawling. Just go to 'Mode > APIs', select your APIs such as GA4, GSC, PSI or link metrics - and upload the list of URLs to pull data quickly.

My first public “vibe-coded” idea-to-product: storeratings.co Building this was fun but definitely tested my patience. A combination of Vercel, GitHub, and Cursor, I got there in the end. I am not a developer or designer, go easy.

So long, and thanks for all the fishes, uhm, memories. I'll miss you, Mr Wiseguy. RIP @alanbleiweiss.bsky.social

Little tip - You can run prompts against specific segments. You can also segment against issues. So you can easily generate alt text only for images missing alt text for example.

Finally getting around to playing with @screamingfrog.co.uk's LLM API integration. My goodness, that's cool.

I need another month off football but here's the FPL link for the SEO league. @screamingfrog.co.uk Here you go! Auto-join: fantasy.premierleague.com/leagues/auto... code: 0xxnvz

AI search is booming, but SEO is still not dead - searchengineland.com/ai-search-bo... from @lilyray.nyc

Little tip - You can customise the OpenAI endpoint to use LLMs that are compatible with the OpenAI API format. For example, this means you can use Grok or DeepSeek by updating the endpoint, and API key.

How To Use Vector Embeddings for Redirect Mapping - www.screamingfrog.co.uk/seo-spider/t...

June 2025 core update - Released the June 2025 core update . The rollout may take up to 3 weeks to complete. (source) #SEO

How to Interpret the Content Clustering Visualisation in @screamingfrog.co.uk #seo seogreetings.com/blogs/news/h...

Noindex No Longer Means Not Rendered - tamethebots.com/blog-n-bits/...

New tutorial available on 'How To Crawl With AI Prompts' - screamingfrog.co.uk/seo-spider/t... Includes - - How to set up and crawl with prompts - Common use cases - Running prompts for specific segments & issues - Custom endpoints & more! Enjoy.

New tutorial on 'How to Identify Semantically Similar Pages & Outliers' - www.screamingfrog.co.uk/seo-spider/t...

Just released @screamingfrog.co.uk SEO Spider v.22: www.screamingfrog.co.uk/blog/seo-spi... Includes - 🔍Semantic Similarity & Outliers 📊Content Cluster Visualisation 🐸 Semantic Search 🤖 AI Integration Improvements 🦾 Column Config, Custom Export, Export To Tabs! And lots more. Enjoy!

Readability of Spanish Texts: Alternative Metrics to ‘Flesch Reading Ease’ - www.screamingfrog.co.uk/blog/readabi... from @mjcachon.bsky.social

How AI Mode Works and How SEO Can Prepare for the Future of Search - ipullrank.com/how-ai-mode-...

Top ways to ensure your content performs well in Google's AI experiences on Search - developers.google.com/search/blog/...

How Long Does It Take to Rank in Google? And How Old Are Top Ranking Pages? - ahrefs.com/blog/how-lon...

Finding and Testing Internal Link Changes with Screaming Frog - www.screamingfrog.co.uk/blog/finding... from @ryanjonesseo.bsky.social

Fresh Doc Dump: PageRank, Navboost, and RankEmbed: DOJ Papers Outline Google’s Stack - www.searchengineworld.com/fresh-doc-du...

Google's AI Adventure - One Year On - www.seoforgooglenews.com/p/googles-ai...

New Study: The 130 Day Indexing Rule - indexinginsight.substack.com/p/new-study-...

11 Lessons Learned From Auditing Over 500 Websites - www.searchenginejournal.com/lessons-lear... from @olgazarr.bsky.social

Little tip - Learning how to debug why a page might be missing in a crawl is a valuable SEO skillset. Common reasons include - Not being linked to internally, JS, uncrawlable pagination, uncrawlable link types, search forms etc. Check out our tutorial - www.screamingfrog.co.uk/seo-spider/t...

Little tip - There are 'outside <head>' filters for key elements such as titles, canonicals, hreflang etc in their tabs. However, often the cause is an invalid element in the <head>, which closes it early. Check the Validation tab to see if this has been triggered when debugging.

Google and the 130-Day Rule : Indexation strategy for already indexed pages - alekseo.com/google-and-t...

The Screaming Frog @brightonseo.com beer mats -

🚨 New Data Study for SEOs: How Often Is Google Really Rewriting Your Titles? And Why? In 2021, Cyrus S. reported that Google rewrote 61% of page titles. That number has now climbed to 76%. 😳 Find out why! www.johnmcalpin.com/seo-data-stu...

The @brightonseo.com Crawling Clinic April ’25 - www.screamingfrog.co.uk/blog/brighto...

What even are log files & bots? - sallymills.com/what-even-ar... from @sallymills.bsky.social

How to Find Internal Linking Opportunities Using Screaming Frog’s N-gram Analysis - gofishdigital.com/blog/n-gram-...

Little tip - You can compare crawls of staging to a live website by using 'URL Mapping' in crawl comparison. Equivalent URLs are then compared against each other to identify changes and issues between the crawls. (More in out tutorial here - www.screamingfrog.co.uk/seo-spider/t...)

5 Early Observations about The New Google AI Mode in Test - www.aleydasolis.com/en/search-en... from @aleyda.bsky.social

Keyword cannibalization can create many issues. Fortunately, Ellie Brett explains how to use N-Grams in Screaming Frog’s SEO Spider to solve these issues, in our latest #WTSKnowledge article Thanks to our WTSKnowledge cohort sponsor: @zipsprout.bsky.social www.womenintechseo.com/knowledge/so...

A new @screamingfrog.co.uk hat

Disallow, Canonicalize, Noindex, or Remove? How to properly block content that’s violating Google’s site reputation abuse policy - www.gsqi.com/marketing-bl... from @glenngabe.bsky.social

Our team are exhibiting @womenintechseo.bsky.social WTSFest London this week. If you're attending, come and say hello, ask us any burning crawling queries, or pick up some 😱🐸 swag!

Generate JSON-LD Schema at Scale With JavaScript Snippets - www.screamingfrog.co.uk/generate-jso...

If you're using @alsoasked.com and @screamingfrog.co.uk to automatically find content gaps on your site (YOU DO THAT right? searchengineland.com/improving-co...) Updated CustomJS to make it more reliable: gist.github.com/thetafferboy...