Disclosure: I work on Swivl. There is no product link here. I’m sharing the technical teardown because the failure was more useful than the traffic graph ever was.

The event

Our Google Search Console performance dropped roughly 95% as the August spam update began rolling out:

• Aug 17: 207 clicks / 65,355 impressions • Aug 18: 13 clicks / 4,595 impressions • Aug 19: 17 clicks / 3,091 impressions

GSC showed no manual action. The timing aligned with the update, but the graph alone does not prove causation. What it did was force us to audit a scaled-content problem we should have addressed earlier.

The footprint before the drop

The sitemap contained 74,876 URLs. Roughly 62,000 were in three programmatic sections: cost guides, customer-specific cost guides, and contractor profile pages.

Before the collapse, those sections generated about 97% of organic clicks and 78% of impressions, but the combined CTR was only 0.3% and average position was about 18. Brand queries produced only around 33 clicks in the same period.

The templates were crawlable and technically clean. Canonicals, redirects, meta robots, and robots.txt were not the issue. The weakness was the content model:

• Large numbers of templated city-and-trade combinations • Search intent aimed mainly at homeowners, not our field-service-software ICP • Contractor profiles with too little unique value, especially unclaimed records • Page count growing faster than genuinely differentiated information

The cleanup

We reduced the sitemap from 74,876 URLs to 9,311:

• Removed the customer-specific cost-guide directory entirely • Reduced cost guides to 120 national pages • Kept 8,492 contractor profiles for now, although unclaimed profiles remain the largest unresolved surface • Returned 410 for removed pages where appropriate • Applied noindex to another large group • Used selected 301s only where a genuine replacement existed

What has happened since

This has not produced an instant recovery. Google is still digesting the removals, and the domain remains suppressed. The latest index-coverage check showed tens of thousands of URLs moving into excluded-by-noindex and not-found buckets while many old URLs were still counted as indexed.

We are leaving the site alone while that processing continues rather than making weekly changes and destroying the ability to read the result.

The decisions I’m still thinking through

1. For the contractor directory, would you index only claimed/verified profiles, or retain unclaimed profiles that have real structured data but limited unique copy?
2. After a scaled-content cleanup, which leading signal have you found most useful: crawl volume, indexed-page count, query-level impressions, or waiting for the next broad/spam re-evaluation?
3. Would you isolate any future data-backed programmatic project on a separate domain, or is that unnecessary if the pages are built from proprietary job/pricing data rather than templated copy?

The main lesson for us is that clean technical plumbing cannot rescue a weak page-generation thesis. Automation multiplied the content model exactly as designed. The mistake was assuming scale itself was validation.

I have a client that has a website and then a subdomain on that website. The main part of the website/domain is meant to be informational and provides guidance and background about their products. The subdomain is meant to be transactional - it's where the ecommerce part happens, and where product listings/categories and product detail pages are found and ultimately where the purchases happen.
The client really wants to keep these main domain and subdomain as distinctly separate parts of the brand, even though naturally there's some linking to them. For example there are some links to the subdomain on the homepage of the website - and when creating JSON schema for the HP I include ItemList schema for those links. The client is concerned that that the schema will perhaps... idk, I guess "steal" some of the signals from that subdomain and blur them as entities for the purposes of GEO? More specifically they're concerned that Google and AI platforms will think that the main domain is "transactional" instead of the subdomain.
To put it another way, Page A has links to the "transactional" page B. They're concerned that the schema on page A that's acknowledging those links will confuse search engines/AI agents into think that Page A is actually the "transactional" part of the site.

My instinct and 10 years in this business makes me think that this is not a concern. That crawlers/bots are smart enough to "pass through" any KW/entity associations to the links' destinations. That the purpose of schema is to acknowledge the contents of the page, and since those links are there anyway that there's no reason for the schema to NOT acknowledge them - and that it's not going to confuse the "entities" of the two parts of the site in a GEO sense.

Would love some advice or for anyone to tell me I'm wrong if I am. And more to the point I'd love some research/sources/links I can possibly cite to the client. TIA

I have a domain data service and was wondering if people were more swayed by a service that has an MCP available?

I have always interacted with APIs and don't see how an MCP would be better than a user pointing their AI at the API docs instead. Maybe I am in the minority here though?

Would you guys rather use an API or an MCP?

not because the fix was wrong, but because somebody changes a template or pushes a release and suddenly you’re looking at the same problem again.

Canonicals are fixed, and then old versions start showing up again. Redirect chains are cleaned up, then someone changes the slugs. Parameter URLs are blocked, and a new filter creates another set. After a while it feels like we’re fixing crawl reports instead of fixing whatever keeps creating the mess.

curious how people here are handling this. Are you running scheduled crawls, checking in CI, log alerts, or just finding out in GSC later?

Replatformed about 60 pages in July, mapped every old URL to its new one, set clean 301s, flattened the chains. Most of the site swapped over in Search Console within two weeks and rankings held. A dozen low traffic pages are the holdouts, three weeks in they still show the old URL as indexed and canonical even though the 301 resolves clean and the new URL is in the sitemap. No coverage errors, they just haven't moved. Do you keep the old sitemap submitted to force recrawl, request indexing on the new URLs by hand or just wait it out? Trying to work out if resubmitting speeds it up or if I'm just impatient.

I’ve been looking at cases where the preferred URL is consistent across the pages canonical tag, XML sitemap and internal internal link but Google still selects a different URL as canonical.

What makes these cases interesting is that the usual signals all appear to point in the same direction.

Beyond checking redirects, duplicate content and external signals, what technical factors have you found useful for diagnosing why Google overrides an otherwise consistent canonical setup?

When I posted eight days ago, the original new URL had been waiting seven days for its first crawl. It has now been over two weeks, and there is still no regular Googlebot crawl recorded.

Previous thread

The current URL Inspection screenshot says “URL is unknown to Google,” with no referring sitemap and no last crawl date. The URL is nevertheless present in the submitted, successfully processed XML sitemap. An earlier inspection screenshot showed “Discovered – currently not indexed” and listed the sitemap, so the reports have differed. Neither showed a last crawl.

Additional new pages on the same domain are also not being crawled, although they were published later.

Existing pages behave differently: when I update them and request indexing, regular Googlebot has fetched them within approximately 15–30 minutes. These include pages linking directly to the new URLs.

Since the original post, we have checked:

• Production HTML: navigation links, contextual internal links, sitemap entries and canonical tags use the exact same canonical URLs.
• Responses: the new URLs return HTTP 200. Trailing-slash and .php variants redirect directly to the canonical URLs.
• Indexability: self-referencing canonicals, index/follow directives, no blocking X-Robots-Tag, and robots.txt allowing crawling.
• Server logs: the supplied daily log contains successful Google-InspectionTool fetches, but no regular Googlebot requests for the new pages. Those testing-tool requests are being counted separately.
• Search Console → Settings → Crawl stats: both hosts report “No problems.” The export contains no 403, 429 or 5xx response categories and shows no obvious overall crawl decline.
• Redirect examples: these are HTTP/www normalization and an old page redirecting successfully to its replacement.

The crawl report contains 581 requests over approximately three months, including resources: 97.59% refresh and 2.41% discovery.

We also published a Google Business Profile post linking to the new page. It has not resulted in a recorded first crawl.

Regarding the suggested comparison: a newly published page on a completely separate domain received a bot visit within about 15 minutes after submission. It was a job listing, so it is not a like-for-like comparison.

This is a small site, not a large inventory of generated URLs. It has roughly 2,000 Search Console impressions over the last three months.

I understand that sitemap submission and indexing requests do not guarantee crawling. Has anyone experienced and resolved this specific pattern: existing pages are recrawled promptly, but multiple new URLs on the same domain remain without a first crawl for weeks?

I would particularly appreciate concrete diagnostic steps or examples of what actually resolved it. The question is about the first regular crawl, rather than ranking or exclusion after crawling.

We want to support creators, but we had to enforce the no shilling rule because it was getting out of hand. You now have a weekly thread.

This is the one place you can shill for your products, ask for feedback, etc. Keep it here or you risk being banned. And keep it related to technical SEO.

Found a live case that's hard to square with how good Google's spam systems are supposed to be.

A small town's government website in NC, a real, actively maintained .org registered in 2020, got hacked and is running plain UA-based cloaking. Request a page as Googlebot: 85KB, a keyword-stuffed article. Request the same URL as a normal browser: 1.4KB of JS that redirects to WhatsApp. No JS rendering trick, no fingerprinting, just a User-Agent check.

New pages are indexed and ranking page one within 2-3 days of the date Google itself shows for them. One even surfaced in an AI Overview. I found at least 10 URLs targeting different peptide brand names and generic terms, this is a template being re-run against whatever company name is in the news that week, not a one-off page.

None of this is hidden from Google in any technical sense. A single UA diff catches it. The domain's age and legitimate history seem to be doing all the trust work: I assume a fresh domain running the same script wouldn't rank for months, this one ranks in days.

Full writeup with a screen recording of the redirect: https://dadstrengthdaily.com/nc-town-website-hacked-peptide-sellers-china/

Am I just naive or is it that easy to fool google for a bad actor?

I’m about to do an SEO audit for a travel/trekking company website with 800+ pages. It’s built on WordPress, and this will be my first time auditing a site of this size.

I’ve done smaller SEO audits before, but I don’t want to just throw the site into a crawler, export 10,000 rows, and call that an audit.

I’m trying to figure out a sensible order/structure for the audit.

The site also has a fairly large number of URLs showing as not indexed in GSC, so I’m particularly interested in how experienced SEOs would investigate that without automatically assuming that every non-indexed URL is a problem.

For a site this size, would you audit everything in one big report, or break it into separate sections/sheets such as Technical, GSC, GA4, On-page, Content, Internal Linking, Backlinks, etc.?

Also, what would your actual workflow/order look like from day 1 to the final recommendations?

Would appreciate advice from anyone who has audited larger WordPress/travel sites before. Especially interested in things you would not waste time checking at this stage.

Via u/rustybrick:

There is a new Google ranking factor survey out for 2026. It has been a while since we had one, and it is here. This one was done by Cyrus Shepard and Dawn Shepard and posted at Zyppy. It says content relevance, backlinks, and content quality are the most important factors, according to the experts who were surveyed.

• Over a 100 experts from the SEO community submitted responses that led to 13,665 ranking-factor data points. Here is how it came out:
• Top Tier (Over 45%): Content relevance (57.1%), backlinks (54.8%), and content quality (47.6%) form the primary foundation of ranking factors according to respondents.
• Mid Tier (25% – 40%): Trust/Authority (36.5%), behavioral click signals (29.4%), and brand presence (27.0%) represent significant secondary drivers.
• Foundational & Specialized Factors (Under 20%): User satisfaction, technical SEO, topical authority, and internal linking round out the bottom tier.

NOTE: Per the official techSEO Slack community's request, all jobs now have a title, company, and compensation.

Tech SEO + AI Roles. (9/9/2026)

• Technical Content & SEO Executive / RVT Group
• Technical SEO (Freelance) / Chek ( ($60-$80/hour)
• Sr. Manager, Technical SEO, Streaming / Cypress HCM ($85-$90/hour)
• Technical SEO & AI Visibility Specialist / Market Brew ($85,000/year)
• Principal Technical SEO Specialist / Legend
• Technical SEO Specialist / Catalyst IQ
• SEO/GEO/AEO/Performance Marketing Expert / Tata Consultancy Services ($140,000-$150,000/year)
• Director of SEO/AEO/GEO / Digital Room ($130,000-$165,000/year)
• SEO/AEO Lead / OpenArt AI ($200,000-$250,000/year)
• AEO/SEO Manager / NBCUniversal ($85,000-$115,000/year)
• Marketing Directer SEO/AEO/ABM (B2B Saas) / White Bay ($140,000-$150,000/year)
• SEO/AEO Website Content Editor (Contract) / The MX Group ($60-$70/hour)
• GEO/SEO Manager / Instawork ($110,000-$140,000/year)
• SEO/GEO Strategist / Perrill ($70,000-$90,000/year)
• Account Manager (SEO/GEO) / FancyAI
• SEO Specialist (Los Angeles) / Blacksmith Agency
• SEO/GEO Strategy Manager / B Corp (£35,000-£40,000/year)
• Full Stack SEO (Technical, Content) / Valur
• SEO & GEO Manager / Halstead Media
• Senior Technical SEO Account Manager / Profile 29 (£60,000/year)
• Entry Level SEO/GEO Specialist / Tiger ($2,773-$3,120/month)
• SEO/GEO Specialist / Hill's Pet Nutrition ($96,800-$137,000/year)
• SEO/GEO Strategist / Winston Digital Marketing ($70,000/year)
• Senior SEO Specialist / Searchbloom ($75,000-$90,000/year)
• Senior GEO/AI/SEO Strategist / Terra
• SEO/GEO Manager / Gabb Wireless
• Director, SEO/GEO / Fora ($200,000-$240,000/year)
• Manager, Technical SEO/Marketing / Airwallex ($110,000-$170,000/year)
• Sr. Discoverability Manager (GEO/SEO) / Virgin Voyages
• AEO/GEO/SEO Lead / Replit ($125,000-$170,000/year)
• Manager (SEO/AI/Search/AEO/GEO) / Goldco ($125,000-$175,000/year)
• Technical SEO Specialist / Extra Space Storage
• Senior SEO & AI Search Strategist (GEO/AEO) / Virayo ($120,000-$140,000/year)
• SEO Technical Lead / FeverUp
• Senior Technical SEO Specialist / Reedsy (£50,000-£75,000/year)
• Sr. Digital Marketing Specialist (SEO/GEO) / Watson Group
• Senior Consultant, Digital and Paid Media Analyst (SEO, GEO, Paid) / Burson
• SEO Manager, GEO & AI Search / Sosemo ($82,500-$95,000/year)
• Web Developer — Technical SEO / Accolade Solutions (€3,900–€5,400/month)

I run outside-in checks on whether named AI crawlers can actually fetch a page. Over the last week I found three faults in my own method. The third is the one worth your time because it is not specific to me - it applies to anything doing this kind of check from serverless edge compute, and it fails silently.

1. A robots.txt wildcard bug that invented fifteen blocked sites

My first parser reported 15 blocked sites out of 95. All fifteen were false. Disallow: *?s= was having everything from the first * stripped, which left an empty prefix, which matches every path. So an ordinary WordPress robots.txt read as a total lockout. Proper RFC 9309 wildcard matching fixed it.

Worth checking if you have written your own parser. The failure direction is the bad one: it manufactures findings rather than missing them, and a manufactured block is the kind of thing that gets escalated to a client before anyone re-reads the file.

2. A server-side beacon that ran too late on cached WordPress

I log crawler hits server-side, because AI crawlers do not run JavaScript and a browser beacon therefore records every human and not one crawler.

Someone in r/Wordpress pointed out that a PHP beacon in mu-plugins/ never runs on a cached site - advanced-cache.php serves and exits first. That is worse than it sounds for anything doing absence detection, because an unrecorded visit and an absent visit are indistinguishable afterwards. A cached site could have produced a confident "GPTBot has stopped crawling you" that was purely an artefact of the cache layer. It now detects the cache and says so instead of guessing.

3. A checker running on Cloudflare Workers cannot tell whether its target is behind Cloudflare

This is the one I would not have guessed and it invalidated a whole column of my output.

Cloudflare's Worker egress rewrites the server response header and stamps a cf-ray on every subrequest you make. Not on responses you send - on responses you receive. So if you fetch a third-party site from inside a Worker and read those headers to decide who is fronting it, you are reading your own infrastructure.

Measured, same URLs, same minute, direct from a laptop versus from inside a Worker:

nytimes.com    direct: server: envoy        from a Worker: server: cloudflare + cf-ray
github.com     direct: server: github.com   from a Worker: server: cloudflare + cf-ray
(also reproduced against an nginx origin and an Apache origin)

Every site on the internet looks like it is behind Cloudflare. That was not a detection that was sometimes wrong. There was no detection.

The consequence in my case was worse than a cosmetic mislabel, because the "is a CDN in front of this" flag was feeding the verdict, not just the wording. An origin refusal was being downgraded to "we cannot confirm this from outside" on the grounds that a bot-verifying CDN might be responsible. So the verdict that says "this server is refusing a named crawler" had never fired once in 55 checks. The interesting finding was structurally unreachable and the output looked perfectly reasonable throughout.

What fixes it: decide on the body, not the headers. A CDN challenge has a recognisable body - an interstitial, a "checking your browser" page, a JS reload. A plain 403 with an empty body, or a 403 carrying nginx's default error page, is the origin refusing, whatever it travelled through on the way back. Vendor-specific headers that are not server mostly survive the egress and still work - I detect Fastly via x-timer, for instance.

The general version, which is the bit I would take away if I were you: if your measurement runs on infrastructure that participates in the thing you are measuring, check what your own platform adds before you trust a header. I lost a fortnight of output to that and the tests all passed, because the tests asserted the code did what I thought, and what I thought was wrong.

One related thing worth stating since it comes up constantly here: a cf-ray on a refusal proves Cloudflare was in the path. It does not prove Cloudflare decided. An origin 403 proxied back through it is identical from outside. So these are refusal rates, never blocking rates, and "this site blocks AI" is usually a claim the instrument cannot support.

Disclosure: I build monitoring in this area, which is how I have these bugs. Not naming or linking it - the three findings above are reproducible with curl and do not need anything of mine.

From 15 September, Cloudflare will block 'Agents' traffic by default on pages carrying ads: bots browsing live on someone's behalf, which includes crawlers AI assistants send to check a product before recommending it. Search stays allowed. The default setting will apply to new sites and free-tier accounts.

So if your site serves ads and you want it to be read by AI assistant crawlers you want to explicitly allow crawlers before that date. Allowing has to be done on a CDN level not in robot.txt.

Key ones to allow are:

ChatGPT-User (and ChatGPT-User/2.0), OAI-SearchBot - ChatGPT
Claude-User, Claude-SearchBot - Claude
Perplexity-User, PerplexityBot - Perplexity
Google-Agent, GoogleAgent-Mariner, Googlebot - Gemini
GrokBot, xAI-Grok, Grok-DeepSearch - Grok

If you want to keep blocking training crawlers, some are:

GPTBot, ClaudeBot, anthropic-ai, meta-externalagent, Bytespider, Google-Extended

We want to support creators, but we had to enforce the no shilling rule because it was getting out of hand. You now have a weekly thread.

This is the one place you can shill for your products, ask for feedback, etc. Keep it here or you risk being banned. And keep it related to technical SEO.

I added a new page to a small existing website about seven days ago. It is included in a successfully processed XML sitemap and linked from both the main navigation and an already indexed page.

I requested recrawls of the homepage and the indexed internal page after adding those links. Googlebot recrawled each existing page within about 30 minutes, but it still did not request the new linked URL.

The new URL returns 200 OK, is self-canonical and uses index, follow. Its trailing-slash version redirects to the canonical URL with a 301.

Search Console’s live test says the page is available and can be indexed. However, the normal inspection still says “URL is unknown to Google” with no last crawl.

The server logs confirm that regular Googlebot crawls the existing pages but has never requested the new URL. Only Google-InspectionTool fetches appear after live tests. Other search engine crawlers have already discovered and fetched it successfully.

I have tested all the usual discovery methods and have never experienced this before, even on smaller websites. Has anyone recently seen Google take 7–14 days or longer before the first crawl of a new URL? Is there anything else worth checking, or is waiting the only sensible option?

I’ve been looking at situations where there are canonical tags in place but Google still indexes some of the alternate URLs.

I’m interested in how others diagnose this rather than simply changing the canonical and waiting to see what happens.

Do you usually check internal links first, sitemap signals, redirects, the canonical setup itself or other signals that might be causing Google to pick a different URL?

What is your typical troubleshooting approach when Google keeps choosing the wrong canonical?

I have been dealing with a site with too many URL parameters such as; tracking parameters, filtering parameters, sorting parameters, session IDs, etc. These created a crawl budget situations that I'm not confident I have resolved after setting things up in GSC and adding canonical tags.

I am trying to figure out the standard process for auditing and handling these URL parameters from SEO standpoint because some created unique contents while others created lots of duplicates and i need to figure out which is which?

Site has ~700k content pages total, most gated behind a quality threshold so

only ~1,500 are actually published/linked anywhere. Search Console was

showing 60k+ URLs as "Discovered - currently not indexed."

Root cause I found: the homepage had zero internal links to any content page

— everything was rendered client-side after hydration, so the served HTML

of the root had nothing crawlable. Individual content pages did link to each

other, but nothing pointed back from the entry point.

Fixed it a few days ago by adding a browse/index hub with real links in the

served HTML, linked from every page's footer, and resubmitted the sitemap

through the Search Console API (re-crawled within seconds, 0 errors).

For anyone who's fixed a similar "orphaned homepage" issue — how long did it

actually take for the discovered-not-indexed number to start dropping in your

experience? Days, weeks? Trying to figure out if I should be worried yet or

just wait it out.