Did you know that sometimes those bot protection services will serve Google and other search engines noindex directives? Google’s John Mueller said this on X, saying, “Sometimes there’s bot-protection (or a login, interstitial, etc) triggering that has a noindex on it.”
He also added that it would be better for them to serve 503 server status codes. “Better would be to use 503 for server-side blocks like bot-protection,” he wrote.
In the end, the person he replied to was having a different issue causing the noindex, it was not bot protection. He said it was “some shenanigans with NextJS doing client side rendering instead of server rendering.” “The HTML had a noindex tag, which was removed after rendering on the browser. The HTML shown on GSC is the one AFTER JavaScript rendering, which is why I couldn’t find it and thought it was a GSC bug,” he added.
This was the error he was seeing:
Here are those posts:
wtf is going on here? why is Google not indexing this one?
They say there’s a “noindex” tag, but there isn’t one. robots.txt looks good too.
Anyone? #seo help pic.twitter.com/DYzJsLQ4Yk
— Guilherme (@goenning) December 26, 2023
Thanks John, eventually found it thanks to @HamilcDev
Some shenanigans with NextJS doing client side rendering instead of server rendering
The HTML had a noindex tag, which was removed after rendering on the browser. The HTML shown on GSC is the one AFTER JavaScript rendering,… https://t.co/NffU7suVLM
— Guilherme (@goenning) December 26, 2023
Forum discussion at X.
