I had a SaaS client for which I was working for around six months, and we had an issue regarding some of their blogs and pages getting de-indexed in Google.
Older pages getting de-indexed is somewhat common if Google finds that the content is thin, not particularly useful, or not getting much traffic. Sometimes crawl depth, internal linking, or overall site quality can also play a role.
What was interesting in this case was that some of the newer blogs, which had already started getting clicks, were also getting de-indexed.
Sometimes these things happen around core updates, and indexing can fluctuate for a period of time. But when we reviewed the website, most of the fundamentals looked fine. The site structure was good, internal linking was being done properly, and there weren’t any major technical issues that immediately stood out.
The client wanted suggestions to improve indexing, so I started looking deeper into the setup.
One thing I noticed was that the website was built on Webflow CMS.
While reviewing the sitemap setup, I realized that their XML sitemap did not include a last modified date. I know an XML sitemap is only one way for search engines to discover content, but it still acts as an additional signal when pages are updated.
Adding a sitemap with last modified dates is not something that Webflow supports natively, which is one of the limitations that some SEO teams run into when working on larger websites.
To be clear, I don’t dislike Webflow. It builds very clean and fast websites, and if you have a good developer, Core Web Vitals are usually not a concern. But compared to WordPress, there are certain SEO customizations that often require developer involvement.
Coming back to the indexing issue, I started searching for a way to generate a sitemap with last modified dates.
The solution we implemented involved Cloudflare.
Custom XML sitemap with Cloudflare on Webflow
Using Cloudflare as a reverse proxy, we generated a custom XML sitemap that could pull the last modified date from Webflow’s backend data and update automatically. The idea was to provide search engines with an additional signal whenever content was updated.
I can’t say this alone solved the indexing issue, but it was one of the improvements we implemented as part of the investigation.
There was another limitation we ran into.
Bing Indexing API Configuration for Webflow
The client also wanted a more streamlined way of submitting URLs to Bing. Unlike WordPress, where plugins make these integrations relatively easy, Webflow doesn’t have the same ecosystem of SEO plugins.
So we configured Bing’s IndexNow API workflow separately.
Of course, URLs can always be submitted manually through Bing Webmaster Tools, but this allowed a more automated submission process whenever content was published.
Interestingly, Cloudflare ended up helping us with another challenge as well.
Previously, we were tracking LLM traffic from sources like ChatGPT, Claude, Perplexity, and other AI tools using custom regex segments inside GA4 Explore reports.
Later, the site implemented a cookie consent banner, which reduced the amount of data flowing into GA4. As a result, visibility into AI-related traffic became less reliable.
Checking LLM & Other Bot Hits on Server via Cloudflare
Once Cloudflare was added, the dashboard gave us additional visibility into the types of bots visiting the website. This included AI crawlers, LLM bots, and other scraping bots.
On a WordPress website, we could have achieved something similar by analyzing server logs. However, Webflow doesn’t provide the same level of server log access.
In fact, we had previously experienced a sudden bandwidth spike on the website.
When we reached out to Webflow, they shared information about the bots accessing the site. After reviewing the list, we found that some of the traffic appeared to be coming from scraping bots, including a few Python-based scrapers.
We blocked some of those crawlers through robots.txt and other controls, which reduced the issue at the time.
Looking back, the project wasn’t really about finding a single fix for indexing.
It was more about identifying platform limitations, improving the signals available to search engines, and finding alternative ways to get visibility into crawling and bot activity.
The custom Cloudflare sitemap, Bing API integration, and Cloudflare bot monitoring all came out of that process.