Bot Mitigation in the Age of AI Crawlers: How to Strategize Your Defense

Not all bot traffic is bad. For years, bots were synonymous with risk: scrapers, scalpers, and credential stuffers trying to steal inventory, stuff stolen credentials into your login page, or scrape pricing and product data out from under you. While that is still true, a growing share of bots crawling your ecommerce site are now AI assistants and LLM-powered search tools. They’re part of the very systems that decide whether your brand shows up when someone asks ChatGPT, Claude, or Perplexity for a product recommendation. Block them indiscriminately and you disappear from a new channel of discovery that powers high-intent traffic. Leave traffic unmanaged and you risk a slower site and distorted analytics.

This nuance creates the need for a new kind of bot mitigation strategy: Allowing the right AI crawlers in to drive discovery while shutting out the ones that drive up cost and risk. The truth is, bot mitigation is a shared responsibility between brands, platform providers, and eCDNs. While your partners handle baseline defense, it’s crucial to actively monitor your own architecture and systems.  

By the numbers: How bots have changed ecommerce

• Automated traffic now makes up more than half of all internet activity.
• During Cyberweek, malicious bot traffic makes up approximately 43% of total network traffic.
• Scraping instances have increased 185% year-over-year and represents the largest and fastest-growing attack vector, accounting for 70% of bad bot traffic.

A layered framework for bot mitigation

There’s no switch that can automatically separate good bot traffic from bad. Getting this right means layering your defenses. Here’s how to do it.

Edge defense (your eCDN/WAF)

This is your first line of defense. A meaningful set of protections are self-service and you can configure them on your own with no approval process. These protections include custom WAF rules, rate limiting, IP/ASN/geo blocking, Managed Challenge, “Under Attack” mode, and Waiting Room for high-traffic events.

Other protections require more coordination. This is where a shared responsibility model comes into play. Custom rules scoped to your SCAPI zones need your edge team’s involvement. And the heaviest-duty tools (like Advanced Bot Management, Advanced Rate Limiting, Turnstile, and Precursor) carry real tradeoffs in cost and compatibility. This is particularly true if you’re running a stacked-CDN setup, so these protections go through a formal evaluation rather than a quick toggle.

One rule applies no matter which tier you’re touching: Always test new rules in log mode before switching to block. A rule promoted too fast can accidentally block legitimate traffic, including the AI crawlers and search engines you actually want to index your site.

→ For flash sales and high-traffic moments specifically, Waiting Room is self-service and worth setting up ahead of time, though it’s not a substitute for strong bot identification at extreme scale.

Further reading:

• Bot Management
• Bot Mitigation Best Practices for Flash Sales
• Traffic Management for Flash Sales
• Configure the Embedded CDN, SCAPI Zone Support

Application-level hardening

Edge defense stops traffic at the door; application-level hardening protects what’s inside. Start with your robots.txt file. Don’t simply leave this on de

fault settings. Set it deliberately based on your own site structure and treat it as guidance for well-behaved bots, not a security control.

From there, focus on your most expensive, most sensitive pages (like cart, checkout, account, and search/filter pages) with rate limiting and smart caching. Adding friction, like Turnstile challenges, to high-value forms for account creation, checkout, and gift card lookups raises the cost of abuse without adding real friction for legitimate shoppers.

Further reading:

• Caching Strategies for Salesforce B2C Commerce
• Secure Your Site
• SLAS Best Practices

Analytics integrity

Server capacity isn’t the only thing that bots can harm. They also harm your ability to understand and report on your own traffic. Non-human visits inflate session counts, skew conversion rates, and can even hit analytics endpoints directly, bypassing your actual site. For marketing and commerce leaders, that means misleading campaign performance and wasted ad spend chasing traffic that was never going to convert.

The fix is server-side validation of analytics events, so the data flowing into your reporting and personalization tools reflects real shopper behavior and not bot noise.

AI crawlers, bots, and how to prepare for peak shopping season

Cyber Week represents the single biggest moment of collision of peak human traffic and peak bot traffic. Real shoppers flood your site at the same moment as scrapers, scalpers, and AI crawlers. And the two types of traffic are getting harder to tell apart at a glance. A thoughtful bot mitigation strategy will help you deflect attacks and make sure your storefront can hold up under the load without losing the AI-driven discovery traffic that has become an essential part of holiday success.

A few things make this window different from the rest of the year:

• AI crawler activity spikes alongside shopper demand. As more shoppers use AI assistants to research gifts and compare prices before they ever land on your site, the crawlers powering those tools are hitting your pages harder. Blocking too aggressively during this window risks losing visibility in AI search right when high-intent traffic is surging the most.
  
• Malicious bot activity spikes, too. Flash sales, doorbusters, and limited-inventory drops are prime targets for scalpers and inventory-hoarding bots. The same infrastructure strain that hurts human shoppers during a legitimate traffic surge.
  
• There’s less room for error. A rule that’s too aggressive can knock out real shoppers or AI crawlers during your highest-revenue days of the year. A rule that’s too permissive can let scalpers clean out inventory before real customers get a chance.

Your pre-peak checklist:

• Audit your robots.txt now, not during the sale. Confirm it reflects your current site structure and controller paths. Don’t let a stale file block crawlers you actually want.
  
• Review your self-service eCDN settings. Rate limiting, IP/ASN/geo blocking, and Managed Challenge should already be configured and tested in log mode well before peak traffic hits, not adjusted live during a sale.
  
• Evaluate Waiting Room for flash sales or drops. Waiting Room is a self-service protection built for exactly this kind of high-traffic moment, but it should be paired with other bot identification tactics, not relied on alone at extreme scale.
  
• Loop in your edge team early. Act now if you think you need approval-gated protections like Advanced Bot Management, Advanced Rate Limiting, Turnstile, or Precursor. These protections all require formal evaluation, which takes time. You don’t want to be racing against the clock a week before Black Friday.
  
• Stress-test your checkout and login flows. These are the pages most likely to be targeted by both scalper bots and credential stuffing attempts during high-traffic events.

The goal isn’t to lock everything down the week before Cyber Week. A thoughtful approach to bot mitigation is to start early and have already made the right calls on what to allow, what to restrict, and what needs a bigger conversation, so your team isn’t making those decisions live under pressure.

Further reading:

• Launching a Successful Flash Sale
• Bot Mitigation Best Practices for Flash Sales
• Traffic Management for Flash Sales

What this means for your business

Brands that get selective about their traffic, letting AI assistants and search crawlers in while restricting the scrapers, scalpers, and credential stuffers that drive up cost and risk, protect two things at once: their site’s performance for real shoppers, and their visibility in a retail market driven by AI-powered search.

That balance doesn’t happen by accident, and it doesn’t happen once. It comes from treating bot mitigation as a shared responsibility.  It’s crucial to lean on the baseline defenses your platform and eCDN already provide, while actively monitoring and tuning your own architecture. The brands that build this muscle before peak season are the ones that will walk into Cyber Week with a fast, stable storefront and full visibility into who’s actually showing up.

Dig into the technical side of building your own bot mitigation strategy with these resources:

• B2C Commerce Shared Responsibility Model
• Bot Management
• Bot Mitigation Best Practices for Flash Sales
• Traffic Management for Flash Sales
• Launching a Successful Flash Sale
• Caching Strategies for Salesforce B2C Commerce
• Secure Your Site
• SLAS Best Practices
• Configure the Embedded CDN
• SCAPI Zone Support for B2C Commerce