Technical notice for site operators

If you found this page in your server logs, UponSale read the product data your store publishes so we could compare your prices against other retailers selling the same items. Here is exactly what we do, and how to make it stop.

Identification

Our crawler sends this user agent on every request:

UponSaleBot/1.0 (+https://uponsale.com/bot; price comparison)

What it requests

• /robots.txt, before anything else and periodically thereafter.
• Your public WooCommerce Store API product endpoint, paginated at up to 100 items.
• Nothing else. No admin paths, no checkout, no customer data, no login attempts.

How politely

• One request at a time per domain.
• A delay between requests, and a hard per-minute ceiling.
• Your Crawl-delay is treated as a floor, never a target.
• Conditional requests where your server supports ETags, so nothing changing costs you almost nothing.
• Exponential backoff on 429, honouring Retry-After.
• A circuit breaker: after repeated errors we pause the domain for hours rather than retrying.
• On a 403 or 401 we stop immediately and do not come back automatically.

What it will never do

• Solve or bypass a CAPTCHA or any bot-detection measure.
• Ignore or work around robots.txt.
• Attempt authentication, or touch any non-public endpoint.
• Place an order, add to a cart, or submit any form.
• Download or re-host your product photography on our servers.
• Republish your product descriptions.

We do display your product images beside your offer, loaded from your own image host at a reduced size, to help a shopper identify the product on a page that links to you. If you would rather we did not, tell us and we will stop using yours — your products stay listed either way.

How to slow it down or block it

Add either of these to your robots.txt:

# Slow us down
User-agent: UponSaleBot
Crawl-delay: 30

# Or stop us entirely
User-agent: UponSaleBot
Disallow: /

We re-check robots.txt regularly and will comply on the next pass. If you want it to take effect sooner, or want your store removed from the site altogether, tell us and we will do it manually — no forms, no argument.

Why we do this

Independent specialist retailers are largely invisible on the big comparison sites. We list them alongside each other so a shopper researching a model can find the store that actually has the best price — which is frequently a small independent rather than a marketplace. Every outbound link sends the visitor straight to your own product page. Read the full methodology or see what we take and what we refuse to take.