The first time a list-building script gets blocked, it is obvious: connections hang, pages return 403, somebody fixes it. The second way is the expensive one, because nothing appears to break. Requests keep succeeding, rows keep landing in the spreadsheet, and the rows are wrong — truncated result sets, stale cached copies, placeholder values served deliberately to a client the site has decided not to trust.
Here is the takeaway up front: anti-bot systems rarely fail loudly, and the real cost of aggressive collection is not downtime — it is a list that looks full and is quietly full of garbage. The teams who build good lists from public sources are not the ones with the cleverest evasion. They are the ones who collect slowly, identify themselves, treat a challenge as information rather than an obstacle, and verify everything before it reaches a sequencer.
What actually blocks automated research
"IP ban" is shorthand for at least six different mechanisms, and they escalate in roughly this order.
Rate limiting. The politest defence: exceed a request budget and you get HTTP 429 with a Retry-After header telling you exactly how long to wait. Honour it and nothing bad happens. Ignore it — which naive scripts do, because a 429 body is short and parses as "no results" — and you get promoted to the next tier.
IP and ASN reputation. Your address is scored, and so is the network it sits in — which is why datacentre ranges get challenged harder than residential ones regardless of behaviour, and why a shared cloud IP inherits whatever the previous tenant did with it.
TLS and HTTP fingerprinting. Before your request reaches application code, the shape of your TLS handshake and the order of your HTTP/2 headers already suggest which client library sent it. A User-Agent claiming Chrome arriving with a fingerprint that is plainly Python's requests is a mismatch that costs nothing to detect.
Behavioural scoring. Request timing without human jitter, perfectly sequential URL traversal, no static assets fetched, no referrers. Individually meaningless; together, a signature.
Interactive challenges. CAPTCHAs and interstitial pages — reCAPTCHA, Cloudflare Turnstile, Geetest and similar. Note that a challenge is not the same as a block. It is a site asking who you are.
Silent degradation. The one that costs you money. Instead of refusing you, the site starts serving stale cache, capping results at the first page, or returning subtly wrong values. Your pipeline reports 100% success. Your data is fiction.
Why aggressive collection burns the asset you're building
The instinct when blocked is to push harder: more proxies, faster rotation, more parallelism. This trades a small problem for three larger ones.
You burn the identity you need. Rotating through residential proxies to hammer a professional network is a good way to lose the account doing the browsing — and for most sales teams that account is the actual asset. A banned scraper is an afternoon of work; a banned company profile is a channel.
You collect the worst version of the data. Sites under defensive pressure serve degraded responses first and block second. The harder you push, the higher the share of your rows that came from a defensive path rather than the real one. Aggression is negatively correlated with accuracy, which is precisely backwards from what the team wants.
You burn the sending domain next. Scraped contact data decays fast and is often role-guessed rather than confirmed. Push it unverified into a sequencer and the bounce rate spikes — the loudest negative signal a mailbox provider reads. You spend a month building a list and a week destroying the reputation you need to mail it. If sourcing is still an open question, the trade-offs in buy vs build a prospect list are worth settling before you write any collection code.
The compliance line around public data
"Publicly accessible" and "free to take and use" are not the same claim, and the gap between them is where teams get into trouble. A workable line has four parts.
Authentication is a hard stop. Content behind a login you agreed to terms to obtain is not public data, and automating access to it against those terms is the version of this that gets companies sued rather than rate-limited. Never build a workflow that depends on getting past an authentication or authorisation control.
Terms of service and robots.txt are the site's stated position. robots.txt is not legislation, but it is an unambiguous, machine-readable statement of what the operator wants automated clients to do, including Crawl-delay. Ignoring it removes any "we acted in good faith" defence you might otherwise have, and reads badly in a way that is hard to walk back.
Personal data stays personal when it is public. Under GDPR and similar regimes, a work email published on a company site is still personal data. Collecting it engages the usual obligations: a lawful basis (usually legitimate interest, which requires a balancing test you should actually document), notice to the individual, and the ability to honour deletion requests. This applies to the list you built yourself exactly as it applies to one you bought.
Volume is itself a signal. Incidental collection and systematic harvesting at scale are treated differently — by regulators, and by the sites you collect from. A process that takes what it needs, when it needs it, sits in a materially different position from one that mirrors a database.
A collection design that does not get blocked
Most blocking problems are solved by design rather than by evasion.
Read robots.txt and mean it. Parse it, cache it, honour Crawl-delay, and skip disallowed paths.
Identify yourself. A descriptive User-Agent with a contact URL turns an anonymous bot into an accountable one. Operators block anonymous traffic reflexively; they usually email accountable traffic first.
Budget requests, do not maximise them. One to two requests per second per host, single-threaded per domain, spread across time rather than compressed into a burst.
Back off on the signal the server gives you. Honour Retry-After exactly; use exponential backoff with jitter when it is absent.
Cache aggressively. Most re-fetching is waste, and a local cache with a sane TTL removes the bulk of the traffic that gets teams blocked.
Prefer the front door. Official APIs, sitemaps, structured data, and bulk exports are almost always cheaper than the version that needs defeating.
import time, random, requests
from urllib.robotparser import RobotFileParser
rp = RobotFileParser()
rp.set_url("https://example.com/robots.txt")
rp.read()
UA = "AcmeResearchBot/1.0 (+https://acme.example/bot; [email protected])"
def fetch(url, attempt=0):
if not rp.can_fetch(UA, url):
return None # disallowed: do not fetch
r = requests.get(url, headers={"User-Agent": UA}, timeout=20)
if r.status_code == 429 or r.status_code >= 500:
if attempt >= 4:
return None # stop, don't grind
wait = int(r.headers.get("Retry-After", 0)) or (2 ** attempt) + random.random()
time.sleep(wait)
return fetch(url, attempt + 1)
if r.status_code == 403:
return None # a decision, not a puzzle
return r.text
delay = rp.crawl_delay(UA) or 1.0
The 403 branch matters. A refusal is a decision by the operator, and the correct response is to stop and find another source — not to rotate identity until the refusal goes away.
When you legitimately hit a challenge
Not every CAPTCHA is a "no". Teams meet them constantly where automation is entirely expected: regression-testing your own sign-up flow, accessibility audits, uptime and content monitoring on properties you own, and licensed sources whose edge provider challenges all datacentre traffic indiscriminately. There the challenge is friction in a permitted workflow, and solving it programmatically is ordinary plumbing.
Services like CaptchaAI cover that layer. The protocol is the familiar 2Captcha-shaped one — submit to /in.php, poll /res.php — so most existing tooling works unchanged:
curl -X POST "https://ocr.captchaai.com/in.php" \
-d "key=YOUR_API_KEY" \
-d "method=userrecaptcha" \
-d "googlekey=6Lc_SITE_KEY" \
-d "pageurl=https://example.com/monitored-page" \
-d "proxy=user:[email protected]:8080" \
-d "proxytype=HTTP" \
-d "json=1"
# {"status":1,"request":"2122988149"}
# 2. Poll roughly every 5s until it stops returning CAPCHA_NOT_READY
curl "https://ocr.captchaai.com/res.php?key=YOUR_API_KEY&action=get&id=2122988149&json=1"
# {"status":0,"request":"CAPCHA_NOT_READY"}
# {"status":1,"request":"03AGdBq26..."}
Handle CAPCHA_NOT_READY as "keep polling", and ERROR_UNSOLVABLE and ERROR_ZERO_BALANCE as terminal — retrying either just burns budget. Routing each task through the same proxy as the page request (proxy + proxytype) matters more than people expect: a token minted from one network and redeemed from another is a mismatch the challenge provider can see. The vendor's own published figures put reCAPTCHA v2 above 99.5% within 60 seconds and Cloudflare Turnstile at 100% under 10 seconds — treat those as a starting expectation to measure, not a guarantee.
What this does not change is the compliance question. A solver removes a technical obstacle; it does not grant permission you did not have. If the source is behind a login, or its terms forbid automated access, the challenge was the least of the reasons to stop.
Verification is what decides whether the list was worth building
Every hour of careful collection is wasted if the output goes into a sequencer unchecked. Collection tells you an address probably exists; verification tells you a mailbox will accept it. Score every record before it can be mailed, drop the invalid, quarantine the risky, and hold each source to a measured accuracy bar on a sample of 25 rows. A smaller verified list beats a larger scraped one on every metric that reaches a revenue report.
FAQ
Is scraping public web data legal?
It depends on what you collect, from where, and how. A public page with no login involved is a materially different position from data behind authentication, and personal data carries obligations — lawful basis, notice, deletion rights — regardless of how public it was. Respect terms of service and robots.txt, keep volumes proportionate, and take advice before building anything systematic. This is not legal advice.
How fast can I collect before I get blocked?
Slower than you want. One to two requests per second per host, single-threaded per domain, is a defensible default; many sites publish a Crawl-delay that is stricter and should win. The better question is how few requests you need, not how many you can get away with — caching and sitemaps eliminate most traffic that triggers blocks.
Do rotating proxies solve the problem?
They move it. Rotation defeats naive IP counting but not fingerprinting or behavioural scoring, and it strips away the accountability that keeps an operator from blocking you outright. Proxies are useful for geography and stable monitoring paths; they are a poor substitute for collecting at a rate the source tolerates.
When is it reasonable to solve a CAPTCHA automatically?
When you have the right to be there and the challenge is friction rather than an answer: testing your own flows, accessibility and uptime checks, or a licensed source whose edge provider challenges all datacentre traffic. When the challenge is the site's way of saying no to your collection, solving it does not make the answer yes.
What does CAPTCHA-solving cost at list-building volume?
CaptchaAI prices by concurrent threads rather than per solve — its published tiers run from BASIC at $15/month for 5 threads to ENTERPRISE at $300/month for 200 threads, with unlimited solves per thread. For research workloads that means cost tracks your concurrency, so the throttling you should be doing anyway also caps the bill.
Next step
Design the collection to be tolerable rather than undetectable: read robots.txt, identify your bot, budget requests, honour Retry-After, cache hard, and stop at a 403 instead of escalating. Where automation is legitimate and a challenge is genuinely in the way — your own QA flows, monitoring, or a licensed source — a solver like CaptchaAI removes the friction without changing what you are permitted to collect. Then do the part that actually protects your pipeline: verify every address before it reaches a sequencer, and keep building prospecting that runs on data you can trust at prospectuso.com.
