Step 1
Crawler ethics and legal boundaries
1 views
Table of contents
The tech is easy, the boundaries less so. The biggest risk isn't bans or lawsuits but accidental outages on someone else's site.
1. robots.txt
User-agent: *
Disallow: /admin/
Crawl-delay: 10
- Weakly enforceable legally
- Ignoring it raises ban and legal risk
- Declare your crawler UA (
MyBot/1.0 (+https://mysite.com/bot))
2. Self rate-limit
await session.get(url)
await asyncio.sleep(1 + random.random())
1–5 req/s is polite. 100 req/s borders on DoS.
3. Terms of Service
- Many sites forbid automation
- Public data is different — often there's an open API
- Portal APIs are the safest path
4. No personal data
Emails, phones, names — PIPA / GDPR territory. Even publicly posted, bulk collection or repurposing is legally fraught.
5. Copyright
- Copying full text is out
- Summaries + links are fine
- Images carry both copyright and portrait rights
- Databases have their own protection (
sui generis)
6. No evasion
CAPTCHA bypass, IP rotation, cookie tricks = intentional circumvention. CFAA / unauthorized access laws apply.
7. Safer defaults
- Use public data portals first
- Obey robots.txt
- 1–5 req/s
- Announce your UA with contact
- Skip personal data
- Summarize and link for copyrighted text
8. If things go wrong
An admin complains about an outage:
- Stop immediately
- Apologise, explain
- Share prevention steps
- Offer help with recovery
Most admins forgive honest mistakes. Dishonesty compounds.
9. Public API portals first
- data.go.kr, data.gov
- opendart.fss.or.kr, data.nps.or.kr, opendata.hira.or.kr
10. Gotchas
- Ignoring robots.txt
- Concurrency too high
- Personal data inclusion
- Forged UAs
Closing
Respect the fact that "your requests live on someone else's server". Speed, politeness, and attribution cover 90%.
Next
- 02-static-vs-dynamic