Crawler ethics and tooling
5 min
Collecting public web data touches not only technology but ethics and law. Hitting too often burdens the target server, and skipping the terms of service can spill into legal trouble.
#crawl-ethics
Posts tagged crawl-ethics
Categories
AI & LLM
9AI/LLM topics — agents, prompts
Web Fundamentals
7HTML, CSS, JS, HTTP, URL basics
Languages
4TypeScript, Java, Python, Rust
Programming
4Data structures, Big-O, patterns
Frontend
18React, Next.js, Tauri, Tailwind
Backend
19Spring, FastAPI, SQL
Agent tooling
15Claude Code · MCP · Context7 · Figma — overview of AI coding tools
Cloud
22Deployment options and decision criteria
Data
20PostgreSQL · Kafka · data strategy notes
Infra
10Docker, Compose, Caddy, SSH
Environment
11Windows vs Mac and other dev-environment notes
Mobile
4Cross-platform options compared (Tauri · Flutter · RN)
Dev tools
12pnpm, uv, Gradle, Git, regex
Philosophy
10KISS · DRY · YAGNI · naming — development philosophy
Security concepts
9OAuth · JWT · security fundamentals
Quality / Tests
11Testcontainers, Vitest, pytest
Learning
6Searching, reading docs, debugging