Scheduler result contracts and safe failure observability
Table of contents
Scheduler result contracts and safe failure observability
The absence of an exception does not mean that a scheduled job succeeded. When jobs catch errors and return None, error, counts, or partial states, monitoring cannot distinguish normal empty work from a failure.
Fix the state vocabulary
Every job should converge on at least one of completed, empty, skipped, partial, retrying, or failed. Counts are supporting evidence; the state comes first. For example, a Da2ari horoscope batch is partial or failed when only some signs were generated, and only a complete batch is eligible for skip.
Do not persist raw failure text
Provider responses, URLs, DSNs, and user input can appear in exception strings. Store bounded codes such as status:partial, status:failed, or exception:TimeoutError in failure tables and logs instead of the original text. Those codes are enough for retry decisions and operational filters.
Completion criteria
- Exceptions, structured failures, and partial results use one scheduler failure path.
- A job with a success count and a failed scope is not observed as success only.
- A failure-store outage does not stop the next scheduled job.
- Operators can query the job ID, status code, and retry scope without seeing sensitive source text.
Related course: Close partial failure, recovery, and platform boundaries