How do I set a clear rule for when to log at WARN versus ERROR?

Short answer: the level should be driven by “does a human need to act?”, not by “what the log line is about”.

Short answer

ERROR = a promise you broke; WARN = something unexpected that the system recovered from on its own. In other words, let the outcome pick the level, not the call site. There’s a volume side to the noise too; I covered the scenario where logs fill the disk and turn into an outage in the log storm answer.

Why

1. Actionability is the axis. ERROR is when a request or job failed because of your fault and someone needs to look. WARN is something unexpected but tolerated: the retry succeeded, a fallback kicked in, a soft limit was approached. If your answer to “if this line fires at 3am, would I need to wake someone?” is yes, it’s ERROR.
2. An expected failure isn’t a bug. A 404, a validation error, a user’s bad input — those aren’t your code breaking; they’re INFO or, at most, WARN. Rough rule: 4xx is the client’s problem, 5xx is yours. Reserve ERROR for when your own code failed to keep its promise.
3. Retries and fallbacks produce different outcomes. A single failed attempt that will be retried is WARN; a transient blip isn’t actionable. But once all retries are exhausted or the message lands in the dead-letter queue, it’s ERROR — because the work is permanently unfinished.
4. ERROR and FATAL/critical aren’t the same thing either. FATAL is when the process can’t continue (missing config, failed boot); ERROR is a request/job failure while the process stays up.

What to do

1. Alert on ERROR only, and by rate. Let your alerts trigger off ERROR (and fatal), and even then on a threshold/rate (“50 ERRORs in 5 minutes”), not on a single line. WARN feeds dashboards, not pagers.

2. Don’t log the same failure five times. In Go, if every layer both returns and logs the error, one failure becomes five lines. Log once at the boundary where the outcome is decided (the handler); lower layers should just wrap and return the error. In Laravel, do the same by using the report() level in the exception handler instead of sprinkling Log::error everywhere.

   if err := publish(ctx, msg); err != nil {
       if attempt < maxRetries {
           log.Warn("publish failed, will retry", "attempt", attempt, "err", err)
           return retry
       }
       // retries exhausted: now it's genuinely actionable
       log.Error("publish failed permanently, sent to DLQ", "err", err)
   }

3. Make every ERROR line actionable. Every ERROR line should carry enough context to assume someone will look at it in the middle of the night: request/trace id, the relevant identifiers, and the wrapped error chain. An ERROR nobody can act on isn’t really an ERROR.

Bottom line: personally I’d give the team a one-sentence rule: “If this log firing at 3am means someone must wake up, it’s ERROR; otherwise WARN.” On top of that I’d enforce logging a failure only once, at its boundary, and wire alerts to the ERROR rate. Those two changes dissolve most of the noise on their own; you clean up the rest over a few weeks by re-leveling the misclassified lines. The goal isn’t zero ERRORs — it’s that every ERROR actually means something.