Failure Handling Pattern for High Availability LLM Deployment Python

• The circuit breaker pattern
• The decorator pattern
• The iterator pattern
• The builder pattern

​

Why the circuit breaker is best for LLM deployments

• Prevents cascading failures by short-circuiting repeated failing calls (for example, when an LLM endpoint is rate-limited, overloaded, or returning errors).
• Conserves scarce resources (threads, sockets, GPU slots) that would otherwise be tied up retrying doomed requests.
• Enables graceful degradation: while the breaker is open you can return cached responses, fallback to a smaller model, or provide an informative error to the user.
• Supports recovery: after a timeout the breaker transitions to half-open to probe the backend; successful probes close the breaker again.

​

Typical circuit breaker states

• Closed: Requests proceed normally; failures are counted.
• Open: Requests are blocked (fast-fail or fallback) once failures exceed thresholds.
• Half-open: A limited number of trial requests are allowed; successes close the breaker, failures re-open it.

​

Recommended implementation considerations for LLM deployments

• Set failure thresholds and reset timeouts based on provider SLAs and observed latency/failure patterns.
• Combine circuit breakers with exponential backoff and limited retries for transient errors.
• Instrument the breaker with metrics and logs (failure counts, state transitions, latencies) and expose those metrics to your monitoring stack.
• Implement fallback strategies: cached responses, a degraded model, explicit user guidance, or a retry endpoint.
• Use health checks to detect and restore healthy model instances, and to drive intelligent routing decisions.

​

Quick comparison: patterns and relevance

​

Example (using the pybreaker library)

from pybreaker import CircuitBreaker, CircuitBreakerError
import time
import random

# Trip the breaker after 5 failures, reset after 60 seconds
llm_breaker = CircuitBreaker(fail_max=5, reset_timeout=60)

def call_llm_client(prompt: str) -> str:
    """
    Simulated LLM client call. Replace this with your real client invocation,
    e.g., llm_client.predict(prompt) or an HTTP request to the model endpoint.
    """
    # Simulate intermittent failures / latency
    if random.random() < 0.25:
        raise RuntimeError("Transient error from LLM backend")
    return f"LLM response for: {prompt}"

@llm_breaker
def call_llm(prompt: str) -> str:
    return call_llm_client(prompt)

def call_with_fallback(prompt: str) -> str:
    # Basic retry with exponential backoff for transient failures
    max_attempts = 3
    backoff = 0.5
    for attempt in range(1, max_attempts + 1):
        try:
            return call_llm(prompt)
        except CircuitBreakerError:
            # Circuit is open: return fallback immediately
            return "Service temporarily unavailable. Please try again later."
        except Exception as exc:
            # Transient error: retry with backoff
            if attempt == max_attempts:
                return "Could not complete request due to backend errors."
            time.sleep(backoff)
            backoff *= 2

# Example usage
if __name__ == "__main__":
    print(call_with_fallback("Summarize this document..."))

​

Why not the other patterns?

• Decorator pattern: Great for adding behavior (logging, metrics, simple retries), but it doesn’t provide the trip/half-open semantics required to isolate a failing service.
• Iterator pattern: Designed for traversal of collections; irrelevant for runtime failure isolation.
• Builder pattern: Used to construct complex objects; not applicable to handling runtime availability or failures.