Self-Hosting Decision Models: Serving, Quantisation, Latency and Operations Training Course

Deployment Options

• Trade-offs between hosted APIs and self-hosting
• On-premises, edge, and air-gapped patterns
• Single model versus model portfolio approaches
• Cost-per-decision comparisons

Hardware Sizing and Capacity

• Memory footprint based on parameter count
• CPU-only versus GPU acceleration
• Concurrency and throughput planning
• Headroom for increasing task lengths

Serving Stacks

• vLLM and SGLang options
• llama.cpp and GGUF execution
• Ollama for local development
• Selecting stacks based on decision workload characteristics

Quantization Without Losing Calibration

• Precision formats and memory savings
• Measuring quality and calibration loss
• Per-tensor and importance-aware variants
• Acceptance testing following quantization

Latency Engineering

• Median versus tail latency
• Latency growth with context length
• Warm start and model caching
• Batching multiple decision queries

Edge and Constrained Environments

• Small decision models on CPU
• Apple Silicon and embedded deployments
• Offline operation and update strategies
• Air-gapped weight distribution

Containerization and Scaling

• Image design for model serving
• Volume and cache management
• Horizontal scaling and routing
• Resource limits and GPU sharing

Observability

• Request, latency, and error metrics
• Confidence distribution monitoring
• Tracking decision outcomes and escalations
• Alerting on calibration drift

Versioning and Rollout

• Model and adapter versioning
• Shadow and canary deployments
• Pinned model files for audit purposes
• Rollback procedures

Security and Multi-Tenancy

• Authentication and rate limiting
• Tenant isolation for prompts and logs
• Network isolation and egress control
• Audit logging of decisions