SageMaker AI: route KV cache without mixing contexts
AWS has announced a routing mode for SageMaker AI real-time endpoints that directs a request to the instance holding the longest matching portion of its prompt in KV cache. The feature targets multi-turn conversations, long shared contexts, and high load. For AI Belgium and AI France projects, the potential latency gain does not remove the need to segregate contexts or measure the benefit using authorised data and actual workloads.
1. What changes in routing
The vLLM container publishes cache-storage and eviction events; SageMaker AI then keeps a managed index to direct a request to an instance that already has useful context. AWS says this index contains neither prompts, token IDs, nor KV tensors, but one-way truncated hashes associated with instances. If no useful cache is available, or matching instances are busy, the service falls back to load-aware routing.
2. Selecting the right test scope
This option applies to compatible GPU endpoints with at least two instances or inference-component copies, and requires a supported AWS vLLM container. It covers OpenAI-compatible Completions or Chat Completions requests. For a RAG pilot, start with a fictitious document base and conversations without personal data: short prompts or limited cache pressure can make prefix-aware routing simpler and sufficient.
3. Underside analysis: isolate customers and permissions
A cache improves execution; it must not become a shortcut between business perimeters. In an Odoo Belgium, Odoo France, or Odoo Enterprise integration, separate test datasets by organisation and confirm that read permissions are applied before retrieval and context construction. Keep write approvals in the ERP: inference optimisation does not entitle an agent to confirm an order, invoice, or payment.
4. Test performance, fallback, and evidence
Compare time to first token, throughput, success rate, and cache-hit rate using identical test traffic. Simulate eviction, an instance replacement, and a cache miss; then ensure fallback maintains service without changing permissions or losing pending cases. Keep the endpoint configuration, container version, model version, and test results in the operating record.
Decide from a business measure: cost and delay per correctly processed case, including denied access. Have context segregation approved before exposing an Odoo workflow to production.
Scope governed enterprise AIRead AWS’s official technical announcement · Read AWS’s product announcement