AI routing and on‑premise model deployment

Early in July, AI teams began experimenting with routing policies and semantic caching aimed at lowering the cost of running large language models. The approach involved directing inference requests to the most appropriate model based on task complexity, and using cache mechanisms to reuse previously computed results.

By late July, AT&T unveiled an AI routing framework that dynamically selects among several open‑source models—ranging from smaller, inexpensive options to larger, high‑performing ones—achieving up to a 90 % reduction in inference expenses while maintaining latency and response quality. The system integrates models such as Google’s Gemma 4, OHI‑4, OSS‑120B, and a telecom‑specific OTel 2.0 model.

Concurrently, the pure‑Java agentic AI platform Tools4AI, paired with the Ollama runtime, enabled enterprises in regulated sectors like insurance to run AI agents on‑premise. A demo showcased a claims‑triage agent that ingests free‑text incident reports, routes them to appropriate business actions, extracts structured data, and enforces human oversight, ensuring sensitive data never leaves the organization.