If you examine the architecture of AWS, it essentially boils down to core infrastructure pillars: Compute, Storage, Networking, and the management layer that ties them together.
I have previously examined the momentary movement of packets in networking, the inertia of unused bytes in storage, and the price of idle potential in compute.
Now, we enter the most volatile frontier on modern cloud bills: Artificial Intelligence (AI) and Machine Learning (ML).
With networking, you are billed for movement. With storage, you are billed for inertia. With compute, you are billed for potential. With AI, you are billed for context.
Generative AI and machine learning introduce non-deterministic billing. Traditional CPU and memory alerts will not protect your budget when a runaway agent loop or an unoptimized RAG pipeline burns millions of tokens in minutes.
The Frontier Model Monolith (Over-Indexing on Top-Tier LLMs)
When teams move GenAI prototypes to production, they often hardcode a premier frontier model across every application component. The same reasoning engine answers a complex planning request and decides whether a string is a question or a command.
High-reasoning frontier models cost well above their lightweight siblings. Often an order of magnitude more per million tokens, with output tokens costing multiples of input tokens on top of that.
Using a frontier model for deterministic tasks, such as intent classification, JSON schema extraction, query routing, or simple sentiment evaluation, is massive architectural overkill. Running simple tasks through a heavy reasoning engine wastes most of your token budget with no measurable improvement in user experience.
The Fix: Decompose First, Then Match the Model to the Subtask
The instinct is to “add a router.” But routing is the last step, not the first.
You might think a single GenAI “feature” is a single model call. In reality, that is almost never the case, with its pipeline of subtasks, each with its own cognitive demand: classify the intent, extract the entities, retrieve the context, then synthesize the answer. The win comes from seeing those seams:
- Decompose the task into subtasks. Pull the deterministic, mechanical steps (classification, extraction, formatting, routing) apart from the genuinely generative step (multi-step contextual synthesis). Most of the volume lives in the mechanical steps.
- Match each subtask to the cheapest model that clears its bar. Send parsing, classification, and routing to lightweight models like Amazon Nova Micro or Claude Haiku 5.5. Reserve a frontier model (Claude Sonnet 5.5, or Opus 5.5 for the heaviest reasoning) strictly for the synthesis step that actually needs it.
- Let routing enforce the match. The router is just the plumbing. The mistake is making it a hardcoded boolean. A developer guessing `is_complex = True` at write time has no idea what the live prompt will look like. Instead, use a lightweight classifier to size each request before anything expensive runs.
Purpose-built tools do exactly this: TypeSafe’s Jev, for example, is a fast decision model that estimates how hard a request is and sends it to the cheapest Claude tier that can still do the job well. Jev leans toward the stronger model only when it is genuinely unsure, so saving money never silently costs you quality.
mport boto3
bedrock = boto3.client("bedrock-runtime", region_name="us-east-1")
def route_and_generate(user_prompt: str, tier: str = "light"):
model_id = {
"frontier": "global.anthropic.claude-sonnet-5-5", # synthesis only
"light": "us.amazon.nova-micro-v1:0", # classify/extract/route
}[tier]
response = bedrock.converse(
modelId=model_id,
messages=[{"role": "user", "content": [{"text": user_prompt}]}],
inferenceConfig={"temperature": 0.1, "maxTokens": 300},
)
return response["output"]["message"]["content"][0]["text"]
Two things that bite people here.
First, many current models can only be invoked through a Cross-Region Inference Profile, not a plain model ID. Call the regional (`us.`) or Global (`global.`) profile, or Bedrock returns an “on-demand throughput isn’t supported” error.
Second, the prefix is not universal: at launch, Claude Sonnet 5.5 is served through the Global profile (`global.anthropic.claude-sonnet-5–5`) while Nova Micro uses the regional US profile.
Confirm the exact current IDs in your Bedrock console before deploying. The lineup moves fast, and a retired ID fails at invocation with no graceful fallback.
The Amnesic Prompt Tax (Ignoring Bedrock Prompt Caching)
In enterprise RAG (Retrieval-Augmented Generation) applications and multi-turn conversational agents, system prompts are dense. Between system personality guidelines, few-shot examples, corporate policies, and retrieved reference documentation, an input prompt can reach 2,000 to 10,000 tokens before you add the user’s query.
Without prompt caching, Amazon Bedrock parses and bills for that static context from scratch on every single invocation.
If 10,000 users query a bot with a 4,000-token system context daily, you are paying to ingest 40 million identical tokens every 24 hours.
The Fix: Bedrock Prompt Caching Checkpoints
Amazon Bedrock supports prompt caching via the Converse API for supported models (including Anthropic Claude and Amazon Nova). By injecting a `cachePoint` block directly after your static context, Bedrock caches the prefix.
Subsequent requests that share the exact prefix receive up to a 90% discount on cached input tokens while significantly reducing time-to-first-token (TTFT) latency.
import boto3
bedrock = boto3.client("bedrock-runtime", region_name="us-east-1")
def invoke_cached_agent(system_instructions: str, user_query: str):
response = bedrock.converse(
modelId="global.anthropic.claude-sonnet-5-5", # Global inference profile at launch
system=[
{"text": system_instructions},
# FinOps Fix: checkpoint so the static prefix is cached
{"cachePoint": {"type": "default"}},
],
messages=[{"role": "user", "content": [{"text": user_query}]}],
)
usage = response.get("usage", {})
cached_tokens = usage.get("cacheReadInputTokens", 0)
print(f"Cached tokens read at up to 90% discount: {cached_tokens}")
return response["output"]["message"]["content"][0]["text"]
Two caveats decide whether caching actually pays off:
- The minimum prefix is model-dependent. Roughly 1,024 tokens for Claude Sonnet and Opus, about 2,048 for Claude Haiku, with separate thresholds for Nova.If your static context falls below that minimum, Bedrock silently skips caching and charges the full input rate — no error is raised. Always confirm `cacheReadInputTokens` in the response usage block to verify a cache hit actually occurred.
- The cache has a TTL — and it is short. The default cache entry lives for only about 5 minutes, and that clock resets with every cache hit. So caching only pays off for prefixes that are reused frequently; a prefix queried once every ten minutes expires before the next hit, and you pay full price every time.As a rule of thumb, your cache hit rate needs to clear roughly 12% at the 5-minute TTL just to break even on the write cost. For sparse-but-valuable traffic, Bedrock now also offers an optional 1-hour cache duration. It is more expensive to maintain, but worth it when reuse is spread out.
The 24/7 Dedicated GPU Tax (SageMaker Real-Time Endpoints)
Engineering teams hosting custom fine-tuned models (e.g., specialized BERT transformers, custom embedding models, or vision classifiers) on Amazon SageMaker typically deploy them behind default Real-Time Inference Endpoints.
A Real-Time Endpoint provisions dedicated, always-on EC2 instances (such as an `ml.g5.2xlarge` at ~$1.21/hr or `ml.g5.12xlarge` at ~$7.09/hr) that run 24 hours a day, 7 days a week.
For internal business applications, batch evaluation queues, or developer tools with zero traffic overnight and over weekends, these instances spend more than 70% of the month burning money in an idle state.
The Fix: SageMaker Serverless Inference
For CPU-bound models — most BERT-class transformers, embedding models, and lightweight classifiers — migrate from dedicated real-time instances to SageMaker Serverless Inference.
It provisions compute only when traffic hits the endpoint, scales down to exact zero during idle periods, and bills strictly for compute duration (per millisecond) and data processed.
One critical caveat: Serverless Inference is CPU-and-memory only. It does not support GPU instances. So if your model genuinely needs the GPU in the `ml.g5` example above (large vision models, LLMs, heavy transformers), serverless is not the right fix.
For those GPU-bound workloads with idle gaps, use Asynchronous Inference endpoints (which also scale to zero), scheduled scaling of the real-time endpoint, or multi-model/multi-container endpoints to share one GPU across several models.
resource "aws_sagemaker_endpoint_configuration" "serverless_nlp" {
name = "serverless-classifier-config"
production_variants {
variant_name = "AllTraffic"
model_name = aws_sagemaker_model.custom_nlp.name
serverless_config {
max_concurrency = 10
memory_size_in_mb = 4096
}
}
}
resource "aws_sagemaker_endpoint" "serverless_endpoint" {
name = "serverless-classifier-endpoint"
endpoint_config_name = aws_sagemaker_endpoint_configuration.serverless_nlp.name
}
The Terraform shows the serverless configuration for a CPU-bound model (the classifier case). For the GPU-bound workloads mentioned above, use an Asynchronous Inference endpoint configuration instead.
Conclusion
AI and Machine Learning do not have to be an open checkbook for infrastructure waste. Cost optimization in modern AI architectures is not about throttling model capabilities; it is about matching cognitive demand to the right class of compute.
- Break the task into subtasks and send each to the cheapest model that can do it.
- Cache the static context. But only where reuse clears the TTL.
- Retire idle dedicated GPU endpoints in favor of serverless or asynchronous inference.
Do those three things and token bloat stops eating your operating margins.
Stop paying for raw context. Pay only for the intelligence your application actually delivers.
Speak with me, or my fellow FinOps experts at Automat-it to build a cost-effective AI architecture today.
