Operational efficiency and optimization

Spend fewer tokens, wait less, watch everything

GenAI bills by the token and users judge you by the second. Domain four asks how you cut both without making the answers worse, and how you notice trouble before the invoice does.

Cost levers

Start with tokens. Input and output are priced separately, and output is usually the pricier side. Trim system prompts, prune retrieved context to the chunks that matter, set maxTokens so a rambling model cannot ramble forever, and count tokens before you send. Next, pick the right model for the job: a small model for classification and extraction, a larger one for hard reasoning, and intelligent prompt routing or your own router to split the difference.

Caching avoids paying twice. Prompt caching reuses a long, stable prompt prefix (system prompt, documents, tool definitions) at a lower token price and with lower latency. A semantic cache embeds incoming prompts and returns a stored answer when a near-identical question was already answered. Deterministic request hashing handles exact repeats. For work that is not urgent, batch inference runs prompts asynchronously from S3 at a discount, and Provisioned Throughput suits steady high volume where guaranteed capacity matters more than flexibility.

Performance

Stream responses so users see progress early, run independent calls in parallel, precompute predictable answers, and pick latency-optimised models for interactive paths. Tune retrieval too: right-size indexes, preprocess queries, and use hybrid search. Parameters matter: lower temperature and top-p for deterministic tasks, higher for creative ones, and A/B test changes instead of trusting a hunch.

Monitoring

Bedrock publishes CloudWatch metrics such as invocations, latency, input and output token counts, throttles and errors. Add model invocation logging for forensics, X-Ray for traces across services, and custom metrics for the things only you know: cache hit rate, retrieval hit rate, thumbs-down rate. Alarm on token bursts and cost anomalies. Watch vector stores as well, since slow queries and stale indexes look like a dumb model.