Inference Autoresearch
Run autonomous GPU serving research: calculate hardware physics ceilings, run noise-controlled streaming benchmarks, and hill-climb vLLM serving settings from the console, CLI, or your coding agent.
Inference Autoresearch measures what a model and GPU can theoretically deliver, runs repeatable streaming benchmarks against live vLLM workers, and tunes serving settings one lever at a time without degrading answer quality.
You can run Inference Autoresearch in two ways:
- Autonomous GPU hill-climbing: Create an autoresearch draft in the console, CLI, or MCP, preview the compute ceiling, and let Tensorant cycle through candidate serving settings on GPUs automatically.
- Agent-driven or manual loop: Connect a coding agent over MCP or run the CLI to compute ceilings, deploy candidates with custom serving settings, run
run_inference_benchmark, inspect the ledger, and stop temporary GPUs when finished.
Inference Autoresearch runs on chat projects for a base model, a compatible Hugging Face checkpoint, or a finished fine-tuning run's LoRA adapter.
Physics ceilings
Before renting a GPU, Tensorant calculates the hardware envelope from the model's config.json (total and active parameter counts, hidden size, layers, attention and KV heads, and head dimension) and the target GPU's VRAM, memory bandwidth, and BF16/FP8 compute throughput:
- Weight memory and KV headroom: Estimates parameter footprint across dense models and sparse MoE architectures (where all weights reside in VRAM while only active parameters are read per decode token) alongside
kv_kb_per_tokenatcontext_lengthandmemory_cap_percent. - Single-stream and concurrent decode ceilings: Computes
c1_decode_ceiling_tpsfrom active weight plus mid-context KV cache bytes divided into effective GPU memory bandwidth, and boundsconcurrent_decode_ceiling_tpsby the minimum of bandwidth saturation and tensor-core FLOPs. - Prefill ceilings and TTFT floor: Models compute-bound prompt ingestion throughput across
prefill_lengths(128to32,768tokens) and derivesttft_floor_ms. - Stretch-target assessment: When you set
stretch_target_tps, Tensorant classifies the goal asfeasible,tight(within 15% of the single-stream ceiling),above_ceiling, orimpossible(weights do not fit VRAM).
Connect your coding agent
Connect Claude Code, Cursor, Codex, or another MCP client to https://mcp.tensorant.io/mcp (see the MCP quickstart):
- Read access lets an agent inspect projects and call
calculate_inference_ceiling,list_inference_optimizations, andget_inference_optimization. - Build access additionally lets an agent queue
run_inference_benchmarkagainst an already-ready deployment or endpoint. - Full access lets an agent create Inference Autoresearch drafts, preview spending, start autonomous GPU runs with your explicit confirmation, launch or stop temporary test deployments, and cancel runs.
For example, ask your agent: "Check the physics ceiling for my project's latest fine-tuned run on an L4 vs H100 at 8 concurrent streams, then preview an Inference Autoresearch run targeting concurrent_decode."
| Tool | Purpose | Access |
|---|---|---|
calculate_inference_ceiling |
Inspect VRAM fit, KV headroom, and roofline tok/s without renting hardware | Read |
list_inference_optimizations |
List Inference Autoresearch runs and winning speedups in a project | Read |
get_inference_optimization |
Read hardware ceilings, ledger metrics, execution logs, and best_config |
Read |
run_inference_benchmark |
Benchmark a ready deployment (managed:<id>) or configured endpoint |
Build |
create_inference_optimization |
Save target metric, concurrency, context length, and trial budget as a draft | Full |
start_inference_optimization |
Preview cost and launch autonomous single-lever GPU autoresearch | Full |
cancel_inference_optimization |
Stop an autonomous autoresearch run and trigger GPU cleanup | Full |
Autonomous GPU runs are subject to Tensorant's two-step spending preview: your agent shows the GPU price per hour, trial budget, and worst-case compute cost and must obtain your confirmation before starting GPUs.
Streaming benchmarks
Whether triggered autonomously or through run_inference_benchmark, every benchmark trial measures live OpenAI streaming responses (stream: true with stream_options: {"include_usage": true}) against a uniform protocol:
- Unmeasured warm-up: Discards an initial short streaming completion so CUDA graphs and kernels stabilize before timing begins.
- Short-prompt TTFT: Records time-to-first-token (
ttft_short_ms) on a compact prompt. - Median single-stream decode: Runs a fixed technical prompt panel across
repsrepetitions (3to8, default5), uses server-reportedusage.completion_tokens, and reports the median single-stream tok/s (c1_decode). - Wall-clock concurrency: Fires
concurrency(1to64) simultaneous streaming completions and measures true overlapping wall-clock elapsed time (earliest starttolatest end) to reportc_n_aggregateandc_n_per_stream. - Unique cold-prefix prefill sweeps: Prepends a unique per-run document prefix (
synthetic_cold_prompt) for each configured prefill length so automated prefix caching cannot artificially inflate prefill speed. - Integrity and needle gates: Flags any repetition loop when an 8-word window repeats more than three times (
LOOP), flags missing server token counts (NO_USAGE), flags failed concurrent requests (REQ_FAIL), and runs a three-code long-context retrieval prompt (CODE_ALPHA,CODE_BETA,CODE_GAMMA, flaggingNEEDLE_FAILif any needle is missed).
Single-lever hill-climbing
Inference Autoresearch starts from a control baseline on the chosen GPU using clean default vLLM settings, then tests one serving setting change at a time in a fixed priority order while guarding all other metrics:
gpu_memory_percent: Expands usable KV cache memory allocation up to your configuredmemory_cap_percent(50to95, default92).max_num_seqs: Aligns concurrent sequence capacity (clamp(concurrency * 2, 16, 256)) when your workload targets high batch concurrency.max_num_batched_tokens: Tunes chunked prefill token batching (2048,4096,8192, or16384), ordered according to yourtarget_metric(c1_decode,concurrent_decode,prefill, orttft).block_size: Evaluates KV cache page block sizes (32and64).prefix_caching: Whentarget_metricisprefill, checks whether disabling automatic prefix caching improves cold ingestion throughput.- Opt-in FP8 (
quality: fp8_allowed): By default (quality: lossless), weights and KV cache stay lossless. Settingfp8_allowedadditionally evaluateskv_cache_dtype: fp8(where supported by the model architecture) andquantization: fp8(for unquantized base models without a LoRA adapter on FP8-capable GPUs such as L4, L40S, H100, H200, and B200).
Each candidate is evaluated against the current best configuration using a 3% improvement gate:
accepted: No integrity flags triggered, the target metric improved by more than 3%, and non-target decode or TTFT metrics did not regress past guard thresholds. The candidate becomes the new winning baseline (best_config).rejected: Either an integrity flag triggered, a non-target metric regressed past threshold, or the change landed within 3% noise.
When two consecutive trials land inside measurement noise without gains, or when max_trials is reached, the autoresearch loop cleans up temporary test deployments and marks the task completed. In the console, open Autoresearch and click Create API model on a completed run to pre-fill Models -> Publish with the winning vLLM configuration.
From the CLI
Use the tensorant optimize group to inspect physics ceilings, run Inference Autoresearch, or drive custom benchmarks from shell scripts and CI:
tensorant optimize ceiling --project PROJECT_ID --gpu "NVIDIA L4" --concurrency 8 --context-length 4096
tensorant optimize create --project PROJECT_ID --config optimize.yaml
tensorant optimize start OPTIMIZATION_ID --preview
tensorant optimize start OPTIMIZATION_ID --wait
tensorant optimize show OPTIMIZATION_ID --jsonSee CLI commands and Advanced serving settings for complete reference details.
Need a hand? Visit troubleshooting