Skip to content
Inference Autoresearch

Inference Autoresearch

Run autonomous GPU serving research: calculate hardware physics ceilings, run noise-controlled streaming benchmarks, and hill-climb vLLM serving settings from the console, CLI, or your coding agent.

Inference Autoresearch measures what a model and GPU can theoretically deliver, runs repeatable streaming benchmarks against live vLLM workers, and tunes serving settings one lever at a time without degrading answer quality.

You can run Inference Autoresearch in two ways:

  • Autonomous GPU hill-climbing: Create an autoresearch draft in the console, CLI, or MCP, preview the compute ceiling, and let Tensorant cycle through candidate serving settings on GPUs automatically.
  • Agent-driven or manual loop: Connect a coding agent over MCP or run the CLI to compute ceilings, deploy candidates with custom serving settings, run run_inference_benchmark, inspect the ledger, and stop temporary GPUs when finished.

Inference Autoresearch runs on chat projects for a base model, a compatible Hugging Face checkpoint, or a finished fine-tuning run's LoRA adapter.

Physics ceilings

Before renting a GPU, Tensorant calculates the hardware envelope from the model's config.json (total and active parameter counts, hidden size, layers, attention and KV heads, and head dimension) and the target GPU's VRAM, memory bandwidth, and BF16/FP8 compute throughput:

  • Weight memory and KV headroom: Estimates parameter footprint across dense models and sparse MoE architectures (where all weights reside in VRAM while only active parameters are read per decode token) alongside kv_kb_per_token at context_length and memory_cap_percent.
  • Single-stream and concurrent decode ceilings: Computes c1_decode_ceiling_tps from active weight plus mid-context KV cache bytes divided into effective GPU memory bandwidth, and bounds concurrent_decode_ceiling_tps by the minimum of bandwidth saturation and tensor-core FLOPs.
  • Prefill ceilings and TTFT floor: Models compute-bound prompt ingestion throughput across prefill_lengths (128 to 32,768 tokens) and derives ttft_floor_ms.
  • Stretch-target assessment: When you set stretch_target_tps, Tensorant classifies the goal as feasible, tight (within 15% of the single-stream ceiling), above_ceiling, or impossible (weights do not fit VRAM).

Connect your coding agent

Connect Claude Code, Cursor, Codex, or another MCP client to https://mcp.tensorant.io/mcp (see the MCP quickstart):

  • Read access lets an agent inspect projects and call calculate_inference_ceiling, list_inference_optimizations, and get_inference_optimization.
  • Build access additionally lets an agent queue run_inference_benchmark against an already-ready deployment or endpoint.
  • Full access lets an agent create Inference Autoresearch drafts, preview spending, start autonomous GPU runs with your explicit confirmation, launch or stop temporary test deployments, and cancel runs.

For example, ask your agent: "Check the physics ceiling for my project's latest fine-tuned run on an L4 vs H100 at 8 concurrent streams, then preview an Inference Autoresearch run targeting concurrent_decode."

Tool Purpose Access
calculate_inference_ceiling Inspect VRAM fit, KV headroom, and roofline tok/s without renting hardware Read
list_inference_optimizations List Inference Autoresearch runs and winning speedups in a project Read
get_inference_optimization Read hardware ceilings, ledger metrics, execution logs, and best_config Read
run_inference_benchmark Benchmark a ready deployment (managed:<id>) or configured endpoint Build
create_inference_optimization Save target metric, concurrency, context length, and trial budget as a draft Full
start_inference_optimization Preview cost and launch autonomous single-lever GPU autoresearch Full
cancel_inference_optimization Stop an autonomous autoresearch run and trigger GPU cleanup Full

Autonomous GPU runs are subject to Tensorant's two-step spending preview: your agent shows the GPU price per hour, trial budget, and worst-case compute cost and must obtain your confirmation before starting GPUs.

Streaming benchmarks

Whether triggered autonomously or through run_inference_benchmark, every benchmark trial measures live OpenAI streaming responses (stream: true with stream_options: {"include_usage": true}) against a uniform protocol:

  1. Unmeasured warm-up: Discards an initial short streaming completion so CUDA graphs and kernels stabilize before timing begins.
  2. Short-prompt TTFT: Records time-to-first-token (ttft_short_ms) on a compact prompt.
  3. Median single-stream decode: Runs a fixed technical prompt panel across reps repetitions (3 to 8, default 5), uses server-reported usage.completion_tokens, and reports the median single-stream tok/s (c1_decode).
  4. Wall-clock concurrency: Fires concurrency (1 to 64) simultaneous streaming completions and measures true overlapping wall-clock elapsed time (earliest start to latest end) to report c_n_aggregate and c_n_per_stream.
  5. Unique cold-prefix prefill sweeps: Prepends a unique per-run document prefix (synthetic_cold_prompt) for each configured prefill length so automated prefix caching cannot artificially inflate prefill speed.
  6. Integrity and needle gates: Flags any repetition loop when an 8-word window repeats more than three times (LOOP), flags missing server token counts (NO_USAGE), flags failed concurrent requests (REQ_FAIL), and runs a three-code long-context retrieval prompt (CODE_ALPHA, CODE_BETA, CODE_GAMMA, flagging NEEDLE_FAIL if any needle is missed).

Single-lever hill-climbing

Inference Autoresearch starts from a control baseline on the chosen GPU using clean default vLLM settings, then tests one serving setting change at a time in a fixed priority order while guarding all other metrics:

  1. gpu_memory_percent: Expands usable KV cache memory allocation up to your configured memory_cap_percent (50 to 95, default 92).
  2. max_num_seqs: Aligns concurrent sequence capacity (clamp(concurrency * 2, 16, 256)) when your workload targets high batch concurrency.
  3. max_num_batched_tokens: Tunes chunked prefill token batching (2048, 4096, 8192, or 16384), ordered according to your target_metric (c1_decode, concurrent_decode, prefill, or ttft).
  4. block_size: Evaluates KV cache page block sizes (32 and 64).
  5. prefix_caching: When target_metric is prefill, checks whether disabling automatic prefix caching improves cold ingestion throughput.
  6. Opt-in FP8 (quality: fp8_allowed): By default (quality: lossless), weights and KV cache stay lossless. Setting fp8_allowed additionally evaluates kv_cache_dtype: fp8 (where supported by the model architecture) and quantization: fp8 (for unquantized base models without a LoRA adapter on FP8-capable GPUs such as L4, L40S, H100, H200, and B200).

Each candidate is evaluated against the current best configuration using a 3% improvement gate:

  • accepted: No integrity flags triggered, the target metric improved by more than 3%, and non-target decode or TTFT metrics did not regress past guard thresholds. The candidate becomes the new winning baseline (best_config).
  • rejected: Either an integrity flag triggered, a non-target metric regressed past threshold, or the change landed within 3% noise.

When two consecutive trials land inside measurement noise without gains, or when max_trials is reached, the autoresearch loop cleans up temporary test deployments and marks the task completed. In the console, open Autoresearch and click Create API model on a completed run to pre-fill Models -> Publish with the winning vLLM configuration.

From the CLI

Use the tensorant optimize group to inspect physics ceilings, run Inference Autoresearch, or drive custom benchmarks from shell scripts and CI:

bash
tensorant optimize ceiling --project PROJECT_ID --gpu "NVIDIA L4" --concurrency 8 --context-length 4096
tensorant optimize create --project PROJECT_ID --config optimize.yaml
tensorant optimize start OPTIMIZATION_ID --preview
tensorant optimize start OPTIMIZATION_ID --wait
tensorant optimize show OPTIMIZATION_ID --json

See CLI commands and Advanced serving settings for complete reference details.

Need a hand? Visit troubleshooting