Prompt Engineering Ablation (τ-bench) / 提示工程消融实验¶

Companion material for AI Agents in Depth, Chapter 2 — Experiment 2-4 ★★: Ablation study in prompt engineering.
配套《深入理解 AI Agent》第 2 章 实验 2-4 ★★：提示工程的消融实验。

← Chapter 2 index / 返回第 2 章目录

English¶

Overview¶

Extends the τ-bench framework with three ablation axes to show why prompt engineering treats an Agent like a smart new hire. Experiments quantify how tone, instruction organization, and tool descriptions affect task success.

Ablation options¶

1. Tone style¶

default — professional baseline
trump — exaggerated, repetitive, confident phrasing
casual — emoji/slang, informal

Rationale: Tone affects professionalism and task quality. Over-casual or exaggerated tone can reduce trust, increase misunderstanding, and hurt execution accuracy.

2. Wiki rule randomization¶

Uses a pre-generated chaotic wiki.md:

Strip section headings/structure
Prefix each rule with operation context (e.g. “When booking flights”)
Fully shuffle into a flat list
Break logical relationships between rules

Rationale: Well-organized instructions are like a training manual. Extreme randomization destroys hierarchy, blurs rule boundaries, and raises misuse/omission risk.

3. Tool description removal¶

Empty tool and parameter descriptions
Tests the value of explicit documentation

Rationale: Clear tool docs are the “how to use the tools” handbook. Without them the Agent misuses tools more often and completion rates drop.

Installation¶

# From the repository root: use the shared Chapter 2 environment
uv sync --locked --python 3.12 --extra ch2

# Activate it before changing directories:
# macOS/Linux:
source .venv/bin/activate
# Windows PowerShell: .venv\Scripts\Activate.ps1
# Windows cmd: .venv\Scripts\activate.bat

# pip fallback when uv is not installed:
# python -m pip install -e ".[ch2]"

cd chapter2/prompt-engineering

# Single-project compatibility path, still supported during migration:
# python -m pip install -r requirements.txt

(Older docs may mention projects/week2/prompt-engineering; use this repo path.)

Usage¶

All entry scripts have Chinese --help: python run_ablation.py --help, python analyze_results.py --help.

One-shot full ablation + comparison table (recommended)¶

--all runs baseline + each single-axis ablation + all combined in one process, prints a success-rate table, and writes summary stats to --output. The frozen canonical protocol uses official Moonshot Kimi K3 for both the action model and user simulator, six arms, and the same ten τ-bench airline tasks in every arm:

export OPENAI_API_KEY="$MOONSHOT_API_KEY"
export OPENAI_API_BASE="https://api.moonshot.cn/v1"
python run_ablation.py \
  --all --model kimi-k3 --user-model kimi-k3 \
  --model-provider openai --user-model-provider openai --temperature 1 \
  --env airline --task-ids 0 1 2 3 4 5 6 7 8 9 \
  --num-trials 1 --seed 20260730 --max-agent-steps 30 \
  --max-concurrency 2 \
  --log-dir runs/exp2-4-kimi-k3-YYYYMMDD-v1 \
  --output runs/exp2-4-kimi-k3-YYYYMMDD-v1/comparison.json \
  --no-verbose

If a campaign stops, resume into a new evidence directory. The runner imports only prior task rows with nonempty provider response IDs/usage and no task error, records the source hash, and never regenerates them:

# Repeat every frozen option above, change --log-dir/--output to ...-v2, and add:
--resume-from runs/exp2-4-kimi-k3-YYYYMMDD-v1

The rejected OpenAI-direct/OpenRouter preflights and any failed tasks remain evidence; they are not converted into zero-score model outcomes. Campaign completion requires every arm/task receipt, objective τ-bench scoring, hashes, usage/cost, and a clean credential scan, regardless of which hypothesis wins.

The completed canonical run is runs/exp2-4-kimi-k3-20260730-v7: all 60 cells have real Kimi K3 action/user receipts and no transport or task errors. Its observed pass counts were baseline 7/10, Trump 6/10, casual 9/10, randomized organization 8/10, no-description 9/10, and all ablations 8/10. These results complete the preregistered experiment but do not reproduce the manuscript's historical “over 30%” and “45%” point estimates; comparison.json records that qualification instead of retrofitting a favorable claim.

Example real smoke table (--model gpt-4o --env airline --end-index 4, only 4 tasks/group—illustrates table shape, not stable science):

Experiment                        Success Rate      Tasks        Relative
----------------------------------------------------------------------
wiki_random                      50.0%         2/  4      200.0%
baseline                         25.0%         1/  4      100.0%  ⭐
tone_trump                       25.0%         1/  4      100.0%
tone_casual                      25.0%         1/  4      100.0%
no_tool_desc                      0.0%         0/  4        0.0%
all_ablations                     0.0%         0/  4        0.0%

⚠️ n=4 per arm is very noisy—e.g. wiki_random above baseline is chance, not a real finding. Directional signals (no tool desc → 0%, full stack → 0%, tone little effect on success) match 实验 2-4; for stable numbers use --end-index ≥ 10 and multiple --seed. Use your full runs, not these smoke digits.

Baseline (single config)¶

python run_ablation.py \
    --model gpt-5.6-luna \
    --env airline \
    --task-split test \
    --start-index 0 \
    --end-index 10
# bare ids → OpenAI direct; ids with '/' → openrouter

Tone ablations¶

python run_ablation.py \
    --model gpt-5.6-luna \
    --env airline \
    --tone-style trump \
    --ablation-name trump_tone

python run_ablation.py \
    --model gpt-5.6-luna \
    --env airline \
    --tone-style casual \
    --ablation-name casual_tone

Wiki randomization¶

python run_ablation.py \
    --model gpt-5.6-luna \
    --env airline \
    --randomize-wiki \
    --ablation-name wiki_random

Remove tool descriptions¶

python run_ablation.py \
    --model gpt-5.6-luna \
    --env airline \
    --remove-tool-descriptions \
    --ablation-name no_tool_desc

Combined ablations¶

python run_ablation.py \
    --model gpt-5.6-luna \
    --env airline \
    --tone-style casual \
    --randomize-wiki \
    --remove-tool-descriptions \
    --ablation-name full_ablation

Experiment scripts¶

Two equivalent ways to run the full suite:

Python one-shot (recommended): python run_ablation.py --env airline --end-index 10 --all
Bash orchestration: run_full_ablation.sh calls run_ablation.py then analyze_results.py:

./run_full_ablation.sh --model gpt-5.6-luna --env airline --num-tasks 10
./run_full_ablation.sh --quick   # 3 tasks/arm smoke

Result analysis¶

Raw trajectories land in results_ablation/ with:

task_id, reward (0/1), info, traj, ablation_config

python analyze_results.py
python analyze_results.py --results-dir results_ablation --output summary.json

--all already prints the comparison table; analyze_results.py is for re-aggregating historical/manual runs. Bundled results_ablation/*.json are small debug samples (1–6 tasks)—not enough for statistical claims; use full runs (--end-index ≥ 10).

Expected ranking¶

Baseline — best
Tone variants — usually little success-rate impact
Wiki randomization — hurts instruction following
No tool descriptions — many bad tool args / wrong ops
Combined — worst

Key insights¶

Treat the Agent as a smart new employee:

Clear instructions matter — structure, task description, tool how-to
Context organization matters — logical order, group related rules, explicit priority
Tool docs are required — purpose, parameters, examples

Parameters¶

Parameter	Description	Options
`--tone-style`	Axis 1: tone on system prompt	default, trump, casual
`--randomize-wiki`	Axis 2: scramble wiki structure	flag
`--remove-tool-descriptions`	Axis 3: strip tool docs	flag
`--all`	Full ablation suite + comparison table	flag
`--output`	Summary JSON path (`--all` only)	string
`--ablation-name`	Run label	string
`--env`	Environment	airline, retail
`--model`	Model id	e.g. gpt-4o-mini, gpt-4o
`--model-provider`	Provider (optional)	auto: bare → openai, `/` → openrouter
`--task-split`	Split	train, test, dev
`--start-index` / `--end-index`	Task range	integers
`--log-dir`	Results directory	string

Troubleshooting¶

ImportError — correct cwd + install deps
API errors — keys and quota
Memory — lower --max-concurrency

Debug:

export LITELLM_LOG=DEBUG
python run_ablation.py ...

Summary¶

Ablations quantify prompt quality: poor structure/docs can cost 30–80% performance. Structure and clarity dominate; professionalism and consistency support effective Agents. Good prompt engineering ≈ good employee training.

Upstream τ-bench (bundled)¶

This tree vendors τ-bench (tool-agent-user interaction benchmark). Upstream news: τ²-bench adds fixes + a telecom domain.

Papers: τ-bench, τ²-Bench

Vanilla τ-bench run (non-ablation path):

python run.py --agent-strategy tool-calling --env retail --model gpt-4o \
  --model-provider openai --user-model gpt-4o --user-model-provider openai \
  --user-strategy llm --max-concurrency 10
# optional: --task-ids 2 4 6

User strategies include llm, react, verify, reflection. See original τ-bench docs for leaderboards, auto error identification, and historical trajectories. License: ./LICENSE.

中文¶

概述¶

扩展 τ-bench 框架，增加三个关键消融维度，演示提示工程：把 Agent 看成聪明的新员工的重要性，并量化语气、指令组织、工具描述对任务成功率的影响。

消融研究选项¶

1. 语气风格¶

default：标准专业语气（基线）
trump：夸张、重复强调、自信表述
casual：表情符号、俚语、轻松口吻

原理： 语气影响专业性与任务质量；过于随意或夸张可能降低信任、增加误解、损害执行准确度。

2. Wiki 规则随机化¶

使用预生成的极度混乱版 wiki：

移除章节标题与结构
每条规则加操作上下文前缀（如 “When booking flights”）
打乱成平面列表
破坏规则间逻辑关系

原理： 组织良好的指令像培训手册；极度随机化破坏层级、混淆规则边界、抬高误用与遗漏风险。

3. 工具描述移除¶

工具与参数描述置空
检验「写清楚怎么用」的重要性

原理： 清晰工具说明像操作手册；去掉后误用上升、完成率下降。

安装¶

# 在仓库根目录使用统一的第 2 章环境
uv sync --locked --python 3.12 --extra ch2

# 切换目录前先激活环境：
# macOS/Linux：
source .venv/bin/activate
# Windows PowerShell：.venv\Scripts\Activate.ps1
# Windows cmd：.venv\Scripts\activate.bat

# 未安装 uv 时可用 pip 兜底：
# python -m pip install -e ".[ch2]"

cd chapter2/prompt-engineering

# 迁移期间仍支持单项目兼容路径：
# python -m pip install -r requirements.txt

（旧文档可能写 projects/week2/prompt-engineering；请使用本仓库路径。）

使用方法¶

入口脚本均提供中文 --help：python run_ablation.py --help、python analyze_results.py --help。

一键完整消融并输出对比表（推荐）¶

--all 在同一进程内依次跑基线 + 三个维度单独消融 + 全部叠加，打印成功率对比表，汇总写入 --output（默认 log-dir/ablation_summary_<时间戳>.json）。复现书中实验 2-4 最直接：

python run_ablation.py \
    --model gpt-5.6-luna \
    --env airline \
    --end-index 10 \
    --all
# 默认 OpenAI 直连（provider=openai），需 OPENAI_API_KEY。
# 走 OpenRouter：模型写成带斜杠 id（如 openai/gpt-5），需 OPENROUTER_API_KEY。
# 通用回退：裸 id（如 gpt-4o-mini）且未设 OPENAI_API_KEY、已设 OPENROUTER_API_KEY 时，
#           自动前缀为 openai/gpt-4o-mini 并切到 openrouter。

真实冒烟表示例（--model gpt-4o --env airline --end-index 4，每组仅 4 任务，只用于展示表格形态）：

Experiment                        Success Rate      Tasks        Relative
----------------------------------------------------------------------
wiki_random                      50.0%         2/  4      200.0%
baseline                         25.0%         1/  4      100.0%  ⭐
tone_trump                       25.0%         1/  4      100.0%
tone_casual                      25.0%         1/  4      100.0%
no_tool_desc                      0.0%         0/  4        0.0%
all_ablations                     0.0%         0/  4        0.0%

⚠️ 每组 4 任务噪声极大——例如 wiki_random 偶然高于 baseline 不是真实结论。方向性信号（去掉工具描述 → 0%、全部叠加 → 0%、语气对成功率影响小）与实验 2-4 一致；要稳定量化请把 --end-index 提到 10 以上并多跑 --seed。以你自己的完整运行为准。

基线（单配置）¶

python run_ablation.py \
    --model gpt-5.6-luna \
    --env airline \
    --task-split test \
    --start-index 0 \
    --end-index 10
# 裸 id → OpenAI 直连；带 / 的 id → openrouter

语气消融¶

python run_ablation.py \
    --model gpt-5.6-luna \
    --env airline \
    --tone-style trump \
    --ablation-name trump_tone

python run_ablation.py \
    --model gpt-5.6-luna \
    --env airline \
    --tone-style casual \
    --ablation-name casual_tone

Wiki 随机化¶

python run_ablation.py \
    --model gpt-5.6-luna \
    --env airline \
    --randomize-wiki \
    --ablation-name wiki_random

移除工具描述¶

python run_ablation.py \
    --model gpt-5.6-luna \
    --env airline \
    --remove-tool-descriptions \
    --ablation-name no_tool_desc

组合消融¶

python run_ablation.py \
    --model gpt-5.6-luna \
    --env airline \
    --tone-style casual \
    --randomize-wiki \
    --remove-tool-descriptions \
    --ablation-name full_ablation

实验脚本¶

完整套消融有两种等价方式：

Python 一键（推荐）： python run_ablation.py --env airline --end-index 10 --all
Bash 编排： run_full_ablation.sh 逐个调用 run_ablation.py 再 analyze_results.py：

./run_full_ablation.sh --model gpt-5.6-luna --env airline --num-tasks 10
./run_full_ablation.sh --quick   # 每组 3 任务冒烟

结果分析¶

原始轨迹在 results_ablation/，含 task_id、reward（0/1）、info、traj、ablation_config。

python analyze_results.py
python analyze_results.py --results-dir results_ablation --output summary.json

--all 结束时已打印对比表；analyze_results.py 用于事后重汇总。仓库内 results_ablation/*.json 为少量调试样本（1–6 任务），不足以做统计结论；请用完整运行（--end-index ≥ 10）。

预期排序¶

Baseline — 最佳
语气变化 — 通常对成功率影响不大
Wiki 随机化 — 严重损害指令遵循
无工具描述 — 大量参数错误 / 错误操作
组合消融 — 最差

关键洞察¶

把 Agent 看成聪明的新员工：

清晰指令至关重要 — 结构化信息、任务描述、工具用法
上下文组织影响理解 — 逻辑排序、相关规则归并、优先级明确
工具文档不可或缺 — 用途、参数、示例

参数说明¶

参数	说明	选项
`--tone-style`	维度一·语气风格	default, trump, casual
`--randomize-wiki`	维度二·随机化 wiki 结构	flag
`--remove-tool-descriptions`	维度三·移除工具描述	flag
`--all`	一键完整消融并打印对比表	flag
`--output`	（仅 --all）汇总 JSON 路径	string
`--ablation-name`	实验名称标识	string
`--env`	环境	airline, retail
`--model`	模型	如 gpt-4o-mini, gpt-4o
`--model-provider`	提供商（可选）	自动：裸 id → openai，带 / → openrouter
`--task-split`	任务集	train, test, dev
`--start-index` / `--end-index`	任务区间	整数
`--log-dir`	结果目录	string

故障排除¶

ImportError：确认目录与依赖
API 错误：密钥与配额
内存：降低 --max-concurrency

export LITELLM_LOG=DEBUG
python run_ablation.py ...

总结¶

消融框架量化展示：提示工程不当时可出现 30–80% 的性能下滑；结构与清晰度最关键；专业性与一致性支撑有效 Agent 系统。记住：优秀的提示工程就是优秀的员工培训。

上游 τ-bench（内嵌）¶

本目录内嵌 τ-bench（工具-Agent-用户交互基准）。上游进展：τ²-bench 含修复与 telecom 域。

论文： τ-bench、τ²-Bench

原版（非消融）运行：

python run.py --agent-strategy tool-calling --env retail --model gpt-4o \
  --model-provider openai --user-model gpt-4o --user-model-provider openai \
  --user-strategy llm --max-concurrency 10
# 可选：--task-ids 2 4 6

用户模拟策略含 llm、react、verify、reflection。排行榜、自动错误识别、历史轨迹等见原版 τ-bench 文档。许可：./LICENSE。

Notes / 说明¶

Book experiment path is run_ablation.py; vanilla run.py is the upstream τ-bench entry.
书中实验主路径是 run_ablation.py；run.py 为上游 τ-bench 原版入口。
Smoke tables in this README are not publishable success rates.
文中冒烟表不可当作可发表的成功率数字。