Learning from Experience: RL vs LLM In-Context Learning / 从经验中学习：RL 与 LLM 上下文学习对比¶

Compares tabular Q-learning with LLM in-context learning on a treasure-hunt game with hidden mechanics (Shunyu Yao, “The Second Half”).
代码位于第 1 章项目树；对应书中 实验 7-1 ★（Q-learning 在寻宝游戏中的表现） 与 实验 7-2 ★★（传统 RL 与 LLM Agent 的对比研究）。

← Chapter 1 index / 返回第 1 章目录 · 📖 Read Chapter 7 / 读第 7 章正文（EN）

English¶

Overview¶

This experiment compares traditional Reinforcement Learning (Q-learning) with LLM-based in-context learning, replicating the key insights from Shunyu Yao's blog post "The Second Half".

It demonstrates how LLMs can generalize through reasoning while traditional RL methods require extensive training to learn game mechanics. We use a text-based treasure hunt game with hidden mechanics that agents must discover through experience.

Key Insights Being Tested¶

Sample Efficiency: LLMs can learn from far fewer examples than traditional RL
Generalization: LLMs use reasoning to understand patterns, while RL memorizes state-action mappings
Prior Knowledge: Language pre-training provides powerful priors for reasoning about new tasks
Hidden Mechanics Discovery: LLMs can form hypotheses and test them, while RL requires exhaustive exploration

What You'll See¶

When running the LLM experiment, you'll see the complete decision-making process:

============================================================
LLM DECISION PROCESS
============================================================
📊 Experiences in memory: 15
🎮 Current room: hallway
🎯 Available actions: 8

💡 Recent successful patterns learned:
   • take red key → +5.0 reward
   • try crafting → +10.0 reward

🤔 LLM is thinking...

📝 LLM Reasoning:
----------------------------------------
  Based on my past experiences, I've learned that:
  1. The red key opens the locked door to the guard room
  2. Crafting rusty sword + magic crystal creates a silver sword
  3. The silver sword can defeat the strong guard

  Since I have the silver sword and I'm in the hallway...
----------------------------------------

✅ Chosen action: go north

This transparency shows exactly how the LLM learns and reasons, unlike the black-box nature of Q-learning.

The Game¶

A text-based treasure hunt game where agents must:

Navigate through multiple rooms
Collect items and keys
Defeat guards using appropriate weapons
Discover hidden mechanics through experience

Hidden Mechanics (Not Revealed to Agents)¶

Color-coded locks: Specific colored keys open matching doors
Weapon effectiveness: Different weapons work against different enemies
Crafting system: Certain items combine to create better items
Potion effects: Temporary abilities from consuming potions

Quick Start¶

Installation¶

# Recommended from the repository root: use the shared Chapter 1 environment
uv sync --locked --extra ch1

# Activate it before changing directories:
# macOS/Linux:
source .venv/bin/activate
# Windows PowerShell: .venv\Scripts\Activate.ps1
# Windows cmd: .venv\Scripts\activate.bat

# pip fallback when uv is not installed:
# python -m pip install -e ".[ch1]"

# Enter this experiment directory for the commands below
cd chapter1/learning-from-experience

# Single-project compatibility path, still supported during migration:
# python -m pip install -r requirements.txt

Q-learning runs fully offline with no API key.
The LLM path needs a Moonshot/Kimi API key (or OpenRouter fallback).

Setting up Kimi K3 API¶

To run the LLM experiments, you need a Kimi (Moonshot) API key:

Get your API key from Moonshot AI
Set the environment variable:

export MOONSHOT_API_KEY="your-api-key-here"

Or create a .env file:

echo "MOONSHOT_API_KEY=your-api-key-here" > .env

Universal OpenRouter fallback: if MOONSHOT_API_KEY is unset but OPENROUTER_API_KEY is set, the LLM path routes through OpenRouter. Because Kimi models are not stably available on OpenRouter, the fallback uses OPENROUTER_MODEL (default openai/gpt-5.6-luna):

export OPENROUTER_API_KEY=your-openrouter-api-key
python quick_demo.py   # runs via OpenRouter when MOONSHOT_API_KEY is missing

Running the Experiment¶

Quick Demo (See LLM Learning in Action)¶

python quick_demo.py

This shows a detailed view of how the LLM learns through reasoning, displaying:

Complete thought process for each decision
How experiences accumulate and influence future decisions
The dramatic difference in learning speed vs traditional RL

Command-Line Interface (`experiment.py`)¶

experiment.py provides a full CLI (Chinese help text). List all flags:

python experiment.py --help

Main parameters:

Parameter	Description	Default
`--mode {both,qlearning,rl,llm}`	Which agent(s): `qlearning`/`rl` = Q-learning only (offline), `llm` = LLM Agent only, `both` = comparison	`both`
`--rl-episodes`	Q-learning training episodes (Experiment 7-1 uses 10000)	`10000`
`--llm-episodes`	LLM Agent training episodes	`20`
`--eval-episodes`	Greedy evaluation episodes after Q-learning	`100`
`--checkpoint-interval`	Learning-curve sample interval (every N episodes)	`1000`
`--model`	LLM model name (or `MOONSHOT_MODEL` env)	`kimi-k3`
`--output`	Results output directory	`results`
`--seed`	Random seed for reproducible Q-learning curves	unset
`--learning-rate` / `--discount` / `--epsilon-decay` / `--epsilon-min`	Q-learning hyperparameters	`0.2 / 0.99 / 0.9995 / 0.1`
`--stochastic`	Use stochastic environment	deterministic
`--skip-llm`	Legacy alias for `--mode qlearning`	—

Q-Learning Only (Experiment 7-1, offline, no API)¶

python experiment.py --mode qlearning --rl-episodes 10000 --seed 42

Training finishes in under ~3 seconds and prints a learning curve table showing how the agent goes from ~0% win rate to mastery over nearly 10k episodes (see Results below).

Full Comparison (RL vs LLM, Experiment 7-2)¶

python experiment.py --mode both --model kimi-k3

For the exact book protocol and acceptance-grade evidence, use the canonical runner. It executes the 10,000-episode Q-learning arm, 100 greedy evaluation episodes, and exactly one first-attempt official Moonshot Kimi K3 trajectory:

python run_experiment_7_2.py

The canonical runner rejects OpenRouter substitution, API errors, missing raw provider response IDs/content, and any parser fallback. It writes validation/<timestamp>/evidence.json; if only post-run serialization needs to be repaired, finalize_experiment_7_2.py <campaign-dir> finalizes the already saved raw campaign without repeating paid model calls.

This will:

Train a Q-learning agent for 10000 episodes (~3 seconds) and print its learning curve
Train an LLM agent for 20 episodes with detailed reasoning display
Evaluate both agents
Generate comparison plots
Save results to the results/ directory

Note: experiment.py is the exploratory multi-episode runner. The canonical book campaign is deliberately one first attempt. The accepted 2026-07-30 Kimi K3 attempt took 416.11 seconds for 17 sequential reasoning calls; the earlier “1–2 minutes per game” estimate was not reproduced on this route.

LLM Only¶

python experiment.py --mode llm --llm-episodes 20

Interactive Game Play¶

Test the game manually:

from game_environment import TreasureHuntGame

game = TreasureHuntGame()
print(game.get_state_description())
print("Available actions:", game.get_available_actions())

# Try an action
feedback, reward, done = game.execute_action("take rusty sword")
print(f"Feedback: {feedback}")
print(f"Reward: {reward}")

Validation¶

Install the dev extra from the repository root before running pytest in a clean environment:

uv sync --locked --extra ch1 --extra dev

# Activate it before changing directories:
# macOS/Linux:
source .venv/bin/activate
# Windows PowerShell: .venv\Scripts\Activate.ps1
# Windows cmd: .venv\Scripts\activate.bat

cd chapter1/learning-from-experience
python -m pytest tests

The longer Q-learning learning-curve check is an offline manual smoke script, kept out of default pytest discovery:

python tests/manual/rl_learning_check.py --episodes 1000

Experiment Results¶

Metrics Compared¶

Sample Efficiency — Episodes needed to achieve good performance; learning speed
Performance — Victory rate in evaluation; average rewards and episode lengths
Computational Cost — Training time; memory (Q-table size vs. experience storage); API calls for LLM

Visualizations¶

The experiment creates comparison plots showing:

Learning curves over time
Victory rate progression
Sample efficiency comparison
Key insights summary

Expected Results¶

Q-Learning learning curve (measured locally, `--mode qlearning --rl-episodes 10000 --seed 42`)¶

Measured curve (deterministic env; victory rate as a sliding window over the last 1000 episodes; full training ~3s):

Episodes	Victory rate	Q-table states	epsilon
1000	0.3%	123	0.606
2000	0.0%	123	0.368
3000	0.1%	126	0.223
5000	0.1%	128	0.100
7000	97.0%	138	0.100
8000	99.6%	138	0.100
9000	99.8%	139	0.100
10000	98.1%	142	0.100

After training, the canonical greedy evaluation reaches 100% win rate, averaging 12 steps. The accepted Kimi K3 arm won on its first attempt in 17 steps with 17/17 real responses, zero API errors, zero fallbacks, and 28,242 tokens. This reproduces the first-attempt conclusion but not the manuscript's historical point estimates of exactly 18 Kimi steps and an 11-step Q-learning solution. See the canonical evidence.

RL vs LLM (Experiment 7-2 conclusions)¶

Q-Learning: Needs ~10000 episodes for stable clears; treats “door / key / sword” as meaningless symbols and only explores statistically.
LLM In-Context: Carries pretrained priors; often clears in the first episode within tens of steps by reasoning about game concepts.
Sample Efficiency: LLM is 2–3 orders of magnitude more sample-efficient; but per-episode inference is slow (~1–2 min API), while Q-learning finishes 10000 episodes in ~3s—trade-off depends on interaction cost (see book Experiment 7-2).

Project Structure¶

learning-from-experience/
├── game_environment.py    # Text-based game with hidden mechanics
├── rl_agent.py            # Q-learning implementation
├── llm_agent.py           # LLM with in-context learning
├── experiment.py          # Main experiment runner
├── demo.py                # Interactive local game demo
├── quick_demo.py          # Short LLM learning demo
├── run_experiment_7_2.py  # Exact real campaign + acceptance gates
├── finalize_experiment_7_2.py # Evidence-only recovery; no API rerun
├── env.example            # Optional API-key template
├── tests/
│   ├── test_basic.py
│   ├── test_zero_episodes.py
│   ├── test_rl_progress_small_episodes.py
│   └── manual/
│       └── rl_learning_check.py
├── requirements.txt       # Python dependencies
├── README.md              # This file
└── results/               # Experiment outputs (created on run)
    └── [timestamp]/
        ├── rl_agent.pkl           # Trained Q-learning agent
        ├── llm_experiences.json   # LLM's collected experiences
        ├── experiment_results.json # Numerical results
        └── comparison_plots.png   # Visualization

Technical Details¶

Q-Learning Agent¶

Algorithm: Tabular Q-learning with ε-greedy exploration
State Representation: Hashed combination of room, inventory, and game state
Learning Rate: 0.2 (configurable via --learning-rate)
Discount Factor: 0.99 (configurable via --discount)
Exploration: ε starts at 1.0, decays by --epsilon-decay (0.9995) to --epsilon-min (0.1)

LLM Agent (Kimi K3)¶

Model: kimi-k3 (override with --model or MOONSHOT_MODEL)
Reasoning model: Kimi K3 emits a chain-of-thought (message.reasoning_content) before its final answer (message.content), so the code uses a generous max_tokens=2048 to make sure the ACTION: line is not truncated by the reasoning budget.
Learning Method: In-context learning with experience memory (up to 50 experiences)
Context Management: Stores successful and failed experiences
Reasoning: Prompts LLM to reason about past experiences before acting
Temperature: requested 0.7, but reasoning models (Kimi K3, GPT-5) only accept temperature=1, so the code auto-forces 1 for those (see _reasoning_safe_temperature)

Extending the Experiment¶

Ideas for Further Research¶

Different Games: Try other hidden-mechanic games
Hybrid Approaches: Combine RL with LLM guidance
Transfer Learning: Test how well agents transfer to similar games
Ablation Studies: Remove reasoning prompts to isolate their impact
Other LLMs: Compare different language models

Modifying the Game¶

Edit game_environment.py to:

Add new rooms and items
Create more complex hidden mechanics
Adjust difficulty and rewards
Add new types of puzzles

Educational Value¶

The Power of Priors: How language pre-training provides useful knowledge
Reasoning vs. Memorization: Different approaches to learning
Sample Efficiency: Why it matters for real-world applications
The Second Half Thesis: Moving from “can we solve it?” to “how efficiently?”

References¶

The Second Half by Shunyu Yao
ReAct: Synergizing Reasoning and Acting in Language Models
Original Q-learning paper: Watkins & Dayan (1992)

中文¶

概述¶

本实验对比传统强化学习（Q-learning）与基于 LLM 的上下文学习（in-context learning），复现 Shunyu Yao 博客 “The Second Half” 中的核心洞见。

目标：展示 LLM 如何通过推理泛化，而传统 RL 往往需要大量试错才能学到游戏机制。我们使用带有隐藏机制的文本寻宝游戏，智能体只能通过经验去发现规则。

代码在 chapter1/learning-from-experience/，对应书中实验 7-1 与 实验 7-2（正文见第 7 章）。

要验证的关键洞察¶

样本效率：LLM 用远少于传统 RL 的样例即可学习
泛化：LLM 用推理理解模式；RL 记忆状态-动作映射
先验知识：语言预训练为新任务推理提供强大先验
隐藏机制发现：LLM 可形成假设并检验；RL 往往需要穷尽式探索

你会看到什么¶

运行 LLM 实验时，会看到完整决策过程：

============================================================
LLM DECISION PROCESS
============================================================
📊 Experiences in memory: 15
🎮 Current room: hallway
🎯 Available actions: 8

💡 Recent successful patterns learned:
   • take red key → +5.0 reward
   • try crafting → +10.0 reward

🤔 LLM is thinking...

📝 LLM Reasoning:
----------------------------------------
  Based on my past experiences, I've learned that:
  1. The red key opens the locked door to the guard room
  2. Crafting rusty sword + magic crystal creates a silver sword
  3. The silver sword can defeat the strong guard

  Since I have the silver sword and I'm in the hallway...
----------------------------------------

✅ Chosen action: go north

这种透明度展示了 LLM 如何学习与推理，有别于 Q-learning 的黑盒性质。

游戏说明¶

文本寻宝游戏，智能体需要：

在多个房间间导航
收集物品与钥匙
使用合适武器击败守卫
通过经验发现隐藏机制

隐藏机制（不对智能体公开）¶

颜色锁：特定颜色钥匙开对应门
武器有效性：不同武器对不同敌人有效
合成系统：特定物品可合成更强物品
药水效果：消耗药水获得临时能力

快速开始¶

安装¶

# 推荐在仓库根目录使用统一的第 1 章环境
uv sync --locked --extra ch1

# 切换目录前先激活环境：
# macOS/Linux：
source .venv/bin/activate
# Windows PowerShell：.venv\Scripts\Activate.ps1
# Windows cmd：.venv\Scripts\activate.bat

# 未安装 uv 时可用 pip 兜底：
# python -m pip install -e ".[ch1]"

# 进入本实验目录，后续命令都在这里运行
cd chapter1/learning-from-experience

# 迁移期间仍支持单项目兼容路径：
# python -m pip install -r requirements.txt

Q-learning 完全离线，无需任何 API Key。
LLM 部分需要 Moonshot/Kimi API Key（或 OpenRouter 兜底）。

配置 Kimi K3 API¶

运行 LLM 实验需要 Kimi（Moonshot）API Key：

从 Moonshot AI 获取 Key
设置环境变量：

export MOONSHOT_API_KEY="your-api-key-here"

或创建 .env：

echo "MOONSHOT_API_KEY=your-api-key-here" > .env

通用兜底（OpenRouter）：若未设置 MOONSHOT_API_KEY 但设置了 OPENROUTER_API_KEY，LLM 部分会自动改走 OpenRouter。由于 Kimi 模型在 OpenRouter 上不稳定可用，兜底时会使用 OPENROUTER_MODEL（默认 openai/gpt-5.6-luna）：

export OPENROUTER_API_KEY=your-openrouter-api-key
python quick_demo.py   # MOONSHOT_API_KEY 缺失时自动经 OpenRouter 运行

运行实验¶

快速演示（观察 LLM 如何学习）¶

python quick_demo.py

会展示：

每一步的完整思考过程
经验如何累积并影响后续决策
与传统 RL 在学习速度上的巨大差异

命令行接口（`experiment.py`）¶

experiment.py 提供带中文帮助的完整 CLI：

python experiment.py --help

主要参数：

参数	说明	默认值
`--mode {both,qlearning,rl,llm}`	运行哪种智能体：`qlearning`/`rl` 只跑 Q-learning（离线）、`llm` 只跑 LLM Agent、`both` 两者对比	`both`
`--rl-episodes`	Q-learning 训练局数（实验 7-1 用 10000）	`10000`
`--llm-episodes`	LLM Agent 训练局数	`20`
`--eval-episodes`	Q-learning 训练后贪婪评估局数	`100`
`--checkpoint-interval`	学习曲线采样间隔（每 N 局记录一次胜率/Q 表规模）	`1000`
`--model`	LLM 模型名（也可用 `MOONSHOT_MODEL` 环境变量）	`kimi-k3`
`--output`	结果输出目录	`results`
`--seed`	随机种子，用于复现 Q-learning 学习曲线	不固定
`--learning-rate` / `--discount` / `--epsilon-decay` / `--epsilon-min`	Q-learning 超参数	`0.2 / 0.99 / 0.9995 / 0.1`
`--stochastic`	使用随机环境	确定性
`--skip-llm`	兼容旧用法，等价于 `--mode qlearning`	—

仅 Q-Learning（实验 7-1，离线，无需 API）¶

python experiment.py --mode qlearning --rl-episodes 10000 --seed 42

训练不到 3 秒即可完成，并打印学习曲线表格，直观展现智能体如何在近万局试错中从 0% 胜率逐步学会通关（见下文“实验结果”）。

完整对比（RL vs LLM，实验 7-2）¶

python experiment.py --mode both --model kimi-k3

严格复现正文协议并生成可验收证据，请运行：

python run_experiment_7_2.py

该入口固定运行 10,000 局 Q-learning、100 局贪婪评估和且仅一局官方 Moonshot kimi-k3 首次尝试；OpenRouter 替代、API 错误、缺失原始响应 ID/正文或任何 parser fallback 都会使验收失败。若模型调用已经完成、仅证据序列化失败，可运行 python finalize_experiment_7_2.py <campaign-dir>，直接从已保留的原始结果完成证据，不重复付费调用。

流程：

训练 Q-learning 10000 局（约 3 秒）并打印学习曲线
训练 LLM Agent 20 局，并展示详细推理
评估双方
生成对比图
结果写入 results/

说明：experiment.py 是多局探索入口；正文规范入口只测第一局。2026-07-30 验收运行包含 17 次串行推理调用，共耗时 416.11 秒，因此没有复现旧版 “每局 1–2 分钟”的估计。

仅 LLM¶

python experiment.py --mode llm --llm-episodes 20

交互式试玩¶

from game_environment import TreasureHuntGame

game = TreasureHuntGame()
print(game.get_state_description())
print("Available actions:", game.get_available_actions())

# Try an action
feedback, reward, done = game.execute_action("take rusty sword")
print(f"Feedback: {feedback}")
print(f"Reward: {reward}")

验证¶

在干净环境中运行 pytest 前，先在仓库根目录安装 dev extra：

uv sync --locked --extra ch1 --extra dev

# 切换目录前先激活环境：
# macOS/Linux：
source .venv/bin/activate
# Windows PowerShell：.venv\Scripts\Activate.ps1
# Windows cmd：.venv\Scripts\activate.bat

cd chapter1/learning-from-experience
python -m pytest tests

较长的 Q-learning 学习曲线检查是离线手动 smoke 脚本，不会被默认 pytest 收集：

python tests/manual/rl_learning_check.py --episodes 1000

实验结果¶

对比指标¶

样本效率 — 达到良好表现所需局数、学习速度
性能 — 评估胜率、平均回报与回合长度
计算成本 — 训练时间；内存（Q 表规模 vs. 经验存储）；LLM 的 API 调用

可视化¶

实验会生成对比图，包括：

随时间的学习曲线
胜率演进
样本效率对比
关键洞察摘要

预期结果¶

Q-learning 学习曲线（本地实测，`--mode qlearning --rl-episodes 10000 --seed 42`）¶

实测学习曲线（确定性环境，胜率按最近 1000 局滑动窗口统计，整段训练约 3 秒）：

Episodes	Victory rate	Q-table states	epsilon
1000	0.3%	123	0.606
2000	0.0%	123	0.368
3000	0.1%	126	0.223
5000	0.1%	128	0.100
7000	97.0%	138	0.100
8000	99.6%	138	0.100
9000	99.8%	139	0.100
10000	98.1%	142	0.100

规范运行训练后 100 局贪婪评估胜率为 100%，平均 12 步通关；Kimi K3 第一局 17 步通关，保留 17/17 条官方响应，零 API 错误、零 fallback，共 28,242 tokens。它复现了“第一局成功”的实质结论，但没有复现历史记录中的 Kimi 恰好 18 步和 Q-learning 恰好 11 步。详见规范证据。

RL vs LLM（实验 7-2 的对比结论）¶

Q-Learning：需要近 10000 局才达到稳定通关；把“门/钥匙/剑”当作无意义符号，只能靠统计式暴力探索。
LLM In-Context：携带预训练先验，往往第一局就能在十几步内通关；靠推理理解游戏概念结构。
样本效率：LLM 高出 2–3 个数量级；但单局推理慢（API 调用 ~1–2 分钟），Q-learning 跑 10000 局只需约 3 秒——权衡取决于交互成本，详见书中实验 7-2。

项目结构¶

learning-from-experience/
├── game_environment.py    # Text-based game with hidden mechanics
├── rl_agent.py            # Q-learning implementation
├── llm_agent.py           # LLM with in-context learning
├── experiment.py          # Main experiment runner
├── demo.py                # Interactive local game demo
├── quick_demo.py          # Short LLM learning demo
├── run_experiment_7_2.py  # 正文规范实测与验收门
├── finalize_experiment_7_2.py # 仅补写证据，不重复 API 调用
├── env.example            # Optional API-key template
├── tests/
│   ├── test_basic.py
│   ├── test_zero_episodes.py
│   ├── test_rl_progress_small_episodes.py
│   └── manual/
│       └── rl_learning_check.py
├── requirements.txt       # Python dependencies
├── README.md              # This file
└── results/               # Experiment outputs (created on run)
    └── [timestamp]/
        ├── rl_agent.pkl           # Trained Q-learning agent
        ├── llm_experiences.json   # LLM's collected experiences
        ├── experiment_results.json # Numerical results
        └── comparison_plots.png   # Visualization

技术细节¶

Q-Learning Agent¶

算法：表格 Q-learning + ε-贪婪探索
状态表示：房间、背包与游戏状态的哈希组合
学习率：0.2（--learning-rate）
折扣因子：0.99（--discount）
探索：ε 从 1.0 起，按 --epsilon-decay（0.9995）衰减到 --epsilon-min（0.1）

LLM Agent（Kimi K3）¶

模型：kimi-k3（可用 --model 或 MOONSHOT_MODEL 覆盖）
推理模型：Kimi K3 会在最终答案（message.content）前输出思维链（message.reasoning_content），因此代码使用较大的 max_tokens=2048，避免 ACTION: 行被思考预算截断。
学习方式：上下文学习 + 经验记忆（最多 50 条）
上下文管理：存储成功与失败经验
推理：行动前提示模型基于过往经验推理
Temperature：请求 0.7，但推理模型（Kimi K3、GPT-5）只接受 temperature=1，代码会自动强制为 1（见 _reasoning_safe_temperature）

扩展实验¶

进一步研究思路¶

不同游戏：尝试其他隐藏机制游戏
混合方法：RL 与 LLM 引导结合
迁移学习：测试向相似游戏的迁移
消融研究：去掉推理提示以隔离其影响
其他 LLM：对比不同语言模型

修改游戏¶

编辑 game_environment.py 可：

增加房间与物品
设计更复杂的隐藏机制
调整难度与奖励
加入新类型谜题

教学价值¶

先验的力量：语言预训练如何提供有用知识
推理 vs. 记忆：不同学习路径
样本效率：为何对现实任务重要
“The Second Half” 论点：从“能否解决”转向“多高效”

参考文献¶

The Second Half — Shunyu Yao
ReAct: Synergizing Reasoning and Acting in Language Models
Q-learning 原文：Watkins & Dayan (1992)

Notes / 说明¶

Project type: ✅ standalone runnable (Q-learning offline; LLM needs API key).
项目类型：✅ 可独立运行（Q-learning 离线；LLM 需 API Key）。
For educational purposes; inspired by academic work on AI and RL.
教学用途，灵感来自 AI 与强化学习相关研究。
Feel free to add mechanics, other RL algorithms (DQN, PPO, …), providers, or richer metrics.
欢迎增加隐藏机制、其他 RL 算法、提供商或更完善的评估指标。

Learning from Experience: RL vs LLM In-Context Learning / 从经验中学习：RL 与 LLM 上下文学习对比¶

English¶

Overview¶

Key Insights Being Tested¶

What You'll See¶

The Game¶

Hidden Mechanics (Not Revealed to Agents)¶

Quick Start¶

Installation¶

Setting up Kimi K3 API¶

Running the Experiment¶

Quick Demo (See LLM Learning in Action)¶

Command-Line Interface (experiment.py)¶

Q-Learning Only (Experiment 7-1, offline, no API)¶

Full Comparison (RL vs LLM, Experiment 7-2)¶

LLM Only¶

Interactive Game Play¶

Validation¶

Experiment Results¶

Metrics Compared¶

Visualizations¶

Expected Results¶

Q-Learning learning curve (measured locally, --mode qlearning --rl-episodes 10000 --seed 42)¶

RL vs LLM (Experiment 7-2 conclusions)¶

Project Structure¶

Technical Details¶

Q-Learning Agent¶

LLM Agent (Kimi K3)¶

Extending the Experiment¶

Ideas for Further Research¶

Modifying the Game¶

Educational Value¶

References¶

中文¶

概述¶

要验证的关键洞察¶

你会看到什么¶

游戏说明¶

隐藏机制（不对智能体公开）¶

快速开始¶

安装¶

配置 Kimi K3 API¶

运行实验¶

快速演示（观察 LLM 如何学习）¶

命令行接口（experiment.py）¶

仅 Q-Learning（实验 7-1，离线，无需 API）¶

完整对比（RL vs LLM，实验 7-2）¶

仅 LLM¶

交互式试玩¶

验证¶

实验结果¶

对比指标¶

可视化¶

预期结果¶

Q-learning 学习曲线（本地实测，--mode qlearning --rl-episodes 10000 --seed 42）¶

RL vs LLM（实验 7-2 的对比结论）¶

项目结构¶

技术细节¶

Q-Learning Agent¶

LLM Agent（Kimi K3）¶

扩展实验¶

进一步研究思路¶

修改游戏¶

教学价值¶

参考文献¶

Notes / 说明¶

Command-Line Interface (`experiment.py`)¶

Q-Learning learning curve (measured locally, `--mode qlearning --rl-episodes 10000 --seed 42`)¶

命令行接口（`experiment.py`）¶

Q-learning 学习曲线（本地实测，`--mode qlearning --rl-episodes 10000 --seed 42`）¶