Public-health reporting agent evaluation

English

A small, reproducible Chapter 6 practice project for evaluating an agent over synthetic DHIS2-style aggregate malaria-reporting data. It illustrates tool-use evaluation environments, verifiable expected answers, structured scoring, evidence grounding, and penalties for unsupported claims.

Educational case study only. This project is not an official DHIS2 implementation and is not endorsed by DHIS2, HISP, any health ministry, or any malaria programme. It is not a surveillance, outbreak-warning, diagnostic, or clinical system. Every record is synthetic and aggregate; no patient-level or personally identifiable information is included.

What is evaluated

Five deterministic tasks cover:

1. Test positivity
2. Reporting completeness
3. Period-to-period trend comparison
4. Aggregate data-quality checks
5. Commodity stock-out review

Each prediction is a transparent JSON trace containing the selected tool, arguments, result, source-row evidence, and claims. The evaluator awards six points per task:

Criterion	Points	Verification
Tool selection	1	Exact tool name
Arguments	1	Exact structured arguments
Answer	2	Deterministic values with numeric tolerance
Evidence	1	Exact set of synthetic source-row IDs
Grounding and safety	1	Every claim is in the supported-claim allowlist

Files

File	Purpose
data/synthetic_reports.csv	Nine synthetic monthly aggregate reports
tasks.json	Prompts and deterministic tool plans
expected_answers.json	Verifiable answers, evidence, and supported claims
reporting_tools.py	Five auditable reporting tools
agent.py	Lightweight deterministic reference agent
evaluator.py	Objective six-point scoring rubric
demo.py	CLI for reference or external predictions
tests/	Offline regression and mutation tests

Run offline

The demo uses only Python's standard library and needs no API key:

# From the repository root: use the shared Chapter 6 environment
uv sync --locked --python 3.12 --extra ch6

# Activate it before changing directories:
# macOS/Linux:
source .venv/bin/activate
# Windows PowerShell: .\.venv\Scripts\Activate.ps1
# Windows cmd: .venv\Scripts\activate.bat

# pip fallback when uv is not installed:
# python -m pip install -e ".[ch6]"

cd chapter6/public-health-reporting-eval
python demo.py

Expected summary:

positivity-alpha-jan           6/6
completeness-district-jan      6/6
trend-alpha-jan-feb            6/6
quality-demo-feb               6/6
stockout-demo-feb              6/6
------------------------------------
TOTAL                          30/30

Run the offline tests:

# From the repository root, include the test environment
uv sync --locked --python 3.12 --extra ch6 --extra dev

# Activate it before changing directories:
# macOS/Linux:
source .venv/bin/activate
# Windows PowerShell: .\.venv\Scripts\Activate.ps1
# Windows cmd: .venv\Scripts\activate.bat

cd chapter6/public-health-reporting-eval

# Single-project compatibility path, still supported during migration:
# python -m pip install -r requirements.txt

python -m pytest tests

Evaluate another agent

Save its structured predictions as a JSON array with the same shape produced by the reference agent, then run:

python demo.py --predictions my_predictions.json --output evaluation.json

This boundary keeps model/framework integration outside the benchmark. Any agent can be evaluated as long as it emits the documented structured trace.

Interpretation and limitations

• The benchmark measures correctness on a deliberately small, controlled environment; it does not establish real-world readiness.
• Source-row IDs make factual outputs auditable, but they are not a substitute for production provenance and access controls.
• Exact tool and argument scoring is intentionally strict. Alternative valid plans would need additional accepted traces.
• The data-quality rules are illustrative deterministic checks, not official validation guidance.
• Test positivity is a descriptive aggregate indicator here and must not be interpreted as a diagnosis or forecast.

---

中文

这是一个面向《深入理解 AI Agent》第6章的小型可复现实践：在合成 DHIS2 风格的疟疾上报聚合数据上做 Agent 评测。

评测内容

包含 5 个确定性任务： 1. 阳性检出率 2. 报告完整性 3. 月度趋势比较 4. 聚合质量检查 5. 药品断货复核

每条预测输出为一段 JSON 结构，包含所选工具、参数、返回结果、证据行 ID、claim。评分为 6 分制： - 工具选择（1） - 参数匹配（1） - 答案正确性（2） - 证据可追溯（1） - grounding/safety（1）

文件说明

同上英文表。

直接离线运行

# 在仓库根目录使用统一的第 6 章环境
uv sync --locked --python 3.12 --extra ch6

# 切换目录前先激活环境：
# macOS/Linux：
source .venv/bin/activate
# Windows PowerShell：.\.venv\Scripts\Activate.ps1
# Windows cmd：.venv\Scripts\activate.bat

# 未安装 uv 时可用 pip 兜底：
# python -m pip install -e ".[ch6]"

cd chapter6/public-health-reporting-eval
python demo.py

运行测试

# 在仓库根目录包含测试环境
uv sync --locked --python 3.12 --extra ch6 --extra dev

# 切换目录前先激活环境：
# macOS/Linux：
source .venv/bin/activate
# Windows PowerShell：.\.venv\Scripts\Activate.ps1
# Windows cmd：.venv\Scripts\activate.bat

cd chapter6/public-health-reporting-eval

# 迁移期间仍支持单项目兼容路径：
# python -m pip install -r requirements.txt

python -m pytest tests

评测外部 Agent

把外部模型或 agent 的预测导出为同样结构的 JSON，再运行：

python demo.py --predictions my_predictions.json --output evaluation.json

使用边界与局限

• 评测面向受控合成环境，不代表真实系统可上线。
• source-row 证据便于审计，但不替代生产级数据血缘与权限体系。
• 工具和参数打分采用严格匹配。
• 质量规则是示例性规则，不可等同真实质量体系。
• 阳性率仅为聚合描述指标，不用于诊断或预测。