判定処理の修正

This commit is contained in:
nrslib 2026-02-13 06:11:06 +09:00
parent 034bd059b3
commit 4919bc759f
41 changed files with 729 additions and 283 deletions

View File

@ -19,11 +19,11 @@
"description": "Instruction for the part agent"
},
"timeout_ms": {
"type": "integer",
"type": ["integer", "null"],
"description": "Optional timeout in ms"
}
},
"required": ["id", "title", "instruction"],
"required": ["id", "title", "instruction", "timeout_ms"],
"additionalProperties": false
}
}

View File

@ -10,6 +10,6 @@
"description": "Why this condition was matched"
}
},
"required": ["matched_index"],
"required": ["matched_index", "reason"],
"additionalProperties": false
}

View File

@ -10,6 +10,6 @@
"description": "Brief justification for the decision"
}
},
"required": ["step"],
"required": ["step", "reason"],
"additionalProperties": false
}

View File

@ -0,0 +1,127 @@
# Structured Output — Phase 3 ステータス判定
## 概要
Phase 3ステータス判定において、エージェントの出力を structured outputJSON スキーマ)で取得し、ルールマッチングの精度と信頼性を向上させる。
## プロバイダ別の挙動
| プロバイダ | メソッド | 仕組み |
|-----------|---------|--------|
| Claude | `structured_output` | SDK が `StructuredOutput` ツールを自動追加。エージェントがツール経由で `{ step, reason }` を返す |
| Codex | `structured_output` | `TurnOptions.outputSchema` で API レベルの JSON 制約。テキストが JSON になる |
| OpenCode | `structured_output` | プロンプト末尾に JSON スキーマ付き出力指示を注入。テキストレスポンスから `parseStructuredOutput()` で JSON を抽出 |
## フォールバックチェーン
`judgeStatus()` は3段階の独立した LLM 呼び出しでルールをマッチする。
```
Stage 1: structured_output — outputSchema 付き LLM 呼び出し → structuredOutput.step1-based integer
Stage 2: phase3_tag — outputSchema なし LLM 呼び出し → content 内の [MOVEMENT:N] タグ検出
Stage 3: ai_judge — evaluateCondition() による AI 条件評価
```
各ステージは専用のインストラクションで LLM に問い合わせる。Stage 1 は「ルール番号を JSON で返せ」、Stage 2 は「タグを1行で出力せよ」と聞き方が異なる。
セッションログには `toJudgmentMatchMethod()` で変換された値が記録される。
| 内部メソッド | セッションログ |
|-------------|--------------|
| `structured_output` | `structured_output` |
| `phase3_tag` / `phase1_tag` | `tag_fallback` |
| `ai_judge` / `ai_judge_fallback` | `ai_judge` |
## インストラクション分岐
Phase 3 テンプレート(`perform_phase3_message`)は `structuredOutput` フラグで2つのモードを持つ。
### Structured Output モード(`structuredOutput: true`
主要指示: ルール番号1-basedと理由を返せ。
フォールバック指示: structured output が使えない場合はタグを出力せよ。
### タグモード(`structuredOutput: false`
従来の指示: 対応するタグを1行で出力せよ。
現在、Phase 3 は常に `structuredOutput: true` で実行される。
## アーキテクチャ
```
StatusJudgmentBuilder
└─ structuredOutput: true
├─ criteriaTable: ルール条件テーブル(常に含む)
├─ outputList: タグ一覧(フォールバック用に含む)
└─ テンプレート: "ルール番号と理由を返せ + タグはフォールバック"
runStatusJudgmentPhase()
└─ judgeStatus() → JudgeStatusResult { ruleIndex, method }
└─ StatusJudgmentPhaseResult { tag, ruleIndex, method }
MovementExecutor
├─ Phase 3 あり → judgeStatus の結果を直接使用method 伝搬)
└─ Phase 3 なし → detectMatchedRule() で Phase 1 コンテンツから検出
```
## JSON スキーマ
### judgment.jsonjudgeStatus 用)
```json
{
"type": "object",
"properties": {
"step": { "type": "integer", "description": "Matched rule number (1-based)" },
"reason": { "type": "string", "description": "Brief justification" }
},
"required": ["step", "reason"],
"additionalProperties": false
}
```
### evaluation.jsonevaluateCondition 用)
```json
{
"type": "object",
"properties": {
"matched_index": { "type": "integer" },
"reason": { "type": "string" }
},
"required": ["matched_index", "reason"],
"additionalProperties": false
}
```
## parseStructuredOutput() — JSON 抽出
Codex と OpenCode はテキストレスポンスから JSON を抽出する。3段階のフォールバック戦略を持つ。
```
1. Direct parse — テキスト全体が `{` で始まる JSON オブジェクト
2. Code block — ```json ... ``` または ``` ... ``` 内の JSON
3. Brace extraction — テキスト内の最初の `{` から最後の `}` までを切り出し
```
## OpenCode 固有の仕組み
OpenCode SDK は `outputFormat` を型定義でサポートしていない。代わりにプロンプト末尾に JSON 出力指示を注入する。
```
---
IMPORTANT: You MUST respond with ONLY a valid JSON object matching this schema. No other text, no markdown code blocks, no explanation.
```json
{ "type": "object", ... }
```
```
エージェントが返すテキストを `parseStructuredOutput()` でパースし、`AgentResponse.structuredOutput` に格納する。
## 注意事項
- OpenAI APICodex`required` に全プロパティを含めないとエラーになる(`additionalProperties: false` 時)
- Codex SDK の `TurnCompletedEvent` には `finalResponse` フィールドがない。structured output は `AgentMessageItem.text` の JSON テキストから `parseStructuredOutput()` でパースする
- Claude SDK は `StructuredOutput` ツール方式のため、インストラクションでタグ出力を強調しすぎるとエージェントがツールを呼ばずタグを出力してしまう
- OpenCode のプロンプト注入方式はモデルの指示従順性に依存する。JSON 以外のテキストが混在する場合は `parseStructuredOutput()` の code block / brace extraction で回収する

View File

@ -0,0 +1,18 @@
name: e2e-structured-output
description: E2E piece to verify structured output rule matching
max_movements: 5
movements:
- name: execute
edit: false
persona: ../agents/test-coder.md
permission_mode: readonly
instruction_template: |
Reply with exactly: "Task completed successfully."
Do not do anything else.
rules:
- condition: Task completed
next: COMPLETE
- condition: Task failed
next: ABORT

View File

@ -0,0 +1,96 @@
import { describe, it, expect, beforeEach, afterEach } from 'vitest';
import { resolve, dirname } from 'node:path';
import { fileURLToPath } from 'node:url';
import { createIsolatedEnv, type IsolatedEnv } from '../helpers/isolated-env';
import { createLocalRepo, type LocalRepo } from '../helpers/test-repo';
import { runTakt } from '../helpers/takt-runner';
import { readSessionRecords } from '../helpers/session-log';
const __filename = fileURLToPath(import.meta.url);
const __dirname = dirname(__filename);
/**
* E2E: Structured output for status judgment (Phase 3).
*
* Verifies that real providers (Claude, Codex, OpenCode) can execute a piece
* where the status judgment phase uses structured output (`outputSchema`)
* internally via `judgeStatus()`.
*
* The piece has 2 rules per step, so `judgeStatus` cannot auto-select
* and must actually call the provider with an outputSchema to determine
* which rule matched.
*
* If structured output works correctly, `judgeStatus` extracts the step
* number from `response.structuredOutput.step` (recorded as `structured_output`).
* If the agent happens to output `[STEP:N]` tags, the RuleEvaluator detects
* them as `phase3_tag`/`phase1_tag` (recorded as `tag_fallback` in session log).
* The session log matchMethod is transformed by `toJudgmentMatchMethod()`.
*
* Run with:
* TAKT_E2E_PROVIDER=claude vitest run --config vitest.config.e2e.structured-output.ts
* TAKT_E2E_PROVIDER=codex vitest run --config vitest.config.e2e.structured-output.ts
* TAKT_E2E_PROVIDER=opencode TAKT_E2E_MODEL=openai/gpt-4 vitest run --config vitest.config.e2e.structured-output.ts
*/
describe('E2E: Structured output rule matching', () => {
let isolatedEnv: IsolatedEnv;
let repo: LocalRepo;
beforeEach(() => {
isolatedEnv = createIsolatedEnv();
repo = createLocalRepo();
});
afterEach(() => {
try { repo.cleanup(); } catch { /* best-effort */ }
try { isolatedEnv.cleanup(); } catch { /* best-effort */ }
});
it('should complete piece via Phase 3 status judgment with 2-rule step', () => {
const piecePath = resolve(__dirname, '../fixtures/pieces/structured-output.yaml');
const result = runTakt({
args: [
'--task', 'Say hello',
'--piece', piecePath,
'--create-worktree', 'no',
],
cwd: repo.path,
env: isolatedEnv.env,
timeout: 240_000,
});
if (result.exitCode !== 0) {
console.log('=== STDOUT ===\n', result.stdout);
console.log('=== STDERR ===\n', result.stderr);
}
// Always log the matchMethod for diagnostic purposes
const allRecords = readSessionRecords(repo.path);
const sc = allRecords.find((r) => r.type === 'step_complete');
console.log(`=== matchMethod: ${sc?.matchMethod ?? '(none)'} ===`);
expect(result.exitCode).toBe(0);
expect(result.stdout).toContain('Piece completed');
// Verify session log has proper step_complete with matchMethod
const records = readSessionRecords(repo.path);
const pieceComplete = records.find((r) => r.type === 'piece_complete');
expect(pieceComplete).toBeDefined();
const stepComplete = records.find((r) => r.type === 'step_complete');
expect(stepComplete).toBeDefined();
// matchMethod should be present — the 2-rule step required actual judgment
// (auto_select is only used for single-rule steps)
const matchMethod = stepComplete?.matchMethod as string | undefined;
expect(matchMethod).toBeDefined();
// Session log records transformed matchMethod via toJudgmentMatchMethod():
// structured_output → structured_output (judgeStatus extracted from structuredOutput.step)
// phase3_tag / phase1_tag → tag_fallback (agent output [STEP:N] tag, detected by RuleEvaluator)
// ai_judge / ai_judge_fallback → ai_judge (AI evaluated conditions as fallback)
const validMethods = ['structured_output', 'tag_fallback', 'ai_judge'];
expect(validMethods).toContain(matchMethod);
}, 240_000);
});

View File

@ -40,6 +40,8 @@ function doneResponse(content: string, structuredOutput?: Record<string, unknown
};
}
const judgeOptions = { cwd: '/repo', movementName: 'review' };
describe('agent-usecases', () => {
beforeEach(() => {
vi.clearAllMocks();
@ -102,83 +104,90 @@ describe('agent-usecases', () => {
expect(detectJudgeIndex).not.toHaveBeenCalled();
});
// --- judgeStatus: 3-stage fallback ---
it('judgeStatus は単一ルール時に auto_select を返す', async () => {
const result = await judgeStatus('instruction', [{ condition: 'always', next: 'done' }], {
cwd: '/repo',
movementName: 'review',
});
const result = await judgeStatus('structured', 'tag', [{ condition: 'always', next: 'done' }], judgeOptions);
expect(result).toEqual({ ruleIndex: 0, method: 'auto_select' });
expect(runAgent).not.toHaveBeenCalled();
});
it('judgeStatus はルールが空ならエラー', async () => {
await expect(judgeStatus('instruction', [], {
cwd: '/repo',
movementName: 'review',
})).rejects.toThrow('judgeStatus requires at least one rule');
await expect(judgeStatus('structured', 'tag', [], judgeOptions))
.rejects.toThrow('judgeStatus requires at least one rule');
});
it('judgeStatus は構造化出力 step を採用する', async () => {
vi.mocked(runAgent).mockResolvedValue(doneResponse('x', { step: 2 }));
it('judgeStatus は Stage 1 で構造化出力 step を採用する', async () => {
vi.mocked(runAgent).mockResolvedValueOnce(doneResponse('x', { step: 2 }));
const result = await judgeStatus('instruction', [
const result = await judgeStatus('structured', 'tag', [
{ condition: 'a', next: 'one' },
{ condition: 'b', next: 'two' },
], {
cwd: '/repo',
movementName: 'review',
});
], judgeOptions);
expect(result).toEqual({ ruleIndex: 1, method: 'structured_output' });
expect(runAgent).toHaveBeenCalledTimes(1);
expect(runAgent).toHaveBeenCalledWith('conductor', 'structured', expect.objectContaining({
outputSchema: { type: 'judgment' },
}));
});
it('judgeStatus はタグフォールバックを使う', async () => {
vi.mocked(runAgent).mockResolvedValue(doneResponse('[REVIEW:2]'));
it('judgeStatus は Stage 2 でタグ検出を使う', async () => {
// Stage 1: structured output fails (no structuredOutput)
vi.mocked(runAgent).mockResolvedValueOnce(doneResponse('no match'));
// Stage 2: tag detection succeeds
vi.mocked(runAgent).mockResolvedValueOnce(doneResponse('[REVIEW:2]'));
const result = await judgeStatus('instruction', [
const result = await judgeStatus('structured', 'tag', [
{ condition: 'a', next: 'one' },
{ condition: 'b', next: 'two' },
], {
cwd: '/repo',
movementName: 'review',
});
], judgeOptions);
expect(result).toEqual({ ruleIndex: 1, method: 'phase3_tag' });
expect(runAgent).toHaveBeenCalledTimes(2);
expect(runAgent).toHaveBeenNthCalledWith(1, 'conductor', 'structured', expect.objectContaining({
outputSchema: { type: 'judgment' },
}));
expect(runAgent).toHaveBeenNthCalledWith(2, 'conductor', 'tag', expect.not.objectContaining({
outputSchema: expect.anything(),
}));
});
it('judgeStatus は最終手段として AI Judge を使う', async () => {
vi.mocked(runAgent)
.mockResolvedValueOnce(doneResponse('no match'))
.mockResolvedValueOnce(doneResponse('ignored', { matched_index: 2 }));
it('judgeStatus は Stage 3 で AI Judge を使う', async () => {
// Stage 1: structured output fails
vi.mocked(runAgent).mockResolvedValueOnce(doneResponse('no match'));
// Stage 2: tag detection fails
vi.mocked(runAgent).mockResolvedValueOnce(doneResponse('no tag'));
// Stage 3: evaluateCondition succeeds
vi.mocked(runAgent).mockResolvedValueOnce(doneResponse('ignored', { matched_index: 2 }));
const result = await judgeStatus('instruction', [
const result = await judgeStatus('structured', 'tag', [
{ condition: 'a', next: 'one' },
{ condition: 'b', next: 'two' },
], {
cwd: '/repo',
movementName: 'review',
});
], judgeOptions);
expect(result).toEqual({ ruleIndex: 1, method: 'ai_judge' });
expect(runAgent).toHaveBeenCalledTimes(2);
expect(runAgent).toHaveBeenCalledTimes(3);
});
it('judgeStatus は全ての判定に失敗したらエラー', async () => {
vi.mocked(runAgent)
.mockResolvedValueOnce(doneResponse('no match'))
.mockResolvedValueOnce(doneResponse('still no match'));
// Stage 1: structured output fails
vi.mocked(runAgent).mockResolvedValueOnce(doneResponse('no match'));
// Stage 2: tag detection fails
vi.mocked(runAgent).mockResolvedValueOnce(doneResponse('no tag'));
// Stage 3: evaluateCondition fails
vi.mocked(runAgent).mockResolvedValueOnce(doneResponse('still no match'));
vi.mocked(detectJudgeIndex).mockReturnValue(-1);
await expect(judgeStatus('instruction', [
await expect(judgeStatus('structured', 'tag', [
{ condition: 'a', next: 'one' },
{ condition: 'b', next: 'two' },
], {
cwd: '/repo',
movementName: 'review',
})).rejects.toThrow('Status not found for movement "review"');
], judgeOptions)).rejects.toThrow('Status not found for movement "review"');
});
// --- decomposeTask ---
it('decomposeTask は構造化出力 parts を返す', async () => {
vi.mocked(runAgent).mockResolvedValue(doneResponse('x', {
parts: [

View File

@ -1,8 +1,12 @@
/**
* Codex SDK layer structured output tests.
*
* Tests CodexClient's extraction of structuredOutput from
* `turn.completed` events' `finalResponse` field.
* Tests CodexClient's extraction of structuredOutput by parsing
* JSON text from agent_message items when outputSchema is provided.
*
* Codex SDK returns structured output as JSON text in agent_message
* items (not via turn.completed.finalResponse which doesn't exist
* on TurnCompletedEvent).
*/
import { beforeEach, describe, expect, it, vi } from 'vitest';
@ -42,34 +46,32 @@ describe('CodexClient — structuredOutput 抽出', () => {
mockEvents = [];
});
it('turn.completed の finalResponse を structuredOutput として返す', async () => {
it('outputSchema 指定時に agent_message の JSON テキストを structuredOutput として返す', async () => {
const schema = { type: 'object', properties: { step: { type: 'integer' } } };
mockEvents = [
{ type: 'thread.started', thread_id: 'thread-1' },
{
type: 'item.completed',
item: { id: 'msg-1', type: 'agent_message', text: 'response text' },
},
{
type: 'turn.completed',
turn: { finalResponse: { step: 2, reason: 'approved' } },
item: { id: 'msg-1', type: 'agent_message', text: '{"step": 2, "reason": "approved"}' },
},
{ type: 'turn.completed', usage: { input_tokens: 0, cached_input_tokens: 0, output_tokens: 0 } },
];
const client = new CodexClient();
const result = await client.call('coder', 'prompt', { cwd: '/tmp' });
const result = await client.call('coder', 'prompt', { cwd: '/tmp', outputSchema: schema });
expect(result.status).toBe('done');
expect(result.structuredOutput).toEqual({ step: 2, reason: 'approved' });
});
it('turn.completed に finalResponse がない場合は undefined', async () => {
it('outputSchema なしの場合はテキストを JSON パースしない', async () => {
mockEvents = [
{ type: 'thread.started', thread_id: 'thread-1' },
{
type: 'item.completed',
item: { id: 'msg-1', type: 'agent_message', text: 'text' },
item: { id: 'msg-1', type: 'agent_message', text: '{"step": 2}' },
},
{ type: 'turn.completed', turn: {} },
{ type: 'turn.completed', usage: { input_tokens: 0, cached_input_tokens: 0, output_tokens: 0 } },
];
const client = new CodexClient();
@ -79,86 +81,64 @@ describe('CodexClient — structuredOutput 抽出', () => {
expect(result.structuredOutput).toBeUndefined();
});
it('finalResponse が配列の場合は無視する', async () => {
mockEvents = [
{ type: 'thread.started', thread_id: 'thread-1' },
{
type: 'item.completed',
item: { id: 'msg-1', type: 'agent_message', text: 'text' },
},
{ type: 'turn.completed', turn: { finalResponse: [1, 2, 3] } },
];
const client = new CodexClient();
const result = await client.call('coder', 'prompt', { cwd: '/tmp' });
expect(result.structuredOutput).toBeUndefined();
});
it('finalResponse が null の場合は undefined', async () => {
mockEvents = [
{ type: 'thread.started', thread_id: 'thread-1' },
{ type: 'turn.completed', turn: { finalResponse: null } },
];
const client = new CodexClient();
const result = await client.call('coder', 'prompt', { cwd: '/tmp' });
expect(result.structuredOutput).toBeUndefined();
});
it('turn.completed がない場合は structuredOutput なし', async () => {
mockEvents = [
{ type: 'thread.started', thread_id: 'thread-1' },
{
type: 'item.completed',
item: { id: 'msg-1', type: 'agent_message', text: 'response' },
},
];
const client = new CodexClient();
const result = await client.call('coder', 'prompt', { cwd: '/tmp' });
expect(result.status).toBe('done');
expect(result.structuredOutput).toBeUndefined();
});
it('outputSchema が runStreamed に渡される', async () => {
it('agent_message が JSON でない場合は undefined', async () => {
const schema = { type: 'object', properties: { step: { type: 'integer' } } };
const runStreamedSpy = vi.fn().mockResolvedValue({
events: (async function* () {
yield { type: 'thread.started', thread_id: 'thread-1' };
yield {
type: 'item.completed',
item: { id: 'msg-1', type: 'agent_message', text: 'ok' },
};
yield {
type: 'turn.completed',
turn: { finalResponse: { step: 1 } },
};
})(),
});
// Mock SDK で startThread が返す thread の runStreamed を spy に差し替え
const { Codex } = await import('@openai/codex-sdk');
const codex = new Codex({} as never);
const thread = await codex.startThread();
thread.runStreamed = runStreamedSpy;
// CodexClient は内部で Codex を new するため、
// SDK クラス自体のモックで startThread の返り値を制御
// → mockEvents ベースの簡易テストでは runStreamed の引数を直接検証できない
// ここではプロバイダ層テスト (provider-structured-output.test.ts) で
// outputSchema パススルーを検証済みのため、SDK 内部の引数検証はスキップ
// 代わりに、outputSchema 付きで呼び出して structuredOutput が返ることを確認
mockEvents = [
{ type: 'thread.started', thread_id: 'thread-1' },
{
type: 'item.completed',
item: { id: 'msg-1', type: 'agent_message', text: 'ok' },
item: { id: 'msg-1', type: 'agent_message', text: 'plain text response' },
},
{ type: 'turn.completed', turn: { finalResponse: { step: 1 } } },
{ type: 'turn.completed', usage: { input_tokens: 0, cached_input_tokens: 0, output_tokens: 0 } },
];
const client = new CodexClient();
const result = await client.call('coder', 'prompt', { cwd: '/tmp', outputSchema: schema });
expect(result.status).toBe('done');
expect(result.structuredOutput).toBeUndefined();
});
it('JSON が配列の場合は無視する', async () => {
const schema = { type: 'object', properties: { step: { type: 'integer' } } };
mockEvents = [
{ type: 'thread.started', thread_id: 'thread-1' },
{
type: 'item.completed',
item: { id: 'msg-1', type: 'agent_message', text: '[1, 2, 3]' },
},
{ type: 'turn.completed', usage: { input_tokens: 0, cached_input_tokens: 0, output_tokens: 0 } },
];
const client = new CodexClient();
const result = await client.call('coder', 'prompt', { cwd: '/tmp', outputSchema: schema });
expect(result.structuredOutput).toBeUndefined();
});
it('agent_message がない場合は structuredOutput なし', async () => {
const schema = { type: 'object', properties: { step: { type: 'integer' } } };
mockEvents = [
{ type: 'thread.started', thread_id: 'thread-1' },
{ type: 'turn.completed', usage: { input_tokens: 0, cached_input_tokens: 0, output_tokens: 0 } },
];
const client = new CodexClient();
const result = await client.call('coder', 'prompt', { cwd: '/tmp', outputSchema: schema });
expect(result.status).toBe('done');
expect(result.structuredOutput).toBeUndefined();
});
it('outputSchema 付きで呼び出して structuredOutput が返る', async () => {
const schema = { type: 'object', properties: { step: { type: 'integer' } } };
mockEvents = [
{ type: 'thread.started', thread_id: 'thread-1' },
{
type: 'item.completed',
item: { id: 'msg-1', type: 'agent_message', text: '{"step": 1}' },
},
{ type: 'turn.completed', usage: { input_tokens: 0, cached_input_tokens: 0, output_tokens: 0 } },
];
const client = new CodexClient();

View File

@ -25,7 +25,7 @@ vi.mock('../core/piece/evaluation/index.js', () => ({
vi.mock('../core/piece/phase-runner.js', () => ({
needsStatusJudgmentPhase: vi.fn().mockReturnValue(false),
runReportPhase: vi.fn().mockResolvedValue(undefined),
runStatusJudgmentPhase: vi.fn().mockResolvedValue(''),
runStatusJudgmentPhase: vi.fn().mockResolvedValue({ tag: '', ruleIndex: 0, method: 'auto_select' }),
}));
vi.mock('../shared/utils/index.js', async (importOriginal) => ({

View File

@ -21,7 +21,7 @@ vi.mock('../core/piece/evaluation/index.js', () => ({
vi.mock('../core/piece/phase-runner.js', () => ({
needsStatusJudgmentPhase: vi.fn().mockReturnValue(false),
runReportPhase: vi.fn().mockResolvedValue(undefined),
runStatusJudgmentPhase: vi.fn().mockResolvedValue(''),
runStatusJudgmentPhase: vi.fn().mockResolvedValue({ tag: '', ruleIndex: 0, method: 'auto_select' }),
}));
vi.mock('../shared/utils/index.js', async () => {

View File

@ -23,7 +23,7 @@ vi.mock('../core/piece/evaluation/index.js', () => ({
vi.mock('../core/piece/phase-runner.js', () => ({
needsStatusJudgmentPhase: vi.fn().mockReturnValue(false),
runReportPhase: vi.fn().mockResolvedValue(undefined),
runStatusJudgmentPhase: vi.fn().mockResolvedValue(''),
runStatusJudgmentPhase: vi.fn().mockResolvedValue({ tag: '', ruleIndex: 0, method: 'auto_select' }),
}));
vi.mock('../shared/utils/index.js', async (importOriginal) => ({

View File

@ -24,7 +24,7 @@ vi.mock('../core/piece/evaluation/index.js', () => ({
vi.mock('../core/piece/phase-runner.js', () => ({
needsStatusJudgmentPhase: vi.fn().mockReturnValue(false),
runReportPhase: vi.fn().mockResolvedValue(undefined),
runStatusJudgmentPhase: vi.fn().mockResolvedValue(''),
runStatusJudgmentPhase: vi.fn().mockResolvedValue({ tag: '', ruleIndex: 0, method: 'auto_select' }),
}));
vi.mock('../shared/utils/index.js', async (importOriginal) => ({

View File

@ -28,7 +28,7 @@ vi.mock('../core/piece/evaluation/index.js', () => ({
vi.mock('../core/piece/phase-runner.js', () => ({
needsStatusJudgmentPhase: vi.fn().mockReturnValue(false),
runReportPhase: vi.fn().mockResolvedValue(undefined),
runStatusJudgmentPhase: vi.fn().mockResolvedValue(''),
runStatusJudgmentPhase: vi.fn().mockResolvedValue({ tag: '', ruleIndex: 0, method: 'auto_select' }),
}));
vi.mock('../shared/utils/index.js', async (importOriginal) => ({

View File

@ -27,7 +27,7 @@ vi.mock('../core/piece/evaluation/index.js', () => ({
vi.mock('../core/piece/phase-runner.js', () => ({
needsStatusJudgmentPhase: vi.fn().mockReturnValue(false),
runReportPhase: vi.fn().mockResolvedValue(undefined),
runStatusJudgmentPhase: vi.fn().mockResolvedValue(''),
runStatusJudgmentPhase: vi.fn().mockResolvedValue({ tag: '', ruleIndex: 0, method: 'auto_select' }),
}));
vi.mock('../shared/utils/index.js', async (importOriginal) => ({

View File

@ -23,7 +23,7 @@ vi.mock('../core/piece/evaluation/index.js', () => ({
vi.mock('../core/piece/phase-runner.js', () => ({
needsStatusJudgmentPhase: vi.fn().mockReturnValue(false),
runReportPhase: vi.fn().mockResolvedValue(undefined),
runStatusJudgmentPhase: vi.fn().mockResolvedValue(''),
runStatusJudgmentPhase: vi.fn().mockResolvedValue({ tag: '', ruleIndex: 0, method: 'auto_select' }),
}));
vi.mock('../shared/utils/index.js', async (importOriginal) => ({

View File

@ -24,7 +24,7 @@ vi.mock('../core/piece/evaluation/index.js', () => ({
vi.mock('../core/piece/phase-runner.js', () => ({
needsStatusJudgmentPhase: vi.fn().mockReturnValue(false),
runReportPhase: vi.fn().mockResolvedValue(undefined),
runStatusJudgmentPhase: vi.fn().mockResolvedValue(''),
runStatusJudgmentPhase: vi.fn().mockResolvedValue({ tag: '', ruleIndex: 0, method: 'auto_select' }),
}));
vi.mock('../shared/utils/index.js', async (importOriginal) => ({

View File

@ -17,7 +17,7 @@ vi.mock('../core/piece/evaluation/index.js', () => ({
vi.mock('../core/piece/phase-runner.js', () => ({
needsStatusJudgmentPhase: vi.fn().mockReturnValue(false),
runReportPhase: vi.fn().mockResolvedValue(undefined),
runStatusJudgmentPhase: vi.fn().mockResolvedValue(''),
runStatusJudgmentPhase: vi.fn().mockResolvedValue({ tag: '', ruleIndex: 0, method: 'auto_select' }),
}));
vi.mock('../shared/utils/index.js', async (importOriginal) => ({

View File

@ -173,7 +173,7 @@ export function createTestTmpDir(): string {
export function applyDefaultMocks(): void {
vi.mocked(needsStatusJudgmentPhase).mockReturnValue(false);
vi.mocked(runReportPhase).mockResolvedValue(undefined);
vi.mocked(runStatusJudgmentPhase).mockResolvedValue('');
vi.mocked(runStatusJudgmentPhase).mockResolvedValue({ tag: '', ruleIndex: 0, method: 'auto_select' });
vi.mocked(generateReportDir).mockReturnValue('test-report-dir');
}

View File

@ -24,7 +24,7 @@ vi.mock('../core/piece/evaluation/index.js', () => ({
vi.mock('../core/piece/phase-runner.js', () => ({
needsStatusJudgmentPhase: vi.fn().mockReturnValue(false),
runReportPhase: vi.fn().mockResolvedValue(undefined),
runStatusJudgmentPhase: vi.fn().mockResolvedValue(''),
runStatusJudgmentPhase: vi.fn().mockResolvedValue({ tag: '', ruleIndex: 0, method: 'auto_select' }),
}));
vi.mock('../shared/utils/index.js', async (importOriginal) => ({

View File

@ -31,7 +31,7 @@ vi.mock('../agents/ai-judge.js', async (importOriginal) => {
vi.mock('../core/piece/phase-runner.js', () => ({
needsStatusJudgmentPhase: vi.fn().mockReturnValue(false),
runReportPhase: vi.fn().mockResolvedValue(undefined),
runStatusJudgmentPhase: vi.fn().mockResolvedValue(''),
runStatusJudgmentPhase: vi.fn().mockResolvedValue({ tag: '', ruleIndex: 0, method: 'auto_select' }),
}));
vi.mock('../shared/utils/index.js', async (importOriginal) => ({

View File

@ -35,7 +35,7 @@ vi.mock('../agents/ai-judge.js', async (importOriginal) => {
vi.mock('../core/piece/phase-runner.js', () => ({
needsStatusJudgmentPhase: vi.fn().mockReturnValue(false),
runReportPhase: vi.fn().mockResolvedValue(undefined),
runStatusJudgmentPhase: vi.fn().mockResolvedValue(''),
runStatusJudgmentPhase: vi.fn().mockResolvedValue({ tag: '', ruleIndex: 0, method: 'auto_select' }),
}));
vi.mock('../shared/utils/index.js', async (importOriginal) => ({

View File

@ -37,7 +37,7 @@ vi.mock('../agents/ai-judge.js', async (importOriginal) => {
vi.mock('../core/piece/phase-runner.js', () => ({
needsStatusJudgmentPhase: vi.fn().mockReturnValue(false),
runReportPhase: vi.fn().mockResolvedValue(undefined),
runStatusJudgmentPhase: vi.fn().mockResolvedValue(''),
runStatusJudgmentPhase: vi.fn().mockResolvedValue({ tag: '', ruleIndex: 0, method: 'auto_select' }),
}));
vi.mock('../shared/utils/index.js', async (importOriginal) => ({

View File

@ -144,7 +144,7 @@ vi.mock('../shared/prompt/index.js', () => ({
vi.mock('../core/piece/phase-runner.js', () => ({
needsStatusJudgmentPhase: vi.fn().mockReturnValue(false),
runReportPhase: vi.fn().mockResolvedValue(undefined),
runStatusJudgmentPhase: vi.fn().mockResolvedValue(''),
runStatusJudgmentPhase: vi.fn().mockResolvedValue({ tag: '', ruleIndex: 0, method: 'auto_select' }),
}));
// --- Imports (after mocks) ---

View File

@ -125,7 +125,7 @@ vi.mock('../shared/prompt/index.js', () => ({
vi.mock('../core/piece/phase-runner.js', () => ({
needsStatusJudgmentPhase: vi.fn().mockReturnValue(false),
runReportPhase: vi.fn().mockResolvedValue(undefined),
runStatusJudgmentPhase: vi.fn().mockResolvedValue(''),
runStatusJudgmentPhase: vi.fn().mockResolvedValue({ tag: '', ruleIndex: 0, method: 'auto_select' }),
}));
// --- Imports (after mocks) ---

View File

@ -114,7 +114,7 @@ describe('Three-Phase Execution IT: phase1 only (no report, no tag rules)', () =
// No tag rules needed → Phase 3 not needed
mockNeedsStatusJudgmentPhase.mockReturnValue(false);
mockRunReportPhase.mockResolvedValue(undefined);
mockRunStatusJudgmentPhase.mockResolvedValue('');
mockRunStatusJudgmentPhase.mockResolvedValue({ tag: '', ruleIndex: 0, method: 'auto_select' });
});
afterEach(() => {
@ -166,7 +166,7 @@ describe('Three-Phase Execution IT: phase1 + phase2 (report defined)', () => {
mockNeedsStatusJudgmentPhase.mockReturnValue(false);
mockRunReportPhase.mockResolvedValue(undefined);
mockRunStatusJudgmentPhase.mockResolvedValue('');
mockRunStatusJudgmentPhase.mockResolvedValue({ tag: '', ruleIndex: 0, method: 'auto_select' });
});
afterEach(() => {
@ -246,7 +246,7 @@ describe('Three-Phase Execution IT: phase1 + phase3 (tag rules defined)', () =>
mockNeedsStatusJudgmentPhase.mockReturnValue(true);
mockRunReportPhase.mockResolvedValue(undefined);
// Phase 3 returns content with a tag
mockRunStatusJudgmentPhase.mockResolvedValue('[STEP:1]');
mockRunStatusJudgmentPhase.mockResolvedValue({ tag: '[STEP:1]', ruleIndex: 0, method: 'structured_output' });
});
afterEach(() => {
@ -298,7 +298,7 @@ describe('Three-Phase Execution IT: all three phases', () => {
mockNeedsStatusJudgmentPhase.mockReturnValue(true);
mockRunReportPhase.mockResolvedValue(undefined);
mockRunStatusJudgmentPhase.mockResolvedValue('[STEP:1]');
mockRunStatusJudgmentPhase.mockResolvedValue({ tag: '[STEP:1]', ruleIndex: 0, method: 'structured_output' });
});
afterEach(() => {
@ -369,7 +369,7 @@ describe('Three-Phase Execution IT: phase3 tag → rule match', () => {
]);
// Phase 3 returns rule 2 (ABORT)
mockRunStatusJudgmentPhase.mockResolvedValue('[STEP1:2]');
mockRunStatusJudgmentPhase.mockResolvedValue({ tag: '[STEP1:2]', ruleIndex: 1, method: 'structured_output' });
const config: PieceConfig = {
name: 'it-phase3-tag',

View File

@ -0,0 +1,86 @@
import { describe, it, expect } from 'vitest';
import { parseStructuredOutput } from '../shared/utils/structuredOutput.js';
describe('parseStructuredOutput', () => {
it('should return undefined when hasOutputSchema is false', () => {
expect(parseStructuredOutput('{"step":1}', false)).toBeUndefined();
});
it('should return undefined for empty text', () => {
expect(parseStructuredOutput('', true)).toBeUndefined();
});
// Strategy 1: Direct JSON parse
describe('direct JSON parse', () => {
it('should parse pure JSON object', () => {
expect(parseStructuredOutput('{"step":1,"reason":"done"}', true))
.toEqual({ step: 1, reason: 'done' });
});
it('should parse JSON with whitespace', () => {
expect(parseStructuredOutput(' { "step": 2, "reason": "ok" } ', true))
.toEqual({ step: 2, reason: 'ok' });
});
it('should ignore arrays', () => {
expect(parseStructuredOutput('[1,2,3]', true)).toBeUndefined();
});
it('should ignore primitive JSON', () => {
expect(parseStructuredOutput('"hello"', true)).toBeUndefined();
});
});
// Strategy 2: Code block extraction
describe('code block extraction', () => {
it('should extract JSON from ```json code block', () => {
const text = 'Here is the result:\n```json\n{"step":1,"reason":"matched"}\n```';
expect(parseStructuredOutput(text, true))
.toEqual({ step: 1, reason: 'matched' });
});
it('should extract JSON from ``` code block (no language)', () => {
const text = 'Result:\n```\n{"step":2,"reason":"fallback"}\n```';
expect(parseStructuredOutput(text, true))
.toEqual({ step: 2, reason: 'fallback' });
});
});
// Strategy 3: Brace extraction
describe('brace extraction', () => {
it('should extract JSON with preamble text', () => {
const text = 'The matched rule is: {"step":1,"reason":"condition met"}';
expect(parseStructuredOutput(text, true))
.toEqual({ step: 1, reason: 'condition met' });
});
it('should extract JSON with postamble text', () => {
const text = '{"step":3,"reason":"done"}\nEnd of response.';
expect(parseStructuredOutput(text, true))
.toEqual({ step: 3, reason: 'done' });
});
it('should extract JSON with both preamble and postamble', () => {
const text = 'Based on my analysis:\n{"matched_index":2,"reason":"test"}\nThat is my judgment.';
expect(parseStructuredOutput(text, true))
.toEqual({ matched_index: 2, reason: 'test' });
});
});
// Edge cases
describe('edge cases', () => {
it('should return undefined for text without JSON', () => {
expect(parseStructuredOutput('No JSON here at all.', true)).toBeUndefined();
});
it('should return undefined for invalid JSON', () => {
expect(parseStructuredOutput('{invalid json}', true)).toBeUndefined();
});
it('should handle nested objects', () => {
const text = '{"step":1,"reason":"ok","meta":{"detail":"extra"}}';
expect(parseStructuredOutput(text, true))
.toEqual({ step: 1, reason: 'ok', meta: { detail: 'extra' } });
});
});
});

View File

@ -23,7 +23,7 @@ vi.mock('../core/piece/evaluation/index.js', () => ({
vi.mock('../core/piece/phase-runner.js', () => ({
needsStatusJudgmentPhase: vi.fn().mockReturnValue(false),
runReportPhase: vi.fn().mockResolvedValue(undefined),
runStatusJudgmentPhase: vi.fn().mockResolvedValue(''),
runStatusJudgmentPhase: vi.fn().mockResolvedValue({ tag: '', ruleIndex: 0, method: 'auto_select' }),
}));
vi.mock('../shared/utils/index.js', async (importOriginal) => ({

View File

@ -85,7 +85,8 @@ export async function evaluateCondition(
}
export async function judgeStatus(
instruction: string,
structuredInstruction: string,
tagInstruction: string,
rules: PieceRule[],
options: JudgeStatusOptions,
): Promise<JudgeStatusResult> {
@ -94,48 +95,47 @@ export async function judgeStatus(
}
if (rules.length === 1) {
return {
ruleIndex: 0,
method: 'auto_select',
};
return { ruleIndex: 0, method: 'auto_select' };
}
const response = await runAgent('conductor', instruction, {
const agentOptions = {
cwd: options.cwd,
maxTurns: 3,
permissionMode: 'readonly',
permissionMode: 'readonly' as const,
language: options.language,
};
// Stage 1: Structured output
const structuredResponse = await runAgent('conductor', structuredInstruction, {
...agentOptions,
outputSchema: loadJudgmentSchema(),
});
if (response.status === 'done') {
const stepNumber = response.structuredOutput?.step;
if (structuredResponse.status === 'done') {
const stepNumber = structuredResponse.structuredOutput?.step;
if (typeof stepNumber === 'number' && Number.isInteger(stepNumber)) {
const ruleIndex = stepNumber - 1;
if (ruleIndex >= 0 && ruleIndex < rules.length) {
return {
ruleIndex,
method: 'structured_output',
};
return { ruleIndex, method: 'structured_output' };
}
}
const tagRuleIndex = detectRuleIndex(response.content, options.movementName);
if (tagRuleIndex >= 0 && tagRuleIndex < rules.length) {
return {
ruleIndex: tagRuleIndex,
method: 'phase3_tag',
};
}
}
// Stage 2: Tag detection (dedicated call, no outputSchema)
const tagResponse = await runAgent('conductor', tagInstruction, agentOptions);
if (tagResponse.status === 'done') {
const tagRuleIndex = detectRuleIndex(tagResponse.content, options.movementName);
if (tagRuleIndex >= 0 && tagRuleIndex < rules.length) {
return { ruleIndex: tagRuleIndex, method: 'phase3_tag' };
}
}
// Stage 3: AI judge
const conditions = rules.map((rule, index) => ({ index, text: rule.condition }));
const fallbackIndex = await evaluateCondition(instruction, conditions, { cwd: options.cwd });
const fallbackIndex = await evaluateCondition(structuredInstruction, conditions, { cwd: options.cwd });
if (fallbackIndex >= 0 && fallbackIndex < rules.length) {
return {
ruleIndex: fallbackIndex,
method: 'ai_judge',
};
return { ruleIndex: fallbackIndex, method: 'ai_judge' };
}
throw new Error(`Status not found for movement "${options.movementName}"`);

View File

@ -220,22 +220,28 @@ export class MovementExecutor {
}
}
// Phase 3: status judgment (resume session, no tools, output status tag)
let tagContent = '';
if (needsStatusJudgmentPhase(step)) {
tagContent = await runStatusJudgmentPhase(step, phaseCtx);
}
// Phase 3: status judgment (new session, no tools, determines matched rule)
const phase3Result = needsStatusJudgmentPhase(step)
? await runStatusJudgmentPhase(step, phaseCtx)
: undefined;
const match = await detectMatchedRule(step, response.content, tagContent, {
state,
cwd: this.deps.getCwd(),
interactive: this.deps.getInteractive(),
detectRuleIndex: this.deps.detectRuleIndex,
callAiJudge: this.deps.callAiJudge,
});
if (match) {
log.debug('Rule matched', { movement: step.name, ruleIndex: match.index, method: match.method });
response = { ...response, matchedRuleIndex: match.index, matchedRuleMethod: match.method };
if (phase3Result) {
// Phase 3 already determined the matched rule — use its result directly
log.debug('Rule matched (Phase 3)', { movement: step.name, ruleIndex: phase3Result.ruleIndex, method: phase3Result.method });
response = { ...response, matchedRuleIndex: phase3Result.ruleIndex, matchedRuleMethod: phase3Result.method };
} else {
// No Phase 3 — use rule evaluator with Phase 1 content
const match = await detectMatchedRule(step, response.content, '', {
state,
cwd: this.deps.getCwd(),
interactive: this.deps.getInteractive(),
detectRuleIndex: this.deps.detectRuleIndex,
callAiJudge: this.deps.callAiJudge,
});
if (match) {
log.debug('Rule matched', { movement: step.name, ruleIndex: match.index, method: match.method });
response = { ...response, matchedRuleIndex: match.index, matchedRuleMethod: match.method };
}
}
state.movementOutputs.set(step.name, response);

View File

@ -114,15 +114,19 @@ export class ParallelRunner {
}
// Phase 3: status judgment for sub-movement
let subTagContent = '';
if (needsStatusJudgmentPhase(subMovement)) {
subTagContent = await runStatusJudgmentPhase(subMovement, phaseCtx);
}
const subPhase3 = needsStatusJudgmentPhase(subMovement)
? await runStatusJudgmentPhase(subMovement, phaseCtx)
: undefined;
const match = await detectMatchedRule(subMovement, subResponse.content, subTagContent, ruleCtx);
const finalResponse = match
? { ...subResponse, matchedRuleIndex: match.index, matchedRuleMethod: match.method }
: subResponse;
let finalResponse: AgentResponse;
if (subPhase3) {
finalResponse = { ...subResponse, matchedRuleIndex: subPhase3.ruleIndex, matchedRuleMethod: subPhase3.method };
} else {
const match = await detectMatchedRule(subMovement, subResponse.content, '', ruleCtx);
finalResponse = match
? { ...subResponse, matchedRuleIndex: match.index, matchedRuleMethod: match.method }
: subResponse;
}
state.movementOutputs.set(subMovement.name, finalResponse);
this.deps.movementExecutor.emitMovementReports(subMovement);

View File

@ -27,8 +27,8 @@ export interface StatusJudgmentContext {
lastResponse?: string;
/** Input source type for fallback strategies */
inputSource?: 'report' | 'response';
/** Structured output mode omits tag-format instructions */
useStructuredOutput?: boolean;
/** When true, omit tag output instructions (structured output schema handles format) */
structuredOutput?: boolean;
}
/**
@ -66,14 +66,17 @@ export class StatusJudgmentBuilder {
contentToJudge = this.buildFromResponse();
}
const isStructured = this.context.structuredOutput ?? false;
return loadTemplate('perform_phase3_message', language, {
reportContent: contentToJudge,
criteriaTable: components.criteriaTable,
outputList: this.context.useStructuredOutput
? ''
: components.outputList,
hasAppendix: components.hasAppendix,
appendixContent: components.appendixContent,
structuredOutput: isStructured,
...(isStructured ? {} : {
outputList: components.outputList,
hasAppendix: components.hasAppendix,
appendixContent: components.appendixContent,
}),
});
}

View File

@ -15,7 +15,7 @@ import { hasTagBasedRules, getReportFiles } from './evaluation/rule-utils.js';
import { executeAgent } from './agent-usecases.js';
import { createLogger } from '../../shared/utils/index.js';
import { buildSessionKey } from './session-key.js';
export { runStatusJudgmentPhase } from './status-judgment-phase.js';
export { runStatusJudgmentPhase, type StatusJudgmentPhaseResult } from './status-judgment-phase.js';
const log = createLogger('phase-runner');

View File

@ -1,75 +1,98 @@
import { existsSync, readFileSync } from 'node:fs';
import { resolve } from 'node:path';
import type { PieceMovement } from '../models/types.js';
import type { PieceMovement, RuleMatchMethod } from '../models/types.js';
import { judgeStatus } from './agent-usecases.js';
import { StatusJudgmentBuilder } from './instruction/StatusJudgmentBuilder.js';
import { StatusJudgmentBuilder, type StatusJudgmentContext } from './instruction/StatusJudgmentBuilder.js';
import { getReportFiles } from './evaluation/rule-utils.js';
import { createLogger } from '../../shared/utils/index.js';
import type { PhaseRunnerContext } from './phase-runner.js';
const log = createLogger('phase-runner');
/** Result of Phase 3 status judgment, including the detection method. */
export interface StatusJudgmentPhaseResult {
tag: string;
ruleIndex: number;
method: RuleMatchMethod;
}
/**
* Phase 3: Status judgment.
* Uses the 'conductor' agent in a new session to output a status tag.
* Implements multi-stage fallback logic to ensure judgment succeeds.
* Returns the Phase 3 response content (containing the status tag).
* Build the base context (shared by structured output and tag instructions).
*/
export async function runStatusJudgmentPhase(
function buildBaseContext(
step: PieceMovement,
ctx: PhaseRunnerContext,
): Promise<string> {
log.debug('Running status judgment phase', { movement: step.name });
if (!step.rules || step.rules.length === 0) {
throw new Error(`Status judgment requires rules for movement "${step.name}"`);
}
): Omit<StatusJudgmentContext, 'structuredOutput'> | undefined {
const reportFiles = getReportFiles(step.outputContracts);
let instruction: string | undefined;
if (reportFiles.length > 0) {
const reports: string[] = [];
for (const fileName of reportFiles) {
const filePath = resolve(ctx.reportDir, fileName);
if (!existsSync(filePath)) {
continue;
}
if (!existsSync(filePath)) continue;
const content = readFileSync(filePath, 'utf-8');
reports.push(`# ${fileName}\n\n${content}`);
}
if (reports.length > 0) {
instruction = new StatusJudgmentBuilder(step, {
return {
language: ctx.language,
reportContent: reports.join('\n\n---\n\n'),
inputSource: 'report',
useStructuredOutput: true,
}).build();
};
}
}
if (instruction == null) {
if (!ctx.lastResponse) {
throw new Error(`Status judgment requires report or lastResponse for movement "${step.name}"`);
}
if (!ctx.lastResponse) return undefined;
instruction = new StatusJudgmentBuilder(step, {
language: ctx.language,
lastResponse: ctx.lastResponse,
inputSource: 'response',
useStructuredOutput: true,
}).build();
return {
language: ctx.language,
lastResponse: ctx.lastResponse,
inputSource: 'response',
};
}
/**
* Phase 3: Status judgment.
*
* Builds two instructions from the same context:
* - Structured output instruction (JSON schema)
* - Tag instruction (free-form tag detection)
*
* `judgeStatus()` tries them in order: structured tag ai_judge.
*/
export async function runStatusJudgmentPhase(
step: PieceMovement,
ctx: PhaseRunnerContext,
): Promise<StatusJudgmentPhaseResult> {
log.debug('Running status judgment phase', { movement: step.name });
if (!step.rules || step.rules.length === 0) {
throw new Error(`Status judgment requires rules for movement "${step.name}"`);
}
ctx.onPhaseStart?.(step, 3, 'judge', instruction);
const baseContext = buildBaseContext(step, ctx);
if (!baseContext) {
throw new Error(`Status judgment requires report or lastResponse for movement "${step.name}"`);
}
const structuredInstruction = new StatusJudgmentBuilder(step, {
...baseContext,
structuredOutput: true,
}).build();
const tagInstruction = new StatusJudgmentBuilder(step, {
...baseContext,
}).build();
ctx.onPhaseStart?.(step, 3, 'judge', structuredInstruction);
try {
const result = await judgeStatus(instruction, step.rules, {
const result = await judgeStatus(structuredInstruction, tagInstruction, step.rules, {
cwd: ctx.cwd,
movementName: step.name,
language: ctx.language,
});
const tag = `[${step.name.toUpperCase()}:${result.ruleIndex + 1}]`;
ctx.onPhaseComplete?.(step, 3, 'judge', tag, 'done');
return tag;
return { tag, ruleIndex: result.ruleIndex, method: result.method };
} catch (error) {
const errorMsg = error instanceof Error ? error.message : String(error);
ctx.onPhaseComplete?.(step, 3, 'judge', '', 'error', errorMsg);

View File

@ -4,9 +4,9 @@
* Uses @openai/codex-sdk for native TypeScript integration.
*/
import { Codex } from '@openai/codex-sdk';
import { Codex, type TurnOptions } from '@openai/codex-sdk';
import type { AgentResponse } from '../../core/models/index.js';
import { createLogger, getErrorMessage, createStreamDiagnostics, type StreamDiagnostics } from '../../shared/utils/index.js';
import { createLogger, getErrorMessage, createStreamDiagnostics, parseStructuredOutput, type StreamDiagnostics } from '../../shared/utils/index.js';
import { mapToCodexSandboxMode, type CodexCallOptions } from './types.js';
import {
type CodexEvent,
@ -150,20 +150,15 @@ export class CodexClient {
const diag = createStreamDiagnostics('codex-sdk', { agentType, model: options.model, attempt });
diagRef = diag;
const runOptions: Record<string, unknown> = {
const turnOptions: TurnOptions = {
signal: streamAbortController.signal,
...(options.outputSchema ? { outputSchema: options.outputSchema } : {}),
};
if (options.outputSchema) {
runOptions.outputSchema = options.outputSchema;
}
// Codex SDK types do not yet expose outputSchema even though runtime accepts it.
const runStreamedOptions = runOptions as unknown as Parameters<typeof thread.runStreamed>[1];
const { events } = await thread.runStreamed(fullPrompt, runStreamedOptions);
const { events } = await thread.runStreamed(fullPrompt, turnOptions);
resetIdleTimeout();
diag.onConnected();
let content = '';
let structuredOutput: Record<string, unknown> | undefined;
const contentOffsets = new Map<string, number>();
let success = true;
let failureMessage = '';
@ -196,20 +191,6 @@ export class CodexClient {
break;
}
if (event.type === 'turn.completed') {
const rawFinalResponse = (event as unknown as {
turn?: { finalResponse?: unknown };
}).turn?.finalResponse;
if (
rawFinalResponse
&& typeof rawFinalResponse === 'object'
&& !Array.isArray(rawFinalResponse)
) {
structuredOutput = rawFinalResponse as Record<string, unknown>;
}
continue;
}
if (event.type === 'item.started') {
const item = event.item as CodexItem | undefined;
if (item) {
@ -291,6 +272,7 @@ export class CodexClient {
}
const trimmed = content.trim();
const structuredOutput = parseStructuredOutput(trimmed, !!options.outputSchema);
emitResult(options.onStream, true, trimmed, currentThreadId);
return {

View File

@ -8,7 +8,7 @@
import { createOpencode } from '@opencode-ai/sdk/v2';
import { createServer } from 'node:net';
import type { AgentResponse } from '../../core/models/index.js';
import { createLogger, getErrorMessage, createStreamDiagnostics, type StreamDiagnostics } from '../../shared/utils/index.js';
import { createLogger, getErrorMessage, createStreamDiagnostics, parseStructuredOutput, type StreamDiagnostics } from '../../shared/utils/index.js';
import { parseProviderModel } from '../../shared/utils/providerModel.js';
import {
buildOpenCodePermissionConfig,
@ -236,16 +236,34 @@ export class OpenCodeClient {
});
}
/** Build a prompt suffix that instructs the agent to return JSON matching the schema */
private buildStructuredOutputSuffix(schema: Record<string, unknown>): string {
return [
'',
'---',
'IMPORTANT: You MUST respond with ONLY a valid JSON object matching this schema. No other text, no markdown code blocks, no explanation.',
'```',
JSON.stringify(schema, null, 2),
'```',
].join('\n');
}
/** Call OpenCode with an agent prompt */
async call(
agentType: string,
prompt: string,
options: OpenCodeCallOptions,
): Promise<AgentResponse> {
const fullPrompt = options.systemPrompt
const basePrompt = options.systemPrompt
? `${options.systemPrompt}\n\n${prompt}`
: prompt;
// OpenCode SDK does not natively support structured output via outputFormat.
// Inject JSON output instructions into the prompt to make the agent return JSON.
const fullPrompt = options.outputSchema
? `${basePrompt}${this.buildStructuredOutputSuffix(options.outputSchema)}`
: basePrompt;
for (let attempt = 1; attempt <= OPENCODE_RETRY_MAX_ATTEMPTS; attempt++) {
let idleTimeoutId: ReturnType<typeof setTimeout> | undefined;
const streamAbortController = new AbortController();
@ -580,17 +598,7 @@ export class OpenCodeClient {
}
const trimmed = content.trim();
let structuredOutput: Record<string, unknown> | undefined;
if (options.outputSchema && trimmed.startsWith('{')) {
try {
const parsed = JSON.parse(trimmed) as unknown;
if (parsed && typeof parsed === 'object' && !Array.isArray(parsed)) {
structuredOutput = parsed as Record<string, unknown>;
}
} catch {
// Non-JSON response falls back to text path.
}
}
const structuredOutput = parseStructuredOutput(trimmed, !!options.outputSchema);
emitResult(options.onStream, true, trimmed, sessionId);
return {

View File

@ -1,10 +1,14 @@
<!--
template: perform_phase3_message
phase: 3 (status judgment)
vars: reportContent, criteriaTable, outputList, hasAppendix, appendixContent
vars: reportContent, criteriaTable, outputList, hasAppendix, appendixContent, structuredOutput
builder: StatusJudgmentBuilder
-->
{{#if structuredOutput}}
**Review is already complete. Evaluate the report below and determine which numbered rule (1-based) best matches the result.**
{{else}}
**Review is already complete. Output exactly one tag corresponding to the judgment result shown in the report below.**
{{/if}}
{{reportContent}}
@ -12,12 +16,21 @@
{{criteriaTable}}
{{#if structuredOutput}}
## Task
Evaluate the report against the criteria above. Return the matched rule number (1-based integer) and a brief reason for your decision.
{{else}}
## Output Format
**Output the tag corresponding to the judgment shown in the report in one line:**
{{outputList}}
{{/if}}
{{#if hasAppendix}}
### Appendix Template
{{appendixContent}}{{/if}}
{{appendixContent}}
{{/if}}

View File

@ -1,10 +1,14 @@
<!--
template: perform_phase3_message
phase: 3 (status judgment)
vars: reportContent, criteriaTable, outputList, hasAppendix, appendixContent
vars: reportContent, criteriaTable, outputList, hasAppendix, appendixContent, structuredOutput
builder: StatusJudgmentBuilder
-->
{{#if structuredOutput}}
**既にレビューは完了しています。以下のレポートを評価し、どの番号のルール1始まりが結果に最も合致するか判定してください。**
{{else}}
**既にレビューは完了しています。以下のレポートで示された判定結果に対応するタグを1つだけ出力してください。**
{{/if}}
{{reportContent}}
@ -12,12 +16,21 @@
{{criteriaTable}}
{{#if structuredOutput}}
## タスク
上記の判定基準に照らしてレポートを評価してください。合致するルール番号1始まりの整数と簡潔な理由を返してください。
{{else}}
## 出力フォーマット
**レポートで示した判定に対応するタグを1行で出力してください**
{{outputList}}
{{/if}}
{{#if hasAppendix}}
### 追加出力テンプレート
{{appendixContent}}{{/if}}
{{appendixContent}}
{{/if}}

View File

@ -11,6 +11,7 @@ export * from './slackWebhook.js';
export * from './sleep.js';
export * from './slug.js';
export * from './streamDiagnostics.js';
export * from './structuredOutput.js';
export * from './taskPaths.js';
export * from './text.js';
export * from './types.js';

View File

@ -0,0 +1,56 @@
/**
* Parse structured output from provider text response.
*
* Codex and OpenCode return structured output as JSON text in agent messages.
* This function extracts a JSON object from the text when outputSchema was requested.
*
* Extraction strategies (in order):
* 1. Direct JSON parse text is pure JSON starting with `{`
* 2. Code block extraction JSON inside ```json ... ``` or ``` ... ```
* 3. Brace extraction find outermost `{` ... `}` in the text
*/
function tryParseJsonObject(text: string): Record<string, unknown> | undefined {
try {
const parsed = JSON.parse(text) as unknown;
if (parsed && typeof parsed === 'object' && !Array.isArray(parsed)) {
return parsed as Record<string, unknown>;
}
} catch {
// Not valid JSON
}
return undefined;
}
export function parseStructuredOutput(
text: string,
hasOutputSchema: boolean,
): Record<string, unknown> | undefined {
if (!hasOutputSchema || !text) return undefined;
const trimmed = text.trim();
// Strategy 1: Direct JSON parse (text is pure JSON)
if (trimmed.startsWith('{')) {
const result = tryParseJsonObject(trimmed);
if (result) return result;
}
// Strategy 2: Extract from markdown code block (```json\n{...}\n```)
const codeBlockMatch = trimmed.match(/```(?:json)?\s*\n(\{[\s\S]*?\})\s*\n```/);
if (codeBlockMatch?.[1]) {
const result = tryParseJsonObject(codeBlockMatch[1].trim());
if (result) return result;
}
// Strategy 3: Find first `{` and last `}` (handles preamble/postamble text)
const firstBrace = trimmed.indexOf('{');
const lastBrace = trimmed.lastIndexOf('}');
if (firstBrace >= 0 && lastBrace > firstBrace) {
const candidate = trimmed.slice(firstBrace, lastBrace + 1);
const result = tryParseJsonObject(candidate);
if (result) return result;
}
return undefined;
}

View File

@ -7,6 +7,7 @@ export default defineConfig({
'e2e/specs/worktree.e2e.ts',
'e2e/specs/pipeline.e2e.ts',
'e2e/specs/github-issue.e2e.ts',
'e2e/specs/structured-output.e2e.ts',
],
environment: 'node',
globals: false,

View File

@ -0,0 +1,20 @@
import { defineConfig } from 'vitest/config';
export default defineConfig({
test: {
include: [
'e2e/specs/structured-output.e2e.ts',
],
environment: 'node',
globals: false,
testTimeout: 240000,
hookTimeout: 60000,
teardownTimeout: 30000,
pool: 'threads',
poolOptions: {
threads: {
singleThread: true,
},
},
},
});