> For the complete documentation index, see [llms.txt](https://ai-os-and-trend-finder.gitbook.io/ai-os-and-trend-finder-docs/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://ai-os-and-trend-finder.gitbook.io/ai-os-and-trend-finder-docs/.spec_system/archive/sessions/phase29-session08-required-derived-field-closeout-gate/spec.md). # Session Specification **Session ID**: `phase29-session08-required-derived-field-closeout-gate` **Phase**: 29 - Trend Finder TrendingAI Comparison Adoption **Status**: Not Started **Created**: 2026-06-20 *** ## 1. Session Overview This session adds a required-derived-field closeout gate for Trend Finder payloads. Sessions 02 through 06 shipped the derived fields that live and static surfaces now expect: `attentionPattern`, `receptionSignal`, `corroboration`, `evidenceRationales`, and report-level `runNarratives`. Those fields have schema defaults and deterministic fallback paths, so new generated payloads should not silently omit them. The implementation should fail loudly at the closeout boundary when a ranked topic is missing one of the required derived fields, or when the report omits a required derived report branch. The gate belongs after topic projection and browser-safe diagnostics slimming, and before snapshot writes, browser payload handoff, and static Brief projection. Static export should run the same check before rendering so stale or malformed generated payloads do not become a polished but misleading artifact. The schema parser must remain backwards compatible. Legacy payloads can still parse with defaults through `parseTrendFinderData()` and `validateTrendFinderData()`. The closeout gate is a stricter generated/export assertion with sanitized issue output: field name, topic ID when applicable, scope, and code only. It must never include raw source content, prompts, provider responses, local paths, or private diagnostics. *** ## 2. Objectives 1. Add a reusable closeout assertion for required derived Trend Finder fields. 2. Gate only fields with existing schema defaults and deterministic fallback support. 3. Wire the assertion before browser payload handoff and static Brief rendering. 4. Emit sanitized validation issues that identify field and topic ID without raw content. 5. Add focused tests proving missing-field failure, legacy parse safety, static export blocking, payload budget visibility, and privacy cleanliness. *** ## 3. Prerequisites ### Required Sessions * [x] `phase29-session02-attention-pattern-and-polarity-grid` - Provides `attentionPattern` defaults and live/static grid projection. * [x] `phase29-session03-reception-signal-aggregate-only` - Provides `receptionSignal` defaults and aggregate-only reception projection. * [x] `phase29-session04-corroboration-gate` - Provides `corroboration` defaults, scoring caps, and source-summary projection. * [x] `phase29-session05-per-evidence-rationale` - Provides `evidenceRationales` defaults and projection behavior. * [x] `phase29-session06-cross-topic-substrate-narratives` - Provides `runNarratives` defaults and static/live projection. ### Required Tools/Knowledge * Bun 1.3.14 workflow through `bun run test`. * Existing Trend Finder schema defaults in `src/extensions/trend-finder/schema.ts`. * Existing collector closeout flow in `scripts/extensions/trend-finder/collector.ts`. * Existing static Brief export, renderer, and QA flow. * Existing payload-size reporter and private-string scan expectations. ### Environment Requirements * Project dependencies installed with Bun. * `src/data/live-data.example.json` available for fixture and schema validation. * No new source, media, credential, database, hosted storage, or third-party transfer approval is required. *** ## 4. Scope ### In Scope (MVP) * Trend Finder can assert required derived fields - Add a closeout helper that checks required topic and report fields on generated/exported payloads. * Required fields stay bounded - Gate `attentionPattern`, `receptionSignal`, `corroboration`, `evidenceRationales`, and `runNarratives` only because they have defaults and deterministic fallback support. * Missing fields fail before rendering - Throw or return a blocking closeout error before collector payload handoff and before static Brief projection. * Legacy parsing stays safe - Keep schema defaults so old payloads without these fields still parse through normal schema APIs. * Validation issues are sanitized - Report field path, topic ID, issue code, and scope only, with no raw evidence, source text, prompts, paths, or private details. * Static Brief stays honest - Prevent static Brief export from rendering a generated payload that is missing required derived branches. * Release checks prove behavior - Add unit, collector, schema, static export, e2e, payload-size, privacy, and ASCII validation coverage. ### Out of Scope (Deferred) * Gating fields without schema defaults or deterministic fallback - Reason: this session only closes fields already shipped in Sessions 02-06. * Source-death baseline alarming - Reason: Session 09 owns private last-good per-source count baselines. * Seed-candidate private review artifacts - Reason: Session 10 owns seed candidate generation. * Changing derived-field formulas, scoring weights, action verdicts, or UI layout - Reason: this is a closeout assertion, not a feature redesign. * Exposing raw source content in validation issues - Reason: closeout failures must remain browser-safe and log-safe. * New source collection, comments, podcasts, transcripts, media, hosted storage, database state, or compliance decisions - Reason: those remain explicit Phase 29 boundaries. *** ## 5. Technical Approach ### Architecture Create a focused `required-derived-fields` helper under `scripts/extensions/trend-finder/`. The helper should own required field definitions, closeout issue types, issue codes, issue caps, error construction, and safe message formatting. It should inspect the generated report shape before schema defaults can hide omitted fields. It should report only stable paths, field names, topic IDs, and safe scope labels. The required topic fields should be `attentionPattern`, `receptionSignal`, `corroboration`, and `evidenceRationales`. The report-level required branch should be `runNarratives`. `evidenceRationales` may be an empty object when no evidence rationale is available; the closeout check is presence and safe shape, not content invention. `runNarratives` may be an empty array; the check is that the generated branch exists and is array-shaped. Wire the helper into `collector.ts` after the final `data` object is assembled and slimmed for browser safety, before `writeSnapshotAfterValidation`, final Engine Replay rebuild, static-brief readiness logging, and return. Also run the helper inside collector `validateData()` so extension payload validation rejects missing generated fields. Static Brief export should run the helper on the raw Trend Finder extension data before report projection and rendering, while schema parsing remains defaultable for legacy callers. Extend static Brief QA or export errors as needed to expose a blocking, sanitized issue list. The closeout error details should be short enough for collector warnings, export CLI output, and tests. The payload-size reporter already includes the target branches; update it only if the closeout issue branch needs visibility. ### Design Patterns * Boundary-nearest assertion: Generated/exported payloads fail before browser or static rendering. * Additive schema evolution: Legacy payloads still parse with defaults outside strict closeout. * Sanitized diagnostics: Closeout issues expose topic IDs and field names only. * Deterministic fallback before gating: Required fields are limited to shipped deterministic fallback branches. * Reusable helper over ad hoc checks: Collector and static export use the same field definitions and issue formatting. ### Technology Stack * TypeScript for closeout contracts, strict issue typing, and exhaustive field definitions. * Zod schema defaults remain in `src/extensions/trend-finder/schema.ts`. * Bun and Vitest for script, schema, collector, and export tests. * Playwright for static Brief and Engine Replay regression guards when needed. * Existing static Brief privacy and payload-size utilities for boundary checks. *** ## 6. Deliverables ### Files to Create | File | Purpose | Est. Lines | | --------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------- | ---------- | | `scripts/extensions/trend-finder/required-derived-fields.ts` | Required field definitions, closeout issue builder, assertion helper, and sanitized error formatting. | \~220 | | `scripts/extensions/trend-finder/__tests__/required-derived-fields.test.ts` | Unit coverage for missing topic fields, missing report branches, allowed empty defaults, issue caps, and privacy-safe messages. | \~220 | ### Files to Modify | File | Changes | Est. Lines | | ----------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------- | ---------- | | `scripts/extensions/trend-finder/collector.ts` | Run closeout assertion before browser payload handoff, snapshot writes, static readiness logging, and `validateData()` success. | \~120 | | `scripts/extensions/trend-finder/static-brief-export.ts` | Run closeout assertion before static Brief report projection and expose sanitized blocking export errors. | \~80 | | `scripts/extensions/trend-finder/static-brief-qa.ts` | Add a blocking issue code or assertion hook for missing required derived fields if export QA needs to surface it. | \~40 | | `scripts/extensions/trend-finder/measure-payload-size.ts` | Keep required derived branches visible in payload reporting and add closeout issue visibility only if needed. | \~20 | | `src/extensions/trend-finder/schema.ts` | Preserve legacy defaults and export any constants/types needed by the closeout helper without weakening parsing. | \~40 | | `src/extensions/trend-finder/fixtures.ts` | Add fixture variants for complete derived fields and closeout-missing field cases. | \~70 | | `scripts/extensions/trend-finder/__tests__/collector.test.ts` | Cover collector rejection of missing generated derived fields and sanitized warning/error detail. | \~100 | | `scripts/extensions/trend-finder/__tests__/static-brief-export.test.ts` | Cover static export blocking for missing required derived fields and successful export for complete payloads. | \~100 | | `src/extensions/trend-finder/__tests__/view-model.test.ts` | Cover legacy schema parse defaults for required derived fields without invoking strict closeout. | \~60 | | `tests/e2e/trend-finder-static-brief.spec.ts` | Add an export regression proving missing required fields block before static rendering and complete payloads still render. | \~50 | | `tests/e2e/trend-finder-engine-replay.spec.ts` | Add or update a browser regression only if closeout issues surface in Engine Replay or extension error state. | \~40 | | `docs/extensions/trend-finder-pipeline.md` | Document the generated-payload closeout gate, sanitized issues, and legacy schema distinction. | \~50 | | `docs/extensions/trend-finder-ui-surfaces.md` | Document that live/static surfaces require derived fields already present in generated payloads. | \~30 | *** ## 7. Success Criteria ### Functional Requirements * [ ] A generated fixture topic missing `attentionPattern` fails closeout before browser/static rendering. * [ ] A generated fixture topic missing `receptionSignal` fails closeout before browser/static rendering. * [ ] A generated fixture topic missing `corroboration` fails closeout before browser/static rendering. * [ ] A generated fixture topic missing `evidenceRationales` fails closeout before browser/static rendering. * [ ] A generated payload missing `runNarratives` fails closeout before static Brief projection. * [ ] Empty but present fallback shapes, such as `evidenceRationales: {}` and `runNarratives: []`, pass closeout. * [ ] Legacy payloads relying on schema defaults still parse safely through normal schema APIs. * [ ] Validation issues identify only field, topic ID, scope, code, and bounded safe path. ### Testing Requirements * [ ] Unit tests written and passing for closeout helper issue detection, safe formatting, issue caps, and allowed empty defaults. * [ ] Collector tests prove generated payload validation fails before browser handoff when required fields are missing. * [ ] Static Brief export tests prove missing fields block before rendering and complete payloads still export. * [ ] Schema/view-model tests prove legacy default parsing remains safe. * [ ] Payload-size and privacy scan checks remain green. ### Non-Functional Requirements * [ ] Browser payload stays under the 1 MB extension budget. * [ ] Closeout issue messages do not expose raw source content, snippets, URLs beyond safe paths, prompts, provider responses, tokens, credentials, local paths, private cache paths, comments, transcripts, stack traces, or private diagnostics. * [ ] No new source, collector boundary, media boundary, database, hosted storage path, dependency, or third-party transfer is introduced. * [ ] Closeout checks are deterministic, bounded, and linear in topic count. ### Quality Gates * [ ] All files ASCII-encoded. * [ ] Unix LF line endings. * [ ] Code follows project conventions. * [ ] New enum or schema branches use explicit fallback behavior. *** ## 8. Implementation Notes ### Key Considerations * The helper should inspect generated/raw object presence instead of trusting schema defaults, otherwise missing fields will be hidden before the closeout gate can catch them. * `evidenceRationales` and `runNarratives` can validly be empty; missing and wrong-shaped are failures, empty present fallback is not. * Error messages must be useful for operators and tests while staying sanitized. Prefer `topicId=... field=...` over any topic name, source title, or evidence snippet. * The collector should fail before writing snapshots so invalid generated payloads do not become historical inputs. * Static Brief export should fail before HTML generation so a polished artifact never masks absent derived fields. ### Potential Challenges * TypeScript types may make missing fields hard to construct in tests. Mitigation: test the helper with unknown/raw records and collector/export paths with deliberately cast fixtures. * Static export currently validates through schema defaults. Mitigation: add the closeout helper before projection, separate from schema parse behavior. * Some fixture payloads may predate Sessions 02-06. Mitigation: update only current fixture paths used by closeout/export tests and keep legacy parse tests separate. * The closeout helper may duplicate field lists from schema. Mitigation: centralize definitions in the helper and import schema constants only for defaults/type names where useful. ### Relevant Considerations * \[P02] **Extension payloads and labels stay bounded**: Closeout issues must be capped and the required branches stay measured against the 1 MB payload budget. * \[P24] **Browser-safe export and triage boundaries**: Static Brief and browser payloads must exclude raw evidence, private paths, local triage notes, prompts, provider responses, and credentials. * \[P27] **Deterministic fallback before AI enrichment**: Required fields are limited to branches with deterministic fallback support. * \[P28] **Direct public source scope is narrow**: This session adds no source and does not widen deferred social or media candidates. * \[P28] **Trend Finder release validation is bundled**: Payload, privacy, docs/reference checks, and scoped formatting evidence should be recorded during implementation and closeout. * \[P00] **Do not document planned features as implemented**: Docs should be updated only after implementation lands and should distinguish legacy parse defaults from strict closeout. ### Behavioral Quality Focus Checklist active: Yes Top behavioral risks for this session: * Schema defaults hide a missing generated field and the UI renders an incomplete surface as if it is trustworthy. * Closeout errors leak raw evidence, provider output, private paths, or source content while trying to explain the failure. * Static Brief export renders an HTML artifact from a stale generated payload that lacks required derived fields. *** ## 9. Testing Strategy ### Unit Tests * Add helper tests for one missing topic field at a time, multiple missing fields, missing report branch, wrong-shaped report branch, allowed empty fallback shapes, issue cap behavior, and sanitized formatting. * Add schema tests proving `parseTrendFinderData()` still defaults missing legacy `attentionPattern`, `receptionSignal`, `corroboration`, `evidenceRationales`, and `runNarratives`. * Add payload reporter or QA tests only if closeout issue reporting changes their output. ### Integration Tests * Add collector validation tests that remove a required topic field from generated data and expect `validateData()` or collector closeout to reject it before success. * Add static Brief export tests that write live-data with a missing required field and expect a blocking `invalid_trend_finder_data` or closeout-specific export error before HTML output. * Add complete-payload export coverage to prove current fixtures still render with polarity, rationale, corroboration, and run narrative branches present. ### Manual Testing * Run focused Vitest suites for required-derived-fields, collector, static Brief export, and view-model defaults. * Run the static Brief e2e check and confirm complete payloads still produce HTML with no private strings or horizontal overflow. * Inspect a forced closeout error and confirm it contains field/topic identifiers only. ### Edge Cases * Topic has no evidence IDs and `evidenceRationales: {}`. * Payload has zero topics and `runNarratives: []`. * Payload has topics with present default values such as `attentionPattern: "unavailable"` and `receptionSignal: "unavailable"`. * Payload has a malformed `corroboration` object that schema would catch separately. * Multiple topics are missing fields and issue output is capped deterministically. *** ## 10. Dependencies ### External Libraries * None. The session uses existing TypeScript, Zod, Bun, Vitest, and Playwright tooling. ### Other Sessions * **Depends on**: `phase29-session02-attention-pattern-and-polarity-grid`, `phase29-session03-reception-signal-aggregate-only`, `phase29-session04-corroboration-gate`, `phase29-session05-per-evidence-rationale`, and `phase29-session06-cross-topic-substrate-narratives`. * **Depended by**: `phase29-session09-source-death-baseline-alarm` indirectly benefits from stronger closeout reliability, and `phase29-session18-documentation-validation-and-release` uses the bundled release-validation record. *** ## Next Steps Run the implement workflow step to begin AI-led implementation. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://ai-os-and-trend-finder.gitbook.io/ai-os-and-trend-finder-docs/.spec_system/archive/sessions/phase29-session08-required-derived-field-closeout-gate/spec.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.