> For the complete documentation index, see [llms.txt](https://ai-os-and-trend-finder.gitbook.io/ai-os-and-trend-finder-docs/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://ai-os-and-trend-finder.gitbook.io/ai-os-and-trend-finder-docs/.spec_system/archive/sessions/phase15-session03-aggregate-orchestration-refactor/spec.md). # Session Specification **Session ID**: `phase15-session03-aggregate-orchestration-refactor` **Phase**: 15 - Scheduler Config And Aggregate Split **Status**: Complete **Created**: 2026-05-31 *** ## 1. Session Overview This session extracts reusable aggregate orchestration from the current monolithic `scripts/aggregate.ts` path while keeping `bun run aggregate` behavior-compatible. The existing full aggregate command scans AI OS host and local-agent data, loads Dream projection data, runs registered extension collectors, validates generated live data, and writes `src/data/live-data.json`. Later Phase 15 sessions need the same pieces to run as scoped scheduler jobs without duplicating this full command. The main outcome is a script-side orchestration surface that can build host/local live data without executing Trend Finder collectors, and can run extension orchestration separately for the future Trend Finder job. This session does not register new scheduler handlers or package scripts; it only creates the compatibility-preserving extraction that Sessions 04 and 05 will wire into scheduler jobs. The implementation should favor small, typed helper boundaries over a broad rewrite. `scripts/aggregate.ts` remains the public compatibility entry point and should delegate to extracted helpers only after focused tests prove output shape, validation, warning behavior, and privacy boundaries remain intact. *** ## 2. Objectives 1. Extract reusable host/local aggregate orchestration from `scripts/aggregate.ts`. 2. Provide a host/local aggregate mode that does not run Trend Finder collector work. 3. Preserve Dream projection loading, local-agent assembly, validation, logs, warnings, and generated live-data compatibility. 4. Expose extension orchestration separately so a later Trend Finder scheduler job can write only its owned branch. *** ## 3. Prerequisites ### Required Sessions * [x] `phase15-session01-scheduler-config-contract` - Provides scheduler config parsing, sanitized status behavior, and reviewed registry constraints. * [x] `phase15-session02-live-data-merge-boundary` - Provides the shared generated-data lock, ownership map, and scoped merge writer. ### Required Tools/Knowledge * Bun script workflow and focused Vitest commands. * Current `scripts/aggregate.ts` host/local scan, Dream load, extension collector, logger, and process guard behavior. * Existing `writeGeneratedLiveDataForScope` full and scoped write semantics. * Trend Finder extension collector enablement and backtest summary preservation. ### Environment Requirements * Repository dependencies installed with Bun. * No private `src/data/live-data.json`, logs, prompts, auth files, or source payloads copied into committed tests or docs. * Tests must use fixtures, temporary roots, or mocks for collector and scan boundaries. *** ## 4. Scope ### In Scope (MVP) * AI OS operators can keep using `bun run aggregate` - preserve command output, summary status, generated data shape, and write validation. * Future `agent-aggregate` can refresh host/local data - expose a helper or mode that builds host/local branches, includes Dream projection data, and skips Trend Finder collector execution. * Future `trend-finder` can run extension work independently - expose extension orchestration that returns a validated `ExtensionRuntimeState` without rebuilding host/local scans. * Aggregate warnings stay safe and deduplicated - preserve existing warning normalization and redaction behavior. * Generated data remains producer-validated - keep all writes routed through `writeGeneratedLiveDataForScope`. * Compatibility is tested - add focused tests proving full aggregate behavior and host/local no-collector behavior. ### Out of Scope (Deferred) * Registering `agent-aggregate` or `trend-finder` scheduler handlers - *Reason: Session 04 and Session 05 own scheduler wiring.* * Adding package scripts for scoped jobs - *Reason: package command contracts land with the handlers.* * Timer rendering, install, uninstall, or drift reporting - *Reason: Session 07 is the only timer-management implementation pass.* * SQLite observation store writes - *Reason: not part of Phase 15 MVP.* * New Trend Finder source adapters or AI planner behavior - *Reason: this session only separates existing collector orchestration.* *** ## 5. Technical Approach ### Architecture Create a script-side aggregate orchestration module under `scripts/lib/` that owns reusable aggregate modes and typed intermediate results. The compatibility path should still run the same sequence as today: install process guards, load env, scan host/local sources, load Dream projection, detect extension env needs, run extension collectors, preserve Trend Finder backtest summaries, assemble the final `LiveData` object, sanitize for emission, and write a full generated-data artifact. The host/local mode should share the host scan and live-data assembly helpers but avoid invoking collector `collect()` functions. It may emit a valid disabled or preserved extension runtime placeholder only as needed to satisfy the `LiveData` producer contract; the later scoped writer will preserve the existing extension branch when writing host/local output. The extension orchestration should be callable independently from host/local scans. It should preserve the current enablement, missing-env, timeout, warning, payload validation, and backtest summary behavior, while keeping source payloads, prompts, auth paths, Actor inputs, and provider credentials out of logs and summaries. ### Design Patterns * Compatibility wrapper: Keep `scripts/aggregate.ts` as the command entry point while moving reusable behavior behind typed helpers. * Extract pure functions, then test, then wire: Move deterministic assembly and warning helpers before changing the command path. * Explicit aggregate modes: Use named modes or helpers for full, host/local, and extension orchestration rather than boolean flags with unclear behavior. * Typed degradation: Return warning-only or degraded summaries for expected source failures instead of throwing when the existing aggregate path does not throw. ### Technology Stack * TypeScript scripts under `scripts/` and `scripts/lib/`. * Bun and Vitest for focused script tests. * Existing extension runner, aggregate run logger, process guards, sanitizer, live-data validation, and generated-data write helpers. *** ## 6. Deliverables ### Files to Create | File | Purpose | Est. Lines | | ------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------- | ---------- | | `scripts/lib/aggregate-orchestration.ts` | Reusable aggregate modes, typed intermediate data, host/local assembly, extension orchestration, and compatibility summary helpers. | \~350 | | `scripts/lib/__tests__/aggregate-orchestration.test.ts` | Focused tests for compatibility assembly, host/local no-collector mode, extension orchestration separation, warnings, and write scopes. | \~260 | ### Files to Modify | File | Changes | Est. Lines | | ----------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------- | ---------- | | `scripts/aggregate.ts` | Delegate the full compatibility command to the extracted orchestration while preserving public output and summary behavior. | \~120 | | `scripts/lib/aggregate-live-data-write.ts` | Adjust exported types or helper names only if orchestration needs stable scoped write imports. | \~20 | | `scripts/lib/scheduler/aggregate-handler.ts` | Keep the existing aggregate scheduler handler behavior compatible if `runAggregate` return types move. | \~20 | | `scripts/lib/__tests__/scheduler-aggregate-handler.test.ts` | Update aggregate handler mocks or assertions if `runAggregate` import boundaries change. | \~60 | | `docs/runbooks/scheduled-aggregate.md` | Document that aggregate orchestration is now reusable while scoped jobs remain deferred until later sessions. | \~60 | *** ## 7. Success Criteria ### Functional Requirements * [x] `bun run aggregate` remains the full compatibility command with unchanged public behavior. * [x] Host/local orchestration can build validated live-data output without running Trend Finder collector work. * [x] Dream projection loading remains part of host/local refresh behavior. * [x] Local agent, session, config, memory, integration, skill, automation, and detection assembly remain present in the host/local path. * [x] Extension orchestration can run separately and return the same `ExtensionRuntimeState` semantics used by the full path. * [x] All generated live-data writes still pass through producer validation. ### Testing Requirements * [x] Unit tests written and passing. * [x] Focused aggregate and scheduler handler tests passing. * [x] Manual compatibility check completed for the aggregate command boundary. ### Non-Functional Requirements * [x] Private paths, raw command bodies, prompts, auth JSON, source dumps, Actor inputs, provider auth paths, and credentials remain absent from emitted status/log/browser surfaces. * [x] Host/local mode avoids external Trend Finder collector calls unless full mode is explicitly selected. * [x] Refactor keeps route and browser code untouched. ### Quality Gates * [x] All files ASCII-encoded. * [x] Unix LF line endings. * [x] Code follows project conventions. *** ## 8. Implementation Notes ### Key Considerations * `scripts/aggregate.ts` currently performs scanning, extension collection, data assembly, validation, writing, logging, and summary generation in one function. Extract carefully so command output and warning behavior do not drift. * The host/local helper must still include Dream projection loading because Session 04 uses it for the future local-agent scheduler job. * `runExtensionCollectors` already has disabled and missing-env behavior; use that existing contract rather than duplicating collector state logic. * `preserveTrendFinderBacktestAggregateSummary` belongs with extension orchestration so future Trend Finder refreshes keep the Phase 14 publish behavior. * Keep `writeGeneratedLiveDataForScope({ scope: "full" })` on the compatibility command. Later sessions will call host/local or Trend Finder scoped writes. ### Potential Challenges * Output drift from extraction: Mitigate with fixture or mock-based tests that compare representative full aggregate output before and after helper wiring. * Hidden collector execution in host/local mode: Mitigate by injecting a collector runner mock that fails the test if called. * Warning or log duplication: Mitigate by keeping warning normalization and dedupe in one helper and testing warning counts. * Process guard cleanup on failures: Mitigate with `try/finally` ownership in the orchestration wrapper. ### Relevant Considerations * \[P02] **Collector timeout of 120s is fragile**: Host/local mode should skip collector work instead of paying extension timeout cost. * \[P06] **Demo labels must stay explicit**: Extension runtime states must keep live, fixture, fallback, disabled, degraded, and blocked labels precise. * \[P11] **Scheduler state/log privacy boundary**: Summaries and logs may expose only safe status, duration, exit code, warning count, lock, and freshness metadata. * \[P12] **Dream private-output boundary**: Dream loading stays private-output backed and must not introduce a browser gateway or service expansion. * \[P01] **Extract pure functions, then test, then wire**: This is the preferred sequence for reducing monolithic script risk. ### Behavioral Quality Focus Checklist active: Yes Top behavioral risks for this session: * A refactor changes full aggregate output or command behavior silently. * Host/local mode still runs heavy Trend Finder collector work. * Logs or summaries leak private paths, source payloads, prompts, or auth material during the extraction. *** ## 9. Testing Strategy ### Unit Tests * Test full orchestration assembles a representative `LiveData` object and calls the write helper with `scope: "full"`. * Test host/local orchestration does not call Trend Finder collector `collect()` functions and still includes Dream projection, local-agent data, and a valid extension runtime placeholder. * Test extension orchestration can run independently and preserves current warning, disabled, missing-env, timeout, and backtest summary behavior. * Test warning normalization and dedupe remain stable across extracted helpers. * Test process guard, logger, and console capture cleanup on success and failure paths where practical. ### Integration Tests * Run focused aggregate orchestration tests. * Run existing aggregate scheduler handler tests. * Run extension runner or Trend Finder collector tests affected by import movement. * Run script typecheck to catch extracted type drift. ### Manual Testing * Run or inspect the full aggregate command boundary and confirm it still writes through the producer gate. * Confirm docs do not claim `agent-aggregate` or `trend-finder` scheduler jobs are runnable yet. ### Edge Cases * Extension collector disabled by env. * Extension collector enabled but missing required env keys. * Extension collector warning-only result. * Existing Trend Finder backtest publish summary exists in live data. * Dream output missing, primary, or legacy fallback. * Host/local scan warning occurs while extension collectors are skipped. * Write validation rejects output before disk write. *** ## 10. Dependencies ### External Libraries * Bun runtime and test runner from the existing project toolchain. * Vitest through the existing `bun run test` script. ### Other Sessions * **Depends on**: `phase15-session01-scheduler-config-contract`, `phase15-session02-live-data-merge-boundary` * **Depended by**: `phase15-session04-agent-aggregate-scheduler-job`, `phase15-session05-trend-finder-scheduler-job`, `phase15-session06-cross-job-integration-dream-freshness` *** ## Next Steps Run the implement workflow step to begin AI-led implementation. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://ai-os-and-trend-finder.gitbook.io/ai-os-and-trend-finder-docs/.spec_system/archive/sessions/phase15-session03-aggregate-orchestration-refactor/spec.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.