> For the complete documentation index, see [llms.txt](https://ai-os-and-trend-finder.gitbook.io/ai-os-and-trend-finder-docs/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://ai-os-and-trend-finder.gitbook.io/ai-os-and-trend-finder-docs/scripts/extensions/readme_extensions-scripts.md).

# Extension Collector Scripts

The `scripts/extensions/` directory contains script-side collectors for repo-local AI OS extensions. Collectors run during `bun run aggregate` and publish browser-safe extension payloads into `LiveData.extensions.items[extensionId]`.

Client-side extension code lives under `src/extensions/`. Shared collector contracts and runner safeguards live under `scripts/lib/extensions/`.

## Structure

| Path                | Purpose                                                               |
| ------------------- | --------------------------------------------------------------------- |
| `registry.ts`       | Static collector registry used by the aggregate script                |
| `trend-finder/`     | Trend Finder collector, normalization, source adapters, traces, tests |
| `<id>/collector.ts` | Optional collector entry point for a future extension                 |

The current registered collector is `trend-finder`.

## Runtime Flow

`scripts/aggregate.ts` loads environment values, parses `VITE_CLAUDE_OS_ENABLED_EXTENSIONS`, and calls the guarded extension runner. The runner then:

1. Skips collectors whose IDs are not enabled.
2. Checks required environment keys without printing secret values.
3. Creates a private cache directory at `.cache/extensions/<extension-id>/`.
4. Passes each collector a scoped context with declared env access, logger, trace logger, abort signal, cache path, and current-time helper.
5. Enforces collector timeout and payload size limits.
6. Validates the returned shape and calls the collector's `validateData()`.
7. Writes ready, disabled, warning, or error state into generated live data.

Generated payloads become browser-visible through `/__live-data`. Private cache files, raw logs, raw provider prompts or responses, account auth files, and generated `src/data/live-data.json` must remain untracked.

## Collector Contract

Collectors implement `CollectorExtension` from [`scripts/lib/extensions/types.ts`](https://github.com/moshehbenavraham/ai-os/blob/main/scripts/lib/extensions/types.ts).

Required fields include:

* `id`, `name`, and `version`.
* `requiredEnvKeys` and optional `optionalEnvKeys`.
* `capabilities` describing collector permissions.
* `validateData(data)` for final payload validation.
* `collect(ctx)` returning `{ data, warnings? }`.

The runner exposes only declared env keys through `ctx.env(key)`. Attempting to read an undeclared key throws, so add new private env names to the collector declaration before using them.

Warnings should be browser-safe. Do not include API keys, account IDs, private paths, raw prompts, raw provider responses, Dataset internals, or personal local telemetry in warnings or payload data.

## Trend Finder Collector

Trend Finder is the working collector implementation.

| File or Folder                 | Purpose                                                           |
| ------------------------------ | ----------------------------------------------------------------- |
| `trend-finder/collector.ts`    | Main Trend Finder pipeline and `CollectorExtension` declaration   |
| `trend-finder/engine-trace.ts` | Sanitized Engine Replay trace construction                        |
| `trend-finder/normalize.ts`    | Evidence normalization helpers                                    |
| `trend-finder/topics.ts`       | Deterministic topic extraction helpers                            |
| `trend-finder/sources/`        | Source adapters, source health, compliance gates, and normalizers |
| `trend-finder/**/__tests__/`   | Collector, source, normalization, and trace tests                 |

Current source paths include:

* Built-in Hacker News public API collection.
* Reviewed configured Apify Actor collection for approved public sources when `APIFY_TOKEN` and reviewed source config are present.
* Deterministic fallback topics when AI runtime or external sources are unavailable.
* Optional AI analysis through the script-side AI runtime when configured.
* Private snapshots, historical context, predictions, retros, and Creator Lens config under `.cache/extensions/trend-finder/`.
* Sanitized Engine Replay output copied into the browser payload.

See [`docs/extensions/trend-finder/pipeline.md`](/ai-os-and-trend-finder-docs/docs/extensions/trend-finder/pipeline.md) for the full Trend Finder aggregate path.

## Enablement

Collectors are disabled unless their extension ID is present in `VITE_CLAUDE_OS_ENABLED_EXTENSIONS`.

```bash
VITE_CLAUDE_OS_ENABLED_EXTENSIONS=trend-finder bun run aggregate
```

For persistent local enablement, put the same value in `.env.local` and rerun:

```bash
bun run aggregate
```

The `VITE_CLAUDE_OS_*` name is a compatibility contract. Do not rename it without an explicit migration.

## Adding A Collector

1. Create `scripts/extensions/<id>/collector.ts`.
2. Export a `CollectorExtension` with the same stable ID used by the client extension under `src/extensions/<id>/`.
3. Declare every required and optional env key.
4. Validate the collector payload before it reaches live data.
5. Register the collector in `scripts/extensions/registry.ts`.
6. Add focused tests under `scripts/extensions/<id>/__tests__/`.
7. Update relevant docs, source compliance notes, and client schema docs.

For public-source collectors, create or update `docs/sources/source-compliance-<source>.md` before implementation. Keep source collection metadata public, bounded, and attributable.

## Validation

Run focused tests for collector changes:

```bash
bun run test -- scripts/lib/__tests__/extensions-runner.test.ts scripts/extensions/trend-finder/__tests__ scripts/extensions/trend-finder/sources/__tests__
```

When collector types, env handling, source parsing, scoring, or payload schemas change, also run:

```bash
bun run typecheck:scripts
bun run test
bun run runtime:check-private
```

Use `bun run aggregate` as a local smoke check when you need to verify actual generated runtime data. The aggregate logs are private diagnostics under `logs/` and should not be committed.

## Related Docs

* [`scripts/README_scripts.md`](/ai-os-and-trend-finder-docs/scripts/readme_scripts.md) - all script commands and shared script runtimes.
* [`docs/extensions/README_docs-extensions.md`](/ai-os-and-trend-finder-docs/docs/extensions/readme_docs-extensions.md)
  * client and collector extension guide.
* [`docs/api/README_api.md`](/ai-os-and-trend-finder-docs/docs/api/readme_api.md) - local live-data delivery notes.
* [`docs/testing.md`](/ai-os-and-trend-finder-docs/docs/testing.md) - test locations, commands, and coverage policy.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://ai-os-and-trend-finder.gitbook.io/ai-os-and-trend-finder-docs/scripts/extensions/readme_extensions-scripts.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.