> For the complete documentation index, see [llms.txt](https://ai-os-and-trend-finder.gitbook.io/ai-os-and-trend-finder-docs/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://ai-os-and-trend-finder.gitbook.io/ai-os-and-trend-finder-docs/.spec_system/archive/sessions/phase28-session11-search-palette-and-deterministic-embeddings/spec.md).

# Session Specification

**Session ID**: `phase28-session11-search-palette-and-deterministic-embeddings` **Phase**: 28 - Trend Finder Trends-Finderz Adoption **Status**: Not Started **Created**: 2026-06-14

***

## 1. Session Overview

This session adds a Trend Finder-local search command palette over the already loaded extension payload and ships the dependency-free feature-hash embeddings that Phase 27 deferred. The palette gives operators one keyboard-first way to search topic names, canonical IDs, aliases, categories, source names, evidence titles, and creator angles, then jump to the owning Trend Finder view with the topic selected.

The embedding work resolves ADR 0002 without adding a model dependency or vector database. Script-side code uses SHA-256 feature hashing over bounded text features for collector clustering, while browser code computes vectors locally from loaded payload text for UI ranking. No 384-float vectors are added to the browser payload.

Collector-side similarity strengthens the Phase 27 theme-rollup fallback and replaces the weakest deterministic fallback topic clustering path, which groups by source when AI analysis is unavailable. AI-mode topic clustering remains unchanged; this session only improves deterministic local fallback behavior and the client-side search surface.

***

## 2. Objectives

1. Add a keyboard-accessible Trend Finder command palette over the in-memory payload with bounded result sets and deterministic navigation.
2. Ship dependency-free SHA-256 feature-hash text vectors with cosine, lexical fallback, and hybrid ranking helpers.
3. Use similarity grouping to strengthen theme-rollup fallback and replace group-by-source fallback clustering for evidence titles.
4. Resolve the Phase 27 embedding ADR in writing with the feature-hash decision and no model dependency.

***

## 3. Prerequisites

### Required Sessions

* [x] `phase27-session11-theme-rollups-and-outlier-ideas` - Theme rollups exist and ADR 0002 records the prior no-model-dependency decision.
* [x] `phase28-session10-watchlist-pin-baselines-notes-tags` - Topic selection, local pin state, and hash navigation exist for palette jump targets.

### Required Tools/Knowledge

* Bun 1.3.14, TypeScript, Vitest, Playwright, React 19, Tailwind CSS 4.
* Existing `cmdk`/Radix command-dialog primitives in `src/components/ui/`.
* Existing Trend Finder payload parsing in `src/extensions/trend-finder/schema.ts`.
* Existing Trend Finder view-model, topic hash, and route patterns in `src/extensions/trend-finder/view-model.ts` and `views/trends-view.tsx`.
* Existing collector fallback paths in `scripts/extensions/trend-finder/topics.ts` and `scripts/extensions/trend-finder/theme-rollups.ts`.

### Environment Requirements

* No live AI runtime credentials are required; fallback clustering must improve when AI is unavailable.
* No new package dependency, vector database, source adapter, source call, or hosted storage path is introduced.
* Browser payloads remain bounded and must not include raw vectors, raw source payloads, private paths, prompts, provider responses, or credentials.

***

## 4. Scope

### In Scope (MVP)

* Trend Finder users can open a local palette with Ctrl/Cmd+K and `/`, search the loaded payload, filter result scope, select the best result with Enter, close with Escape, and navigate to the owning view.
* Trend Finder can build a bounded search index from topics, aliases, canonical keys, theme/category labels, source names, evidence titles/snippets, and creator/action angles without fetching new data.
* Trend Finder can rank palette results with client-side hybrid scoring: lexical match first, cosine similarity when vector generation succeeds, and pure lexical fallback when it does not.
* Collector-side code can create deterministic SHA-256 feature-hash vectors, compare cosine similarity, and group text with stable thresholds and tie breakers.
* Theme-rollup fallback can use similarity grouping above keyword overlap while preserving analyst labels when valid.
* Deterministic fallback topic clustering in `topics.ts` can cluster similar evidence titles instead of grouping by source when AI analysis is unavailable.
* ADR 0002 is updated or amended to record that feature-hash embeddings shipped without a model dependency.
* Tests cover vector determinism, L2 normalization, cosine grouping, palette ranking, lexical fallback, keyboard behavior, navigation, and payload-size safety.

### Out of Scope (Deferred)

* Embedding model dependency or vector database - *Reason: the session resolves the need with dependency-free feature hashing.*
* Persisting vectors in generated browser payloads - *Reason: size and privacy boundary; vectors are computed locally when needed.*
* Replacing AI-mode analyst clustering - *Reason: AI-mode clustering remains the higher-fidelity path and is not changed by this session.*
* Server-side search endpoint - *Reason: the palette searches already-loaded local extension payload data only.*
* New source adapters or changed Actor inputs - *Reason: collection changes are owned by later compliance-gated sessions.*

***

## 5. Technical Approach

### Architecture

Create a shared feature-extraction layer that normalizes text, filters stopwords, expands a small reviewed synonym map, emits weighted token/stem, bigram, trigram, and character 4-gram features, and exposes vector math helpers. Script-side vectors use `node:crypto` SHA-256 for synchronous feature hashing. Browser vectors use Web Crypto for the same SHA-256 index/sign derivation and fall back to lexical ranking if async vector work fails, times out, or becomes stale after the query changes.

Add a Trend Finder search index builder that projects the parsed payload into bounded documents. Documents carry only display-safe strings, result kind, owning view, topic ID, score metadata, and matched fields. The palette component consumes that index, renders scoped results through the existing command-dialog primitive, manages focus and keyboard shortcuts, and navigates to `/extensions/trend-finder/<view>#topic:<id>` for topic-owned results.

Collector-side use stays deterministic and browser-safe. Theme rollups accept validated analyst labels first, then use vector similarity to merge related fallback groups where the threshold is met, with keyword overlap as a fallback. Fallback topic extraction clusters evidence titles by similarity instead of source ID, while retaining stable ordering, bounded group sizes, and explicit fallback provenance.

### Design Patterns

* Project before rendering: Palette consumes a safe search-index projection, not raw collector internals.
* Deterministic fallback first: Lexical ranking and token overlap remain the fallback when vector generation is unavailable.
* Script/browser boundary: Node hashing stays in scripts; browser hashing uses Web Crypto; no Node-only imports enter client bundles.
* Bounded lists: Search documents, matched fields, result rows, theme groups, and fallback clusters have explicit limits and stable tie-breakers.
* Existing primitives: Use local `Command`, `Dialog`, Button, Tailwind, and lucide conventions instead of adding UI dependencies.
* No payload vector arrays: Generated data keeps strings and IDs only; vectors are transient computed values.

### Technology Stack

* TypeScript 6, Bun 1.3.14, Vite/TanStack Start.
* React 19, existing `cmdk` command primitive, Radix Dialog, lucide-react.
* Node `crypto` for script-side SHA-256; browser Web Crypto for client-side SHA-256.
* Vitest with happy-dom for unit/component tests and Playwright for focused route-level keyboard validation if needed.

***

## 6. Deliverables

### Files to Create

| File                                                                             | Purpose                                                                                | Est. Lines |
| -------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------- | ---------- |
| `src/lib/trend-text-vector-features.ts`                                          | Shared tokenizer, feature weights, lexical scoring, cosine math, and vector contracts. | \~220      |
| `scripts/lib/ai-runtime/text-vectors.ts`                                         | Script-side SHA-256 feature-hash vectors, similarity grouping, and hybrid scoring.     | \~220      |
| `src/extensions/trend-finder/text-vectors.ts`                                    | Browser Web Crypto feature-hash vectors with lexical fallback behavior.                | \~180      |
| `src/extensions/trend-finder/search-index.ts`                                    | Safe in-memory Trend Finder search-index builder and bounded ranking helper.           | \~260      |
| `src/extensions/trend-finder/components/trend-search-palette.tsx`                | Keyboard command palette UI, scoped result groups, and navigation handling.            | \~320      |
| `scripts/lib/ai-runtime/__tests__/text-vectors.test.ts`                          | Script vector determinism, normalization, cosine, and grouping tests.                  | \~180      |
| `src/lib/__tests__/trend-text-vector-features.test.ts`                           | Shared feature and lexical scoring tests.                                              | \~140      |
| `src/extensions/trend-finder/__tests__/search-index.test.ts`                     | Palette index and ranking tests over fixture data.                                     | \~180      |
| `src/extensions/trend-finder/components/__tests__/trend-search-palette.test.tsx` | Component keyboard, scope, focus, empty, and navigation tests.                         | \~180      |

### Files to Modify

| File                                                              | Changes                                                                                                     | Est. Lines |
| ----------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------- | ---------- |
| `scripts/extensions/trend-finder/theme-rollups.ts`                | Add similarity-assisted fallback grouping while preserving analyst labels and deterministic ordering.       | \~140      |
| `scripts/extensions/trend-finder/topics.ts`                       | Replace source-ID fallback clustering with similarity clusters over evidence titles.                        | \~160      |
| `scripts/extensions/trend-finder/collector.ts`                    | Wire similarity theme grouping, fallback clustering warnings, and trace summaries without exposing vectors. | \~80       |
| `src/extensions/trend-finder/client.tsx`                          | Mount palette state and keyboard shortcut handling inside the Trend Finder shell.                           | \~80       |
| `src/extensions/trend-finder/view-model.ts`                       | Export search-safe labels and navigation helpers used by the palette index.                                 | \~80       |
| `src/extensions/trend-finder/fixtures.ts`                         | Add alias, evidence, source, and creator-angle fixture coverage for palette ranking.                        | \~80       |
| `scripts/extensions/trend-finder/__tests__/theme-rollups.test.ts` | Cover similarity grouping, analyst precedence, lexical fallback, and thresholds.                            | \~120      |
| `scripts/extensions/trend-finder/__tests__/collector.test.ts`     | Cover no-vector-payload publication and deterministic fallback clustering integration.                      | \~120      |
| `docs/adr/0002-trend-finder-embedding-fallback-clustering.md`     | Amend ADR with the shipped feature-hash decision and no model dependency.                                   | \~80       |
| `docs/extensions/trend-finder-ui-surfaces.md`                     | Document the search palette behavior and browser-only ranking boundary.                                     | \~80       |
| `docs/extensions/trend-finder-runtime-and-provenance.md`          | Document deterministic feature-hash clustering and fallback provenance.                                     | \~80       |

***

## 7. Success Criteria

### Functional Requirements

* [ ] Palette opens with Ctrl/Cmd+K and `/`, closes with Escape, focuses the search input, and does not steal typing shortcuts from inputs, textareas, or contenteditable elements.
* [ ] Palette searches topic names, canonical IDs, aliases, categories/themes, source names, evidence titles/snippets, and creator/action angles from the loaded payload only.
* [ ] Enter selects the best result and navigates to the owning view with a `#topic:<id>` target when the result belongs to a topic.
* [ ] Result sets are bounded, deterministically ordered, and show explicit loading, empty, error, and offline/fallback states.
* [ ] Feature-hash vectors are deterministic, L2-normalized, dependency-free, and backed by lexical fallback when vector ranking is unavailable.
* [ ] No generated browser payload stores 384-float vector arrays.
* [ ] Theme-rollup fallback grouping improves over keyword overlap while preserving valid analyst labels.
* [ ] Fallback topic clustering uses title similarity instead of source grouping when AI analysis is unavailable.
* [ ] ADR 0002 records the shipped feature-hash outcome without adding a model dependency.

### Testing Requirements

* [ ] Unit tests cover vector determinism, cosine thresholds, lexical fallback, bounded grouping, and no-vector-payload publication.
* [ ] View-model/index tests cover search scopes, aliases, source/evidence matching, hybrid ranking, and deterministic tie-breakers.
* [ ] Component tests cover keyboard open/close, focus, scope changes, Enter selection, empty/error states, and input shortcut suppression.
* [ ] Collector tests cover similarity grouping and fallback topic clustering without changing AI-mode clustering.
* [ ] Manual or e2e validation confirms the palette opens and navigates on the Trend Finder route.

### Non-Functional Requirements

* [ ] No new dependencies or vector database are added.
* [ ] Browser payload remains under the existing Trend Finder payload boundary.
* [ ] Palette remains keyboard reachable, screen-reader labeled, and reduced-motion safe.
* [ ] All files ASCII-encoded.
* [ ] Unix LF line endings.
* [ ] Code follows project conventions.

### Quality Gates

* [ ] Focused Vitest suites pass for vector helpers, search index, palette, theme rollups, topics, collector, and schema/payload safety.
* [ ] `bun run typecheck` and `bun run typecheck:scripts` pass or scoped failures are documented.
* [ ] No secret, private path, raw source payload, prompt, provider response, or credential-shaped string is introduced into browser data.

***

## 8. Implementation Notes

### Key Considerations

* Use source code as source of truth for existing topic hash navigation. The Trends view already supports `#topic:<id>` and can show suppressed topics when a hash target points to one.
* Keep the palette extension-local. It should not become a global AI OS command system and should not search host telemetry.
* Treat Web Crypto vector ranking as asynchronous. Ignore stale results when the query or payload changes, and keep lexical ranking available immediately.
* Keep synonym expansions as reviewed local config and avoid implying that they are new source approvals or keyword-pack collection inputs.
* The client index can include document text, but not raw provider responses, cache paths, local file paths, raw Actor input, Dataset rows, or credentials.

### Potential Challenges

* Browser and Node SHA-256 parity: Share feature extraction and test stable script outputs; test client ranking behavior through deterministic fixtures.
* Async search ranking races: Use query/run keys and discard stale vector results after scope or query changes.
* Palette shortcut conflicts: Ignore `/` inside editable elements and respect Ctrl/Cmd+K only as a Trend Finder-local shortcut while the extension shell is mounted.
* Similarity thresholds: Start conservative so unrelated topics do not merge; preserve keyword overlap fallback and stable tie-breakers.
* Payload safety regression: Add tests that fail if vector arrays or private strings are published into generated data.

### Relevant Considerations

* \[P00] **Stack conventions**: Use Bun, Vite, TanStack Start, React 19, Radix/cmdk, Tailwind CSS 4, and existing test locations.
* \[P02] **Extension payloads and labels stay bounded**: Keep search documents and result lists bounded; do not store vectors in payloads.
* \[P05] **Script-only runtime boundary**: Collector similarity helpers belong in scripts or shared client-safe utilities, with Node-only crypto kept out of browser bundles.
* \[P24] **Browser-safe export and triage boundaries**: Do not expose local triage notes, private paths, or raw evidence in palette results.
* \[P27] **Trend Finder payload growth needs release checks**: Add payload-safety assertions because search and embeddings touch many text fields.
* \[P27] **Deterministic fallback before AI enrichment**: Improve fallback clustering without weakening AI-mode analyst paths.

### Behavioral Quality Focus

Checklist active: Yes Top behavioral risks for this session:

* Keyboard shortcut handling can conflict with host inputs or leak listeners after route changes.
* Async vector ranking can show stale results after query, scope, or payload changes.
* Search and clustering can accidentally become unbounded or publish vectors in generated browser payloads.

***

## 9. Testing Strategy

### Unit Tests

* Test feature extraction normalization, stopword filtering, synonym expansion, lexical scoring, cosine similarity, vector dimensions, and L2 normalization.
* Test script-side vector determinism with stable SHA-256 results and grouping thresholds.
* Test search index documents for bounded fields, scopes, aliases, evidence, sources, angles, and deterministic ranking.
* Test theme rollups and fallback topic clustering with related and unrelated topic/evidence fixtures.

### Integration Tests

* Test collector output to prove vectors are not serialized into browser data and fallback clustering does not change AI-mode clustering.
* Test palette component integration with parsed fixture payloads, scope changes, Enter navigation, and empty/error states.
* Run focused type checks for both app and scripts because this session crosses the browser/script boundary.

### Manual Testing

* Open `/extensions/trend-finder/trends`, press Ctrl/Cmd+K and `/`, search for a topic alias, evidence title, source name, and creator angle, then verify the selected result navigates to the expected Trend Finder view and selected topic.
* Verify Escape closes the palette, focus returns to the trigger context, and shortcuts do not fire while typing in existing inputs.
* Use reduced-motion preference to confirm the palette does not introduce motion that obscures analysis.

### Edge Cases

* Empty payload, disabled extension, or malformed legacy payload.
* Query with stopwords only, punctuation-only query, duplicate aliases, missing evidence IDs, missing source rows, hidden/suppressed topics, and offline browser state.
* Vector generation unavailable, Web Crypto failure, stale async search result, and all-zero vectors.
* Similar evidence titles from different sources, unrelated same-source titles, and analyst-provided theme labels that must not be overridden.

***

## 10. Dependencies

### External Libraries

* No new dependencies.
* Existing `cmdk`, Radix Dialog, lucide-react, React 19, Tailwind CSS 4, Vitest, and Playwright may be used.

### Other Sessions

* **Depends on**: `phase27-session11-theme-rollups-and-outlier-ideas`, `phase28-session10-watchlist-pin-baselines-notes-tags`
* **Depended by**: `phase28-session12-brief-qa-markdown-export-and-kpi-strip`, `phase28-session15-documentation-validation-and-release`

***

## Next Steps

Run the implement workflow step to begin AI-led implementation.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://ai-os-and-trend-finder.gitbook.io/ai-os-and-trend-finder-docs/.spec_system/archive/sessions/phase28-session11-search-palette-and-deterministic-embeddings/spec.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.