Smart Skills Platform

The company went AI-native. Every function — engineering, analytics, PM, marketing, people-ops — got Claude Code, Cursor, and a starter library of SKILL.md files: reusable agent workflows for things like "open a PR" or "query Redshift." Anyone could drop one into their session and get a sharp, on-brand result instead of a generic one.

The library was good. The system around it was breaking down in three predictable ways:

1. Distribution was manual. To use a skill you copy-pasted a markdown file into the right folder, hoped you got the path right, and trusted the README was current. Most people didn't bother. They reinvented the workflow in-prompt every time.
2. Discovery was worse. "Is there a skill for X?" got answered by Slack-scrolling. No search, no ratings, no signal of which skills were good or stale. The catalog was invisible at the moment of need.
3. We had no idea what was missing. Curators wrote skills based on intuition. No systematic view of the prompts the catalog failed to answer. No evidence trail when prioritising what to build next.

A skill library without a discovery surface, without quality signals, and without a feedback loop is just a folder of markdown files. So we built the platform around it.

The deeper bet: the same agent runtime that consumes skills can also produce evidence about them. Hooks fire on every event. The trainer threads them. Clusters surface gaps. Curators promote drafts into real skills. Every loop tightens the catalog. The product is the methodology rendered legible.

A live, internally-deployed product that closes the loop end-to-end.

1 · The catalog

Browsable, searchable directory of every SKILL.md, mirrored from the canonical internal/agent-skills repo. Live filter search backed by OpenAI text-embedding-3-small and Supabase pgvector — semantic, not just substring. Each skill shows description, tags, scope, required MCPs, install count, and average rating.

2 · Skill detail pages

Full canonical SKILL.md content rendered editorially. Inline ratings (1–5 + comment) and usage stats. One-command install instructions for Claude Code and Cursor.

3 · Ask — RAG chat

Natural-language questions about the catalog, answered by Claude Sonnet 4.6 doing RAG over skill content with citations. Used for "how do I write a release note?", "which skill should I use to query Redshift?", "what's our convention for PRDs?"

4 · Two MCP servers

Both work with Claude Code and Cursor today. The user-facing MCP lets agents search and manage the catalog from inside the session, no context switch: find_skill(query), install_skill(slug), rate_skill(slug, rating, comment), ask(question). Browser-based Google SSO restricted to the company domain. Token cached locally.

The trainer MCP is the behind-the-scenes one. record_observation captures prompt, tool call, file edit, or agent response with full context. list_opportunities, classify_opportunity, and draft_skill_proposal drive the curator's queue. log_recommendation_outcome tracks whether surfaced skills got accepted, dismissed, ignored, or installed.

5 · Hooks — universal capture across both clients

Project hooks in .cursor/hooks.json and .claude/hooks/ give us the same observation shape across both surfaces:

A single user task can span Cursor and Claude Code — start exploring in Cursor, switch to Claude Code mid-task — and the trainer threads them together via shared session_id + recency signals.

6 · The trainer admin — the curator's queue

The thing nothing else internally had: a UI that turns raw AI usage into prioritised skill investments backed by real evidence. Clusters of similar prompts the catalog doesn't serve well, ranked by opportunity_score. Cluster detail with thread timeline: the initial prompt, follow-ups, tool traces, agent responses, and outcomes — not isolated prompts. Each opportunity is typed: missing_skill, improve_skill, promote_skill, dependency_gap. Curator actions: Promote, Mark Reviewed, Needs More Evidence, Archive.

The single insight: the same agent runtime that consumes skills can also produce evidence about them. Hooks fire on every event. The trainer threads them. The clustering surfaces gaps. The curator promotes a draft into the catalog. The MCP installs the new skill into the next session that asks for it. Every loop tightens the catalog.

The hardest engineering problem on the project: turning a stream of disconnected hook events into a coherent picture of "what was this user trying to do?"

Threading. A thread is one user task. The heuristic scorer attaches a new prompt to an existing thread when its follow-up score crosses 0.45. The score is composed from six signals — try the sliders below.

The scorer surfaces a human-readable followup_reason ("same session, recent activity, follow-up phrase") so curators can see why a prompt was attached — and override if needed.

Clustering. The thread is the cohesive unit, but a single thread can contain multiple clusters when follow-ups are typed differently. Conversely, a single cluster aggregates evidence from many threads across many sessions.

Scoring. opportunity_score = evidence_count × confidence × gap_severity. Confidence is composed from match-score gap (how much worse than a real hit), evidence count, and opportunity type. This gives curators a single sortable column instead of a wall of clusters.

Real validation in flight. The strongest case: "can you do the second one?" after a numbered list scored 0.85 and joined the right thread. The known weakness: "now" combined with same-session and recent-activity hits 0.7 even on topic switches. Both are surfaced honestly in the admin view rather than papered over.

Worked example — gap to fix

The clearest signal of the system doing its job: a cluster surfaces a gap, the curator queue routes it, and the response is a product change rather than another skill.

1. Cluster. A coherent cluster of failed prompts emerged: colleagues asking questions that required data from a third-party user-research provider, none of which the existing MCPs could answer well.
2. Root cause. The provider hadn't built a programmatic surface for the data; analyst-driven manual queries took several days per request and were a recurrent bottleneck.
3. Response. The cluster became evidence used to scope a custom MCP with the provider's product team. The MCP exposes the full data repository — user-testing sessions, transcripts, structured metadata — through a small set of scoped operations.
4. Outcome. Query time dropped from several days to minutes. Subsequent prompts in the same cluster are now answered automatically and surfaced for periodic quality review.

This is the loop running end-to-end: cluster → diagnosis → upstream change → cluster closes. The platform's value isn't the catalog. It's the routing surface that turns repeated failure into evidence and evidence into directed work.

Things that started as open questions and got pinned down during the three days. Each is dated, each names a role.

Honest about what's not yet decided. Each has a trigger condition for resolution.

Honest about the limits — these are future bets, not blockers.

This was a vibe-coded project: three days, three people, dozens of Vercel preview URLs, Co-Authored-By footers all over the git log. Hand-written code under 100 lines. Every commit on main was AI-authored; every line was reviewed before merge.

Team & roles

• LouisResearcher + builder — idea, system architecture, MCP design and configuration, project hooks across Cursor and Claude Code, PM and conductor for the parallel agent runs.
• Two collaboratorsBackend + tests — Supabase wiring (schema, RLS, embedding pipeline, vector index) and the test surface (integration tests across both MCPs, end-to-end smoke checks against the preview URLs, regression on the threading scorer).

The AI recipe

• Claude Code (Opus 4.7, 1M context) — primary driver: backend logic, schema, hook adapters, agent orchestration, integration tests, and this README.
• Conductor + four parallel agents (Cursor Composer + Claude Code) — each rebuilt one surface (catalog, skill detail, ask, trainer admin) against a shared design-system bootstrap Claude Code authored first. Each agent got its own Vercel preview URL and self-verified before opening a PR.
• Cursor Composer — narrower targeted refactors (filter UX, trainer-polish pass, network-graph view).
• Anthropic API (Claude Sonnet 4.6) — /ask RAG chat in the live web app.
• OpenAI — text-embedding-3-small for catalog search; vectors stored in Supabase pgvector.

A shared UX_AGENT_BRIEF.md and METRIC_GLOSSARY.md acted as the spec across human and AI contributors — every agent read them before writing UI.

This was as much an experiment in how a small team uses AI to ship a real internal product end-to-end as it was a product. The product is the methodology rendered legible.