User Research Slack Helper

Inside any AI-native company, four patterns emerge fast:

1. Generic chat models can't safely answer internal questions — they don't have the data.
2. Each team has its own private corpus (warehouse, docs, ticketing, code) that nobody routes through.
3. The most valuable internal assistant is one that learns from how colleagues actually use it — what they ask, what they accept, what they ignore — and adapts its tools, skills, and routing accordingly.
4. A centrally configured service account is the right deployment shape: one place to audit, update, and iterate on every MCP, skill, and routing rule. The configuration is what compounds — not the model.

The Slack helper is built around that third pattern. Not a chat interface to internal data. An instrumented surface where every interaction becomes evidence, and that evidence feeds back into how the helper is configured.

The same thesis as the Smart Skills platform: the runtime that consumes a tool can produce evidence about it. Hooks first. Configuration follows.

A Slack app — backed by a Claude Code service account running a frontier model — that any colleague can interact with from a DM or a channel mention.

MCP connections to internal data

Behind the app: connections to the warehouse, internal documentation, ticketing system, and code repositories. Each MCP exposes a small, scoped surface — not a full dump, just the operations that map to actual asked-for tasks. Read-only by default. Deliberate.

A skills library layered on top

Skills are reusable, opinionated workflows: "summarise yesterday's customer feedback," "find the most-cited internal doc on X," "draft a release note from the linked PR." Skills compose MCP operations into the answers people actually want. The model's job is routing, paraphrasing, and conversational glue. The skills do the work.

Subject matter expert clarification

When the helper encounters a question that's outside the scope of its current MCPs — or that genuinely requires human judgement — it can post a clarifying question to a private user research Slack channel. UR colleagues respond asynchronously; the answer is threaded back to the original requestor. This keeps SME load low while closing gaps the tool surface can't yet cover.

A backend that surfaces every trace

Every prompt, tool call, response, and outcome is captured. Colleagues can react to responses (👍/👎/comment); reactions are logged alongside the trace. This is the data that drives the iteration loop — not an afterthought, the point.

A continuous iteration loop

The team behind the helper reviews logs, identifies gaps, and edits tools, MCPs, and skills on a cadence. The configuration is the product. A powerful model with no iteration loop is still just a generic assistant; a modest model with a sharp, tuned toolkit compounds over time.

Safety & governance — built in, not bolted on

The helper's deployment shape was chosen for safety as much as capability. The patterns below are non-negotiable defaults — not features added later.

• Permission-mirrored access. The helper inherits the data-permission structure of whoever invokes it. No broader access granted by virtue of being an AI agent. Differential versioning is in design to serve colleagues with narrower permission credentials.
• Anonymisation before ingest. User data is anonymised before reaching the model context. Personal identifiers never enter the inference path.
• Read-only on internal systems; write-only to outputs. The helper cannot mutate source-of-truth data. The cost of a bad write into the warehouse or ticketing system is much higher than the cost of a missing answer.
• Explicit attack-surface reduction. Contact-database access was deliberately disabled to de-risk prompt-injection-driven exfiltration and to honour GDPR / data-governance commitments. Capability was chosen down; the option to broaden remains a deliberate governance decision, not a default.
• Confidence-gated automation. Verified-metric queries answered automatically with periodic quality review. Non-standard-metric queries surfaced to a human reviewer (SQL inspection / analyst escalation). Prediction-based queries flagged and disclaimed. Use-case scoping: advised for discovery use; high-impact insights flagged for analyst verification.
• Staged capability rollout. Initial deployment to high-access colleagues with active testing; narrower-permission variants and broader rollout gated on the failure-mode surface maturing.

The helper runs as a Claude Code service account on a dedicated machine inside the company perimeter. That means frontier model intelligence — not a capability-constrained local model — with the data never leaving the network boundary. For a financial-services context, perimeter residency is not optional; the service account shape is what makes frontier capability and data residency compatible.

MCPs and skills are configured centrally on that service account. One place to audit, update, and iterate on every routing rule, tool, and skill. When a gap is identified in the trace log, the fix goes in once — not across individual installations. This is the architectural advantage the service account shape gives you: configuration is the product, and there's a single surface to configure.

Multi-turn agentic loops are enabled. The helper can chain tool calls to answer complex questions — warehouse lookup → docs cross-reference → synthesised answer — without requiring the colleague to drive the sequencing manually. The traces from those loops feed directly into the iteration surface: which chains fired, which broke, what the gaps were.

The service account is the load-bearing element: it centralises every MCP, skill, and routing rule in one place, auditable and iterable by the team that owns the helper. The Private UR Channel is the first active SME route — async clarification without broadening access beyond the user research team. The amber arrow closes the loop: traces surface gaps; gaps drive configuration changes.

Every Slack interaction produces a trace: the prompt, which skill (if any) routed it, which MCP operations fired and what they returned, the model's response, and the colleague's reaction (👍, 👎, follow-up, silence).

Logs are reviewed on a cadence. Four patterns drive configuration changes:

• Wrong tool fired — tighten the skill's routing logic, or split into two skills. The trigger condition was too broad.
• Right tool, bad output — adjust the MCP's response shape: less noise, better fields, different aggregation. The tool fired correctly; the output contract was wrong.
• No tool fired, model freestyled — identify the missing skill, draft it, register. This is the most common gap pattern and the strongest signal for what to build next.
• Successful pattern emerging in raw prompts — promote to a named skill so the next colleague gets the same quality for free, without reinventing the prompt. This is how the helper's value compounds.

This is the same evidence-backed curator workflow as the Smart Skills platform, applied to a different surface. The skill catalog there. The Slack helper here. Same shape.

Trace review — three overlapping observability strategies

Reviewing every trace by hand doesn't scale. Reviewing only what's flagged misses the unknown unknowns. The cadence runs three approaches in parallel:

• Automated batch review — every trace is processed for known failure signatures (multiple-prompt resolution paths, low-confidence routings, contested reactions).
• Random sampling — a stratified random slice is reviewed by hand each cycle. This is the surface for the unknown failure modes — the ones automated review can't yet detect because we haven't seen them.
• High-value outlier surfacing — traces flagged as high-impact (touching pricing, financial advice, or escalated incidents) are reviewed individually regardless of other signal.

The dominant failure mode that's emerged: not hallucination, but non-standard metric application. A worked example: volume calculations were surfacing inconsistent active-user numbers because skills were defaulting to differential active-user designations rather than a consistent monthly-active-user (MAU) framing. Trace review surfaced the pattern, corrected the skills, and added a glossary check to the routing layer. Hallucination remains rare; the system is taught to scope research requests against available sources and known interpretations, and that scoping reduces the model's room to invent.

The traces are more than operational data. They're a longitudinal record of how colleagues think about their own work — taken at the moment of actual need, not recalled in a research session. What they reveal:

Most user research projects stop at "what do users want." This one is set up to also answer "what do they do, repeatedly, and what does that reveal about the system they're working inside?"

Resolved

Open