Vibe Analytics

Rob Stilson leads a build-along workshop that reimagines the analytics lifecycle — define, build, validate, operationalize, monitor — for an era of capable AI agents. The session opens with an honest account of why coding agents succeeded first and analytics did not: software has tests and documentation as natural guardrails, while an analytics question often has a single correct answer from a single correct source and no deterministic way to prove correctness. Participants then watch a naive agent answer a routine workforce question confidently and incorrectly against a purpose-built synthetic warehouse, seeded with the three failure modes that account for most inaccurate agent responses: concept-to-entity ambiguity, data staleness, and retrieval failure. The core lab inverts the usual order. Participants write offline evaluations for a recurring question from their own work before building anything, pinning ground truth so results cannot drift, then author a skill — a reference document plus a routing layer — encoding the domain judgment an experienced analyst holds and a model cannot infer. A controlled ablation measures the difference. Because a skill is written in markdown rather than code, a SQL analyst and a senior data scientist can both produce credible work in the same block. The workshop advances I-O practice by insisting that agent-assisted analysis be evaluated, not merely felt to be better, and it contributes a methodological guardrail for causal overclaiming and small-cell reporting that general engineering practice does not supply. Contributions are published to an open-source repository after the session.

 

Workshop Facilitators

Rob Stilson, Lockheed Martin

---

 

For the Vibe Analytics workshop, attendees should have the following with them:

• A laptop with admin rights (they’ll be installing Python packages)
• Python 3.9 or newer
• VS Code with the Microsoft Python extension
• The workshop materials from github.com/RobStilson/analytics-skills (git clone or ZIP download),
• with the dependencies in requirements.txt installed
• An Anthropic API key (console.anthropic.com) with a small amount of credit loaded (about $5)