Human-in-the-Loop Where It Matters

Human-in-the-Loop Where It Matters is my operational framework for delegating product development and studio operations to AI agents, with my direct intervention reserved for specific, high-impact decision points.

What it is

This approach defines explicit boundaries for human oversight in an agent-driven studio. Instead of reviewing every commit or every generated piece of content, I architect the system to flag only the decisions that carry significant risk or directly impact the customer experience. This means agents handle the bulk of the execution, from code generation and infrastructure updates to internal documentation and data analysis. My role shifts from constant supervision to targeted validation, ensuring strategic alignment and quality at critical junctures. It’s about trust in the system I've built, coupled with a clear understanding of where machine judgment needs human validation.

Why I do it this way

I learned the hard way that constant micro-management of AI agents defeats the purpose of automation. Early on, I found myself reviewing every line of agent-generated code or every draft of copy, burning through time and attention. The goal of running a multi-product studio with AI as the team is leverage. That leverage disappears if I'm the bottleneck for every minor decision. This approach allows me to maintain high velocity across products like Total Formula 1 and Pregnancy Power Hour, while still mitigating the risks associated with fully autonomous AI. It's a pragmatic balance between speed and control, designed for a solo operator. This is especially critical when managing multiple distinct products within a Monorepo as Operating System, where shared infrastructure changes could have cascading effects. It frees my cognitive load to focus on product strategy, new market opportunities, and the architectural work that agents can't yet do reliably. I'm building systems, not just writing code.

How it works in practice

My agent orchestration layer, VERA, is configured with explicit approval gates. For code, any internal refactor, linting fix, or documentation update proposed by Claude Code is automatically committed and pushed. However, any change to a public API endpoint, a database schema migration, or a new feature that impacts user data requires my explicit approval via a Slack notification or a custom dashboard. For content, agents using Gemini generate first drafts for blog posts on Inky or marketing copy for Total Formula 1. These drafts are pushed to a staging environment, and a notification is sent to me. I review the staging URL, make final edits directly, and then approve for deployment. Pricing adjustments for any product, or changes to Stripe checkout flows, trigger a mandatory manual review and confirmation. My Queue-Driven Attention system ensures these critical review items land in my inbox, prioritized. Similarly, when agents propose infrastructure changes—like a new Vercel deployment configuration or a Firebase security rule update—they halt and await my explicit `ACK`. This is also where CI Local Before CI Remote plays a role; agents run comprehensive local checks, but the final push to production for critical changes still needs my eyes.

Where this breaks down

This approach relies heavily on a well-defined set of rules and a robust agent architecture. If the rules for what requires human approval are ambiguous, agents will either over-flag (leading to approval fatigue) or under-flag (leading to unnoticed errors). I've had to refine these rules repeatedly. The system also assumes a certain level of agent reliability; if agents frequently generate low-quality output even for "safe" tasks, the cost of human intervention rises. For example, if Claude Code consistently introduces subtle bugs in linting fixes, I can't trust the automated approval. Furthermore, the notification and approval mechanisms need to be low-friction. If approving a critical change takes five steps and three different tools, I'll become a bottleneck. The initial setup and ongoing maintenance of these approval gates is an investment. It's not a set-it-and-forget-it system; it requires continuous calibration, especially as agent capabilities evolve.