Critic Gates for Mass Content

Critic Gates for Mass Content is an architectural pattern I use to programmatically enforce quality and relevance on large volumes of AI-generated text, preventing low-value output from reaching production.

What it is

This approach involves deploying a dedicated AI agent, or "critic gate," whose sole function is to evaluate content generated by other agents. Instead of directly publishing content, the primary generation agent submits its output to the critic. The critic then scores the content against a rubric covering factors like specificity, factual accuracy (where applicable), differentiation from existing content, and overall helpfulness to the target audience. This is distinct from simple validation; it's an active, subjective quality assessment. For instance, when I generate product descriptions for Inky, the critic ensures each description highlights a unique selling proposition and avoids generic marketing language.

Why I do it this way

I learned the hard way that unconstrained content generation, even with sophisticated prompts, quickly leads to a flood of mediocre output. Google's quality guidelines are clear: generic content offers no value. My goal is to build durable assets, not just fill pages. Without a critic gate, I'd spend significant human time manually reviewing thousands of pieces of content, which defeats the purpose of Agent-First Development. This system allows me to maintain a high bar for quality without becoming a bottleneck myself. It's a pragmatic response to the reality of operating a multi-product studio where content volume can quickly overwhelm human review capacity. This also aligns with my approach to Human-in-the-Loop Where It Matters, ensuring my intervention is focused on refining the critic's rubric, not sifting through raw output.

How it works in practice

The process begins with a primary agent generating content based on a specific brief. For Total Formula 1, this might be a race recap or driver profile. Once generated, this content is passed to a dedicated critic agent, typically running on a different LLM or a finely tuned instance of Claude Code as the Engineer. The critic operates with a detailed JSON schema for evaluation, scoring the content on a scale of 1-10 for several dimensions. For example, a critic for Pregnancy Power Hour content might score for medical accuracy (cross-referencing against a trusted knowledge base), empathetic tone, and clarity. If the aggregate score falls below a threshold, the content is rejected, and the primary agent receives feedback to iterate. If it passes, the content is then staged for publishing, often to a Vercel-hosted frontend or a Firebase database. This entire flow is orchestrated within my VERA agent system, ensuring deterministic outcomes and clear audit trails for rejected content.

Where this breaks down

The primary challenge with critic gates is the initial calibration of the critic's rubric and the LLM's interpretation of it. If the rubric is too vague, the critic becomes ineffective. If it's too rigid, it can stifle creativity or reject perfectly good content for minor infractions. I've found that the critic itself needs iterative refinement, often requiring several rounds of human-in-the-loop review to fine-tune its scoring logic. Another point of failure is when the primary generation agent learns to "game" the critic, producing content that technically meets the rubric but still lacks genuine depth. This requires constant vigilance and updates to the critic's prompt and evaluation criteria. Finally, the computational cost of running a second, often complex, LLM evaluation for every piece of content adds to the operational expense. This is a tradeoff I accept for quality, but it's a real consideration.