A Control Is Not Effective Merely Because It Is Configured
A control is not effective merely because it's configured. Learn the ten-step test for whether an AI approval control actually operated as designed.
Who this is for
Internal auditors and controls testers — Internal auditors and controls testers evaluating whether an approval control governing AI-assisted outputs actually operated as designed over a period — not whether the policy exists on paper
The problem
A policy that says 'high-risk AI outputs require manager approval' is a control design. Whether that control actually operated — whether approvals happened, by the right person, with evidence of genuine review, consistently across the population of qualifying outputs — is a separate question that policy language cannot answer.
Most teams check that the control is configured: the workflow routes flagged outputs to a reviewer, the system logs an approval. Few teams test whether that configured control produced the evidence a genuine control test requires.
How ConvergePanel helps
ConvergePanel's audit log gives testers a population to sample from — every flagged output, its assigned reviewer, the review timestamp, and the decision. That data supports a proper operating-effectiveness test; it does not substitute for one. The test itself — sample selection, evidence inspection, and the effectiveness conclusion — remains the tester's professional work.
How it works
- 1Define the control objective precisely — what is this approval control actually supposed to catch
- 2Identify the expected approver for the control's population of outputs
- 3Identify the full population of outputs that should have gone through the control over the test period
- 4Select a sample from that population — risk-based, not just the most recent items
- 5Inspect the evidence for each sampled item: was it actually reviewed, by whom, with what timing
- 6Test timing — did approval happen before the output was used, not after
- 7Inspect exceptions — items that bypassed the normal approval path, and whether that was documented
- 8Evaluate consistency — were the same approval criteria applied across the sample
- 9Document findings for each sampled item
- 10Conclude on operating effectiveness for the period tested
Use cases
- Testing whether a 'high-risk AI output requires sign-off' control operated consistently over a quarter
- Sampling flagged outputs to check whether the assigned reviewer, not a substitute, actually approved them
- Identifying exceptions that bypassed the approval path without documented justification
- Distinguishing a control that is well-configured from one that is operating effectively
- Building a repeatable test approach for approval controls across multiple AI-assisted workflows
Worked Example: Testing a Quarter of Flagged Approvals
A controls tester is asked to evaluate whether a 'manager sign-off required for regulatory-sensitive AI outputs' control operated effectively last quarter. The population is 140 flagged outputs. The tester selects a risk-based sample of 25 — weighted toward the highest-materiality items rather than a simple random draw — and inspects each: who approved it, how long after flagging, whether reviewer comments exist, and whether any of the 25 bypassed the normal path.
Of the 25, 3 were approved by someone other than the named manager, 2 show approval times under 30 seconds with no comments, and 1 bypassed the flag entirely with no documented exception. The control is configured correctly — every item did route through the system — but the test finds it is not operating effectively for a meaningful share of the sample. That distinction is the entire point of testing operation, not just design.
Design vs. Operation vs. Evidence
- Control design — does the policy, if followed exactly, address the risk it's meant to catch
- Control implementation — was the workflow actually built to route outputs the way the policy describes
- Control operation — did it actually run that way, consistently, across the real population
- Control evidence — can each of the above be demonstrated with a record, not asserted from memory
How This Differs from Using AI to Help Test Internal Controls Generally
ConvergePanel's internal-controls research assistant helps auditors compare AI models' characterizations of control design expectations for any control — financial, operational, IT. This page is a different exercise: testing the operating effectiveness of the control that specifically governs AI-assisted outputs. The approval gate itself is the subject under test here, not a research method for testing something else.
Common Testing Mistakes
- Sampling only the most recent approvals, which tend to look better than a representative sample
- Treating 'the workflow routed it correctly' as proof the control operated, without inspecting the actual review evidence
- Not checking timing — approvals recorded after the output was already acted on are not effective preventive controls
- Skipping exception review — items that bypassed the control are often where the real gaps are
- Concluding on effectiveness without a defined, risk-based sample size and selection method
Frequently asked questions
What's the difference between testing this control and design testing?
Design testing asks whether the control, if followed exactly, would address the risk. Operating-effectiveness testing asks whether it actually was followed, consistently, across a real population over a period — which requires inspecting evidence, not reading policy.
How large should the sample be?
Sample size and selection methodology are professional testing judgments that depend on population size, risk, and your audit methodology's standards. ConvergePanel provides the population data to sample from; sample size determination is not something it prescribes.
Can ConvergePanel conclude on the control's effectiveness?
No. ConvergePanel provides workflow evidence — timestamps, reviewer identity, decisions, and exceptions. Designing the test, selecting the sample, evaluating the evidence, and reaching the operating-effectiveness conclusion require qualified assurance professional judgment.
What counts as an exception that needs investigating?
Any output that should have gone through the approval control based on the policy's stated scope, but didn't — or went through a different path than defined. Undocumented exceptions are frequently where control failures concentrate, so they warrant disproportionate testing attention relative to their share of the population.
How is this different from testing AI controls more broadly?
This page is scoped specifically to approval controls — the gate requiring human sign-off before an AI-assisted output is used. Broader AI control types, such as data-quality or model-monitoring controls, involve different evidence and different testing approaches outside this page's scope.
How often should this control be tested?
Frequency should match the risk of the decisions the control governs and your organization's audit cycle — commonly quarterly or annually for consequential controls. Testing only at rollout and never again is a common way approval drift goes undetected.
Explore related pages
ConvergePanel provides AI-assisted verification for informational purposes only. Not forensic analysis. Not legal evidence.
More in Governance
AI Governance Workflow for Enterprise Teams
Enterprise AI governance: automatic policy checks, peer review workflows, and full audit trails. ConvergePanel makes AI verification auditable.
AI Peer Review for High-Stakes Workflows
Use AI peer review to compare models, surface disagreement, document review notes, and create decision receipts for serious work.
AI Trust Dashboard for Decision Support
Use trust signals, model agreement, disagreement, source review, and audit trails to support AI-assisted decisions.