Moderating AI-generated images at scale

Use a score to order a review queue, never to action an account. Detection has an error rate that makes automatic enforcement unfair, and the appeals cost more than the triage saves.

Why automatic enforcement fails here

At 91 percent balanced accuracy under laboratory conditions, roughly one call in eleven is wrong. On user-generated content, which arrives screenshotted, recompressed and cropped, the real rate is worse.

Applied to a hundred thousand items a day, that is thousands of wrong actions. Each one is an appeal, and appeals cost more per case than the review the automation replaced. The maths does not work even before you consider the reputational cost of removing genuine posts.

There is a second problem specific to moderation. Enforcement decisions get audited. A regulator or a court asking why an account was removed will not accept a model output with an unpublished threshold as a reason.

Score to sort, not to act

The useful framing is queue ordering. You have more items than reviewers. A score decides which ones a person sees first, and that is a job it can do well even at 85 percent accuracy, because being wrong just means somebody looks at something unnecessarily.

1 Above 90 Front of the review queue
2 65 to 90 Reviewed within the normal window
3 45 to 65 Only if reported by a user
4 Below 45 No action, sampled for quality checks

Bands map to queue routes rather than to outcomes. Nothing in this diagram removes content on its own.

Sampling the bottom band matters more than it looks. It is the only way to measure your false negative rate, and without that number you cannot tell whether the system is working or has quietly stopped catching anything.

What your policy needs to say

Most platforms now have a synthetic media rule, and most of them are written badly. The common failure is banning AI content generally, which is unenforceable and catches things nobody meant to catch.

Rules that can be enforced against rules that cannot
Enforceable	Not enforceable	Why
Undisclosed synthetic media of a real person	All AI-generated images	The second bans illustration and design work
Generated images presented as documentary evidence	Anything that scores above 65	A threshold is not a policy
Synthetic media used to deceive for money	Images that look AI-generated	Looking is not a standard
Failure to label when asked	Undisclosed AI in any form	Unprovable, and users cannot comply

Write the rule around harm and disclosure rather than around technique. A generated illustration on a fiction post harms nobody. A generated photograph of a product, a person or an event, presented as real, is the thing you actually want to stop.

The three cases a region map separates

Whole-frame scores collapse three situations that need different handling. The tile view tells them apart, and the distinction usually decides the outcome.

Every tile high. A wholly generated image. If it is presented as a photograph of something real, that is your clearest case.
One tile high. A real photograph with something added or removed. In a marketplace or news context this is often more serious than a full generation, because it is designed to be believed.
Evenly warm, none high. Heavy processing. Usually a genuine photo through a filter or an upscaler. Almost never worth actioning.

13 17 11 15 91 14 12 16 10

A real photograph with an inserted element. Queue ordering on the headline score alone would never surface it.

The middle case. A whole-frame score of 33 would have cleared this item.

Measuring whether it works

Track precision at the top band
Of the items scored above 90 that a moderator opened, how many were actioned? If that number is low, the tool is generating work rather than saving it.
Sample the bottom band weekly
Pull a random hundred from below 45 and review them. This is the only measurement of what you are missing, and it is the one teams skip.
Watch the appeal rate
A rise in successful appeals on synthetic media decisions means the threshold has drifted or a new generator has arrived that scores low.
Re-baseline after every model change
Detector updates move scores. A threshold tuned six months ago against a different model version is no longer the threshold you think it is.

Staffing the queue the tool creates

Adding a detector to a moderation pipeline does not reduce headcount, and teams that plan for it to do so end up worse off. What it changes is the order work arrives in, which raises the value of each reviewer hour without reducing how many you need.

Budget for the flagged queue to be slower per item than the general queue. A reviewer looking at a flagged post is making a harder call on more ambiguous material, and rushing that is how wrong enforcement happens. Two minutes per item is realistic; thirty seconds is not.

Give reviewers the region map rather than the score. A moderator who can see that one corner of an image is hot makes a better decision than one handed a number, and they can explain that decision afterwards to an appeals team or a regulator.

Rotate people off synthetic media review. It is repetitive work with a high proportion of ambiguous cases, and accuracy drops measurably across a long shift. This is the same lesson every moderation function has already learned about other categories.

Questions people ask

Can we auto-remove content above a score threshold?

You can, and you should not. At realistic accuracy that means thousands of wrong removals at scale, each one an appeal and some of them a regulatory problem. Use the score to order a human queue. The reviewer time saved by good ordering is larger than the time automation would save you.

What threshold should we set for review?

Start by routing everything above the published decision line and sampling below it, then tune on your own precision numbers rather than on a vendor recommendation. Your content mix decides the right threshold, and a platform full of illustration needs a different line from one full of news photos.

How do we handle appeals?

Ask for the original file. Most successful appeals come from users whose genuine photo was recompressed on upload, and re-checking the source usually resolves it in one step. Record the outcome, because appeal data is the fastest signal that a threshold has drifted.

Should users be told an image was flagged by a detector?

Telling them a check happened is fair and increasingly expected. Publishing the exact threshold is not, because it hands an evasion target to anybody willing to test against it. State that automated signals inform review and that a person makes every decision.

Does this work for video?

Not with a still image detector. Frame-by-frame checking of video produces enormous volume and misses temporal artefacts, which are the strongest signal in manipulated video. Treat video as a separate problem needing separate tooling.

What about images our own users generate legitimately?

Label rather than remove. A disclosure field at upload, backed by a check that flags undisclosed cases for review, gets you a workable rule. Users who follow the labelling requirement should never see enforcement, which is what makes the policy defensible.