ai-code-detector

Methodology & limits

How accurate is an AI code detector?

Useful enough to help you choose what to review. Not reliable enough to decide a case by itself. That is the standard this product is built around.

Useful for screening, not proof

An AI code detector can help you sort a large set of files and find the ones that deserve a closer read. It should not be treated as proof of misconduct. A high score means "review this first." It does not mean "this person cheated."

Why code is hard to classify

Short programs, starter templates, common libraries, and strict style rules can make human code look generic. Rewrites and mixed authorship can make AI-assisted code look more human. The same submission can look different depending on language, assignment, and prompt design.

A 2024 code-specific study found that existing AIGC detectors performed poorly when separating human-written and AI-generated code.
arXiv:2401.03676 · ICSE-SEET 2024 · research context, not a competitor attack

What our report shows

The report is designed to make the score inspectable. Instead of one number and a shrug, it should show the file-level result, the snippets that contributed most, and enough context for a human to decide what to ask next.

  1. Input handling: file type, language, and assignment context. 待墨枢确认
  2. Signals: code patterns associated with AI-generated output. 待墨枢确认
  3. Output: per-file score, snippet highlights, exportable report.
  4. Human step: review, compare, ask, document. 待墨盾裁决

Our current evaluation status

We are not publishing an accuracy number yet. Before any benchmark claim appears on this page, it needs a named test set, a date, a method summary, and a clear failure mode section.

Future benchmark

This card will hold a named test set, evaluation date, method summary, and known failure modes — published only when the underlying evaluation meets that bar. No estimate is shown before then.

The human review workflow

  1. Read the flagged snippets before you read the score.
  2. Check whether starter code or required libraries explain the pattern.
  3. Compare with earlier work from the same author when available.
  4. Ask the author to explain, modify, or extend the code.
  5. Record the review outcome separately from the detector output.

Get the report template

What we will not claim

We will not claim a fixed accuracy percentage without evidence. We will not say a report proves cheating. We will not tell students how to make AI-generated code pass screening. Those claims make the category less trustworthy and make real review harder.

Questions

Why not publish one accuracy number?

Because one number hides the part that matters: which kinds of code fail, on which languages, and under which edits. Until we can publish that honestly, we will not publish a number.

What is a false positive here?

A false positive is human-written code that gets flagged. Short assignments, starter templates, and strict formatting can all raise that risk.

What is a false negative?

A false negative is AI-assisted code that does not get flagged. Rewrites, mixed authorship, and heavy editing can all reduce the signal.

Should we stop using detectors completely?

Not necessarily. If you treat the output as a screening signal and keep a human review step, it can still save time and focus attention.

Can students check their code before submitting?

Students can use a free check to understand what might get flagged and to prepare an explanation of their work. We do not provide advice for hiding AI use.

Use the detector where it helps. Keep the decision with a human.