Ai Detect

$ARGUMENTS

OUTPUT FORMAT

Structure your report exactly as follows:

AI Detection Analysis Report

Document: [title/description] Word count (approx): [estimate] Domain: [academic field]

Category 1: Lexical Markers (20%)

Score: X/5 [Evidence and reasoning with specific quotes]

Category 2: Statistical Properties (15%)

Score: X/5 [Evidence and reasoning -- assess perplexity and burstiness with specific examples of sentence length variation, structural patterns]

Category 3: Citation Integrity (20%)

Score: X/5

Citation Verification Results (100% coverage required):

For each citation in the paper, report:

[Summary of findings and score justification]

Category 4: Metadiscourse and Stance Markers (10%)

Score: X/5 [Evidence of hedging, boosters, authorial stance with quotes]

Category 5: Structural Characteristics (10%)

Score: X/5 [Analysis of organization, paragraph variation, list usage, formulaic patterns]

Category 6: Stylometric Features (15%)

Score: X/5 [Function word patterns, POS patterns, phrase structures, swap test results]

Category 7: Voice and Authorial Presence (10%)

Score: X/5 [Evidence of position-taking, engagement with objections, tonal consistency]

Category 8: Content Authenticity (optional weight)

Score: X/5 [Novel synthesis, engagement with tensions, specificity of limitations]

Category 9: Smoking Guns (Definitive)

Result: DETECTED / NONE FOUND [Any definitive AI artifacts found]

Weighted Score Calculation

Confidence Level

[High/Medium/Low -- with modifiers applied]

Assessment

[Score range interpretation -- what collaboration pattern this suggests]

Domain Adjustments Applied

[Any field-specific calibrations]

False Positive Risk Factors

[Any factors that may inflate AI signals: ESL, formal genre, template-driven, etc.]

Actionable Recommendations

[Specific passages or patterns the author should revise to reduce AI-like signals, organized by category. Focus on concrete edits, not vague advice.]

DETECTION FRAMEWORK REFERENCE

The complete framework follows. Apply every element of it.

Comprehensive AI-Generated Text Detection Framework v2.0

Part I: Research Foundation

1.1 Academic Research Sources

This framework synthesizes findings from peer-reviewed research and independent benchmarks:

1.2 Commercial Tool Methodologies

1.3 Key Empirical Findings

Detection Accuracy vs. False Positive Rate Tradeoff (RAID 2024)

• Most detectors achieve high accuracy only at high FPR
• At FPR <1%, most commercial detectors become ineffective
• Binoculars method showed best performance at low FPR
• Adversarial attacks reduce accuracy by 15-40% for most tools

Vocabulary Shift Data (Kobak 2025)

• "Delve/delving" increased 28x in biomedical literature post-ChatGPT
• "Underscores" increased 13.8x
• At least 13.5% of 2024 abstracts were processed with LLMs (lower bound)
• Effect exceeded even COVID-19 pandemic's vocabulary impact

Stylometric Discrimination (2024-2025 studies)

• Integrated stylometric features achieve 99%+ discrimination in controlled studies
• Three most effective features: function word unigrams, POS bigrams, phrase patterns
• Human raters struggle with AI detection (false positive rates 5%, vs. 1.3% for tools)
• Humans make judgments based on surface features; stylometry captures deeper patterns

Part II: Evaluation Categories

Category 1: Lexical Markers (Weight: 20%)

Rationale: Kobak et al. (2025) demonstrated that LLMs have distinctive vocabulary preferences that create measurable "excess words" in academic writing.

High-Signal Markers (Frequency Ratio >10x post-ChatGPT)

Flowery Phrase Patterns

These multi-word constructions strongly indicate AI generation:

• "meticulously [examining/analyzing/exploring]..."
• "the intricate [web/tapestry/landscape] of..."
• "comprehensive [overview/analysis] that delves..."
• "pivotal role in [fostering/enhancing]..."
• "navigate the [complex/nuanced] landscape..."
• "a testament to the [power/importance]..."

Scoring Rubric

Category 2: Statistical Properties (Weight: 15%)

Rationale: GPTZero, Turnitin, and academic research consistently identify perplexity and burstiness as core discriminators between AI and human text.

Perplexity (Word Predictability)

Human writing indicators:

• Idiosyncratic vocabulary choices
• Unexpected but fitting word selections
• Domain-specific jargon used naturally
• Personal stylistic preferences evident

Burstiness (Sentence Variation)

Assessment Checklist:

• Sentence lengths vary substantially (fragments to complex sentences)
• Paragraph lengths respond to content needs (not uniform)
• Mix of simple, compound, and complex sentence structures
• Occasional intentional sentence fragments or run-ons
• Rhythm changes with content (dense technical to flowing narrative)

Scoring Rubric

Category 3: Citation Integrity (Weight: 20%)

Rationale: Citation hallucination is one of the most definitive markers of AI generation. Walters & Wilder (2023) found 55% fabrication rate in GPT-3.5 and 18% in GPT-4.

Verification Protocol — 100% COVERAGE REQUIRED

Every citation in the paper must be individually verified. For each citation, check:

1. Author Existence: Do these authors exist and work in this field?
2. Publication Existence: Does this paper actually exist?
3. Journal/Venue: Is the journal real? Does it publish this topic?
4. Date/Volume/Pages: Do the bibliographic details match?
5. DOI Verification: Does the DOI resolve to the claimed paper?
6. Claim-Source Alignment: Does the cited source actually support the specific claim made in the paper?

Red Flags for Hallucinated Citations

• Authors whose names sound plausible but don't exist
• Papers that combine elements from multiple real sources
• Journals that don't exist or don't cover the topic
• Volume/page numbers that don't exist for the claimed year
• DOIs that lead to different papers or don't resolve
• Claims that don't match the actual source content

Scoring Rubric

Category 4: Metadiscourse and Stance Markers (Weight: 10%)

Rationale: Research shows AI text has lower interactional metadiscourse, fewer hedges, and more impersonal tone compared to human academic writing.

Hedging Patterns

Human indicators:

• Appropriate epistemic caution: "might," "could," "may suggest"
• Uncertainty acknowledgment: "remains unclear," "further investigation needed"
• Qualification of claims: "in some cases," "under certain conditions"
• Personal epistemic markers: "we believe," "it seems to us"

AI indicators:

• Over-confident assertions
• Lack of appropriate hedging in speculative claims
• Generic uncertainty: "more research is needed" without specificity

Boosters and Attitude Markers

Human indicators:

• Strategic emphasis: "clearly," "importantly," "notably" used judiciously
• Personal attitude: "surprisingly," "unfortunately," "remarkably"
• Authorial stance: "we argue," "we contend," "our position is"

AI indicators:

• Flat emotional expression
• Absence of authorial stance
• Generic language without personal investment

Scoring Rubric

Category 5: Structural Characteristics (Weight: 10%)

Rationale: AI text tends toward formulaic organization, uniform paragraph lengths, and predictable structures.

Human Structure Indicators

• Section organization: Responds to content needs, not template
• Non-formulaic ordering: May place literature review after intro (field conventions) or integrate throughout
• Variable paragraph lengths: 1 sentence to 10+ sentences as appropriate
• Prose-heavy argumentation: Ideas developed in flowing prose, not lists
• Idiosyncratic organization: Personal approach to presenting material

AI Structure Indicators

• Formulaic organization: Rigid intro-lit review-methods-results-discussion
• Uniform paragraph lengths: Consistently 4-6 sentences
• Heavy list usage: Bullet points and numbered lists as primary format
• Template adherence: "Tell them what you'll tell them" structure
• Predictable transitions: "Firstly... Secondly... In conclusion..."

Scoring Rubric

Category 6: Stylometric Features (Weight: 15%)

Rationale: Integrated stylometric analysis achieves near-perfect discrimination between human and AI text (Zaitsu & Jin, 2024; Opara, 2024). Three feature categories are most effective: function word unigrams, POS bigrams, and phrase patterns.

6.1 Function Word Patterns

Assessment Method: Sample 5-10 function word choices throughout the document. Do the same choices recur? Does the author seem to have preferences, or are choices interchangeable?

6.2 POS (Part-of-Speech) Patterns

Assessment Method: Examine 10 random sentences. Do they show varied syntactic structures, or could they be generated from the same template?

6.3 Phrase Structure Patterns

Assessment Method: Examine the first word of 10 consecutive sentences. High variety suggests human authorship; repetitive patterns suggest AI.

6.4 Quantitative Benchmarks

6.5 The "Swap Test"

Could this sentence have been written by any competent author, or does it bear marks of a specific person?

• If sentences feel interchangeable with generic academic prose -> AI indicator
• If sentences feel like they could only have been written by this particular author -> Human indicator

Scoring Rubric

Category 7: Voice and Authorial Presence (Weight: 10%)

Indicators of Human Voice

• Position-taking: Clear arguments, not just summaries
• Engagement with objections: Anticipates and addresses counterarguments
• Personal metaphors: Original analogies and comparisons
• Tonal consistency: Stable voice throughout, appropriate to genre
• Intellectual curiosity: Questions emerge naturally from argument

Indicators of AI Voice

• Summary without synthesis: Lists positions without arguing for one
• Generic comparisons: "Like a blank canvas" type cliches
• Tonal instability: Shifts between registers
• Assertion without justification: Claims without reasoning
• Lack of curiosity: Presents information without wondering

Scoring Rubric

Category 8: Content Authenticity (optional weight)

Signs of Authentic Scholarship

• Novel theoretical framework; original synthesis
• Engagement with contradictions in literature
• Specific limitations (not generic "more research needed")
• Research directions that emerge organically from argument
• Original examples; intellectual risk-taking

Signs of AI Generation

• Literature as list; summarizes without synthesizing
• Contradiction avoidance; generic limitations
• Disconnected future work; familiar examples; safe consensus

Scoring Rubric

Category 9: Smoking Guns (Definitive)

If any of these appear, the overall assessment is "AI-generated" regardless of weighted score:

• "As a large language model..." or similar self-identification
• "I cannot verify events after my knowledge cutoff..."
• "Regenerate response" or interface artifacts
• Embedded instructions or prompt leakage
• "I don't have access to real-time information..."
• Responses to hypothetical user queries embedded in text
• Obvious placeholder text ("[Insert X here]")

Part III: Interpretation Scale

Part IV: Confidence Modifiers

Increase confidence if: Multiple categories converge; citation verification yields concrete evidence; smoking guns detected; long document shows consistent patterns.

Decrease confidence if: Document is short (<500 words); technical/formal writing; non-native English speaker; mixed signals across categories.

Part V: Application Protocol

Step 1: Initial Scan

1. Check for smoking guns (Category 9)
2. Run lexical marker check for high-signal words
3. Assess overall structure and burstiness

If smoking guns detected: Stop -- document is AI-generated. If multiple high-signal markers: Continue with detailed evaluation. If clean initial scan: Continue with full evaluation anyway (thoroughness required).

Step 2: Detailed Evaluation

For each category:

1. Review indicators and scoring rubric
2. Document specific evidence from text
3. Assign score with brief justification

Step 3: Citation Verification (MANDATORY — 100% COVERAGE)

For academic documents, verify every single citation:

• Confirm the publication exists and authors are real
• Confirm bibliographic details are accurate
• Confirm the source supports the specific claim made
• Report each citation individually in the verification table

Step 4: Synthesis and Assessment

1. Calculate weighted score
2. Note convergent/divergent categories
3. Consider confidence modifiers
4. Apply domain-specific adjustments
5. Write summary assessment

Step 5: Actionable Recommendations

Provide specific, concrete revision suggestions organized by category, so the author knows exactly what to change to strengthen the paper against AI-detection signals.

Part VI: Domain-Specific Adjustments

Part VII: False Positive Risk Factors

• Non-native English speakers
• Highly formal writing (legal, regulatory, policy)
• Template-driven genres (grant proposals, IRB applications)
• Technical documentation with standardized vocabulary
• Professionally edited text