Data Annotation

Instruction Dataset Creation

Guidelines for building instruction datasets:

• Cover diverse task types (QA, summarization, code, reasoning, creative writing).
• Vary instruction complexity (simple → multi-step → chain-of-thought).
• Include negative examples and edge cases.
• Avoid repetitive templates — aim for linguistic diversity.
• Target minimum 1,000 high-quality examples for domain-specific tuning.

Synthetic Data Generation

When manual annotation volume is insufficient:

• Use a strong reference model to generate candidate responses.
• Apply LLM-as-judge scoring for quality filtering.
• Cross-validate synthetic outputs against human-written gold samples.
• Report synthetic-to-human ratio and quality metrics.
• Flag hallucination-prone categories for human review.

Preference Data (DPO/RLHF)

For alignment datasets:

• Present same prompt with two responses (chosen vs rejected).
• Ensure clear quality separation between chosen and rejected.
• Cover safety, helpfulness, and factuality dimensions.
• Minimum 500 preference pairs for meaningful alignment signal.
• Track annotator agreement rate when multiple reviewers are involved.

Quality Assurance

Run validation checks on completed datasets:

• Schema validation (required fields, types, lengths).
• Duplicate detection (exact and near-duplicate prompts/responses).
• Length distribution analysis (flag outliers).
• Toxicity/safety scan on responses.
• Coverage analysis across intended task categories.

Deliverables

1. Formatted dataset files (JSONL/Parquet)
2. annotation_guidelines.md (format spec, quality criteria)
3. dataset_quality_report.md (stats, coverage, quality metrics)
4. category_distribution.json (task type breakdown)

Python Dependencies

• datasets
• pandas
• langdetect
• regex