synthetic-data-generation

name: "synthetic-data-generation" description: 'Generate synthetic data for fine-tuning, eval-set bootstrapping, RAG corpus augmentation, and rare-case coverage. Covers Self-Instruct, Evol-Instruct, persona-based generation, distillation from larger models, dataset curation (filtering, dedup, decontamination), and provenance / dataset cards.' metadata: author: "AgentX" version: "1.0.0" created: "2026-04-30" updated: "2026-04-30" compatibility: frameworks: ["distilabel", "argilla", "self-instruct", "evol-instruct", "openai", "anthropic"] languages: ["python"]

Synthetic Data Generation

Purpose: Build training and eval datasets quickly without violating user privacy, while controlling for quality, diversity, and contamination.

When to Use This Skill

• Bootstrapping eval sets when you have no real labeled data
• Augmenting fine-tuning data for under-represented intents / slices
• Generating adversarial test cases (combine with ai-safety-and-red-teaming)
• Creating QA pairs over a private RAG corpus
• Privacy-safe substitutes for production data in dev/test

When NOT to Use

• If real, well-labeled data exists at sufficient scale -- use it
• For final benchmark numbers in regulated decisions -- use human-labeled holdouts

Common Techniques

Pipeline

1. Define target distribution and slices (intents, difficulty, demographics)
2. Choose generator model (often a strong reasoning model)
3. Seed with examples or schema
4. Generate at over-sample rate (2-5x target)
5. Quality filter: schema validation, toxicity, dedupe
6. Diversity filter: embedding clustering, slice balancing
7. Decontamination: remove overlap with eval/test sets
8. Human review on a sample (5-10%)
9. Dataset card + version + provenance

Quality Filters

Apply filters in order from cheap to expensive.

Decontamination

Critical before using synthetic data near eval sets.

• N-gram overlap (13-gram is a common threshold) against held-out eval
• Embedding-based near-duplicate detection across train and test
• Source-traceable seeds: never seed from the eval set itself

Document decontamination in the dataset card. A model that has seen its eval is not measuring what you think it is.

Diversity and Bias

• Stratify by intent, language, register (formal/informal), persona, slice
• Measure diversity: distinct n-grams, embedding-cluster spread, persona coverage
• Watch for mode collapse: generator produces narrow patterns; mitigate with temperature, persona conditioning, or rejection sampling
• Audit for amplified bias (compare slice distributions to target)

Legal and Ethical

• Check provider Terms of Service before training models on outputs (some providers restrict use to train competing models)
• Honor source licenses (do not generate near-copies of copyrighted text)
• Do not generate synthetic personally-identifying data of real individuals
• Document the generator model and prompt; downstream consumers must know provenance

Dataset Card (Required Output)

• Purpose and intended use
• Generation method (model, prompt template, seeds)
• Filters applied with retention rates
• Slice distribution (counts per category)
• Decontamination evidence
• Known limitations and biases
• Version, license, contact

Skills to Load Alongside

References

• Self-Instruct (Wang et al.), Evol-Instruct (WizardLM)
• Distilabel and Argilla pipelines
• HuggingFace datasets cards format