Training on Model-Labeled Data: Distillation or Just Synthetic Data?

Key Info

A developer proposes a workflow: take 10,000 scenarios, have a model called “Fable” label them into 4 categories, then use that labeled data to train a model — not even an LLM — and asks whether this process counts as distillation.

Highlights

• The workflow relies on another model to generate labels for downstream training, which sits at the intersection of synthetic data and knowledge distillation.
• Training a smaller or non-LLM model on labels produced by a more capable model is a common technique to compress or transfer capability.
• The question highlights how blurry the line can be between “distillation” and simply using model-generated training data.