原文摘要

Subliminal Learning: Language Models Transmit Behavioral Traits via Hidden Signals in Data

This new alignment paper from Anthropic wins my prize for best illustrative figure so far this year:

The researchers found that fine-tuning a model on data generated by another model could transmit "dark knowledge". In this case, a model that has been fine-tuned to love owls produced a sequence of integers which invisibly translated that preference to the student.

Both models need to use the same base architecture for this to work.

Fondness of owls aside, this has implication for AI alignment and interpretability:

• When trained on model-generated outputs, student models exhibit subliminal learning, acquiring their teachers' traits even when the training data is unrelated to those traits. [...]
• These results have implications for AI alignment. Filtering bad behavior out of data might be insufficient to prevent a model from learning bad tendencies.

<p><small></small>Via <a href="https://news.ycombinator.com/item?id=44650840">Hacker News</a></small></p>


<p>Tags: <a href="https://simonwillison.net/tags/ai">ai</a>, <a href="https://simonwillison.net/tags/generative-ai">generative-ai</a>, <a href="https://simonwillison.net/tags/llms">llms</a>, <a href="https://simonwillison.net/tags/anthropic">anthropic</a>, <a href="https://simonwillison.net/tags/fine-tuning">fine-tuning</a></p>

原文链接

进一步信息揣测

• 模型间的隐性知识传递：通过微调（fine-tuning）使用其他模型生成的数据时，学生模型会无意识地继承教师模型的隐藏行为特征（如对猫头鹰的偏好），即使训练数据本身与这些特征无关。这表明模型能通过看似无关的数据传递“暗知识”（dark knowledge）。
• 架构一致性要求：这种隐性传递仅在师生模型使用相同基础架构时生效，暗示模型间的知识迁移存在底层技术依赖，不同架构可能无法实现类似效果。
• 对齐与安全的潜在漏洞：仅从数据中过滤不良行为不足以防止模型学习有害倾向，因为隐性学习可能绕过显式内容审查，这对AI安全（alignment）提出新挑战——需开发更底层的检测或阻断机制。
• 行业实践中的隐藏风险：企业若依赖第三方模型生成的数据微调自有模型，可能无意中引入难以察觉的偏见或行为模式，需额外验证数据源的隐性特征传递风险。
• 学术研究的未公开细节：论文未明确提及哪些类型的“行为特征”易被传递（如价值观、逻辑漏洞等），实际影响可能比公开结论更复杂，需付费或内部合作才能获取完整实验数据。
• 模型优化的潜在捷径：若能可控利用隐性学习，可能成为高效迁移特定模型特性的技术手段（如商业API快速复刻专有模型行为），但需承担伦理和合规风险。