https://auto-driving-blog.vercel.app/tags/-zero/ Recent content in 💾 ZeRO on Elon's AD Insight https://auto-driving-blog.vercel.app/images/share.png https://auto-driving-blog.vercel.app/images/share.png Hugo zh-cn Sun, 19 Jul 2026 00:00:00 +0000 https://auto-driving-blog.vercel.app/posts/knowledge/%E5%88%86%E5%B8%83%E5%BC%8F%E8%AE%AD%E7%BB%83%E7%9A%84%E5%B7%A5%E7%A8%8B%E5%AE%9E%E8%B7%B5/ Sun, 19 Jul 2026 00:00:00 +0000 https://auto-driving-blog.vercel.app/posts/knowledge/%E5%88%86%E5%B8%83%E5%BC%8F%E8%AE%AD%E7%BB%83%E7%9A%84%E5%B7%A5%E7%A8%8B%E5%AE%9E%E8%B7%B5/ 大模型训练需要分布式系统支持，但3D并行（数据并行/张量并行/流水线并行）的配置与调优极为复杂。本文从单卡训练痛点出发，系统讲解DeepSpeed的ZeRO stages、Megatron-LM的张量切片策略、流水线并行的micro-batch调度，以及VLA模型训练中activation checkpointing与通信拓扑选择的工程经验。