We terminated a TPU mid-training and it recovered in seconds: Introduction to elastic training with MaxText
Same process, same PID, under two minutes of downtime, and almost all of it was waiting for Kubernetes to schedule the replacement worker. It still has every entry from before the failure. That object lives in CPU memory on the controller, not on any TPU, so an elastic event can't touch it.
- ▪Same process, same PID, under two minutes of downtime, and almost all of it was waiting for Kubernetes to schedule the replacement worker.
- ▪It still has every entry from before the failure.
- ▪That object lives in CPU memory on the controller, not on any TPU, so an elastic event can't touch it.
Google Developers Blog files mainly under programming. We currently carry 20 of its stories.
Opening excerpt (first ~120 words) tap to expand
Zero restarts. Same process, same PID, under two minutes of downtime, and almost all of it was waiting for Kubernetes to schedule the replacement worker. For comparison, a full job restart on this same cluster pays for everything we just skipped: tearing down and rescheduling the entire workload — controller and all workers, not just the failed slice — sometimes a full termination grace period, then starting fresh containers and Python processes before it even gets to the checkpoint load.
…
Excerpt limited to ~120 words for fair-use compliance. The full article is at Google Developers Blog.
Discussion
0 commentsMore from Google Developers Blog
