Anomaly detection for a 9,000-GPU supercomputing fleet
- Situation
- A leading US national laboratory operated a supercomputing fleet of roughly 9,000 GPUs. Faults on individual devices were expensive in wasted allocation and hard to find early, and the volume of telemetry made manual inspection impossible.
- Work
- I designed and built a production anomaly-detection engine for the fleet, covering the whole path from method selection through implementation to a deployable service. The engine was written for the compute budget it had to run in, and it came with a validation harness so that its behaviour could be reproduced and challenged rather than taken on trust.
- Outcome
- A working detection service running against live fleet telemetry, documented to the point where other engineers could take it on, with the limitations of the method written down alongside its results.
Work performed while employed at Dell Technologies.