Where exactly is my fabric problem?
A job slows down or fails. Is the network to blame? And if so, where exactly? Clockwork FleetLens definitively rules whether the fabric is at fault and, when it is, pinpoints the exact link, switch or NIC, so you stop guessing and stop burning GPU-hours on blind spots.
Today’s AI observability tools leave you running blind.
The fabric is complex, and the hardest part of the stack to diagnose
Today’s large AI factories have hundreds of switches and tens of thousands of links and transceivers, any one of which can quietly degrade. Troubleshooting at this scale is like finding a needle in a haystack.
DGX GB300 SuperPOD
Clockwork has built the foundational blocks needed to uncover fabric blind spots.
FleetLens is built on four foundational technologies that see every link in the fabric, every 10 milliseconds, and pinpoint exactly where it is degrading. This is supplemented by telemetry from the host, fabric and workload.
Probemesh: thousands of tiny probes measure every path in the fabric, every 10 milliseconds.
Probemesh sends lightweight out-of-band probes between NIC pairs across the fabric every 10 milliseconds. Because every clock is synchronized to sub-µs accuracy, the probes can measure the exact one-way delays in each direction. The result is a live, high-resolution directional map of the whole fabric.
FleetLens detects the problem, and definitively rules whether the fabric is at fault.
FleetLens detects slow or failing jobs and definitively rules whether the fabric is to blame. When it is, FleetLens pinpoints the exact link, switch or NIC at fault. It surfaces the hidden, workload-impacting issues other tools miss: congestion and micro-congestion, topology-specific degradation, gray failures and unexplained slowdowns.
The issues FleetLens finds but other tools miss
Congestion and micro-congestion
Oversubscribed paths, queue buildup, head-of-line blocking
Topology-aware degradation
Path asymmetry and cabling inconsistencies
Gray failures
Silently degraded switches and fabric components
Workload-visible slowdowns
Partial fabric, path or NIC problems with no obvious root cause
Misconfigurations
GDR, PCIe topology, firmware and driver mismatches
Detections with evidence and visualizations
Every detection comes with contextual drilldown to supporting evidence shown in Grafana dashboards.
All available as Prometheus metrics
Telemetry is available for scraping through a Prometheus endpoint.
FleetAudit: a known-good baseline before you commit GPU time.
Teams lose hours to faults that only surface once training or inference is already running: a degraded PCIe riser, a firmware mismatch, a misconfigured NCCL setting. Every one of them could have been caught before the job started. FleetAudit is the pre-flight check for your cluster.
Integrates Seamlessly With Your Observability Stack
Expose fleet telemetry through open APIs and familiar dashboards.
Clockwork streams cross-stack metrics via a Prometheus-compatible API, integrating directly with Grafana and existing observability tools. Operators can unify fabric, workload, and system telemetry in a single pane of glass—eliminating silos, accelerating troubleshooting, and making AI infrastructure easier to operate at scale.
Single Pane of Glass Grafana Dashboards
See it in four minutes
“Is it the network, compute, or our code?” Stop guessing.
See FleetLens on your own fabric.
