AWS ships cheap GPU fault detection while the audit that works can cost a third of training speed

AWS's new HyperPod dashboard flags sick GPUs for a sliver of CPU; the audit that catches the fault it cannot see costs up to a third of training speed and appears on no invoice. Anyone capitalizing the AI buildout's committed compute is pricing hours nobody can cheaply verify.

Richard TangRichard Tangthree-minute read
Share
Close-up of an Nvidia H100 data center GPU, with the H100 label and NVLink connectors visible above a gold heatsink
An Nvidia H100 GPU. AWS's test of in-loop audits ran a 953-million-parameter model on 32 of these cards, and audits every 10 steps added about a third of step time.

Yesterday AWS−2.75% — AWS, down 2.75 percent today gave SageMaker HyperPod operators a diagnostic summary that names the suspect GPU in a slowed training job: each card is compared against its peers on utilization, memory, temperature, power draw and clock speed, while static rules watch ECC errors, Xid faults, network degradation, throttling and overheating 1. Eight days earlier, AWS's own guide on silent data corruption stated the limit of that panel in one line: "a clean panel doesn't clear the node" 1. The dashboard is cheap; the check that matters is not, and no invoice carries it.

Peer comparison catches what announces itself

The design fits how the visible failures behave. In synchronous training every GPU waits at each step, so one slow card sets the pace for the whole job 1. Peer comparison finds such stragglers, and the rules catch faults that declare themselves. But researchers at New York University and ByteDance, tracing 3,079 pretraining jobs, found most stragglers come from uneven workloads, sequence-length imbalance and Python's garbage collector, not broken hardware 2. AWS publishes no deviation thresholds, minimum group size or price for the comparison, so an operator cannot yet judge how much of its output to trust 1.

The fault that never raises an error

Silent data corruption is different in kind: a GPU returns a wrong but plausible number with no ECC error, because ECC protects memory rather than arithmetic, and no Xid code, because a GPU raises one only when it detects a fault 1. The next synchronization step then spreads the bad value into every replica's weights 1. AWS's own blog put it plainly this week: large training jobs can compute wrong results even when every node looks healthy 3. Meta's−4.82% — Meta, down 4.82 percent today Llama 3 authors traced about 1.4% of unexpected interruptions to it; rare, and unbounded when it happens 1.

Certainty is bought in step time, not dollars

AWS's remedy runs inside the training loop: gradient checks, a ring replay audit where each GPU recomputes another's work, and weight fingerprints compared across replicas 1. On AWS's test, audits every 100 steps added about a tenth of step time; every 10 steps, about a third 1. AWS is the one compute seller that also sells the audit while telling customers the cheap dashboard cannot replace it.

Certainty costs up to 35% of training step time

The audit that catches what the dashboard cannot is paid in speed, not dollars

0%10%20%30%40%No audit0%Audit every 100 steps10%Audit every 10 steps35%
Data
Value
No audit0%
Audit every 100 steps10%
Audit every 10 steps35%
Added step time from deterministic in-loop audits (non-finite gradient checks, ring replay, weight fingerprints) in AWS's test of a 953-million-parameter model on 32 Nvidia H100 GPUs; the no-audit baseline is shown for comparison. Source: AWS silent data corruption guide via Tech Times, 20 Nov 2026.1

The timing lands in a market repricing the compute bill. Nvidia−10.86% — Nvidia, down 10.86 percent today and its ecosystem shed $1.4 trillion of market value in weeks 4, the neoclouds hardest, and our US AI 100 index is down about 11% over the month 5. Labs training on committed capacity should treat the audit not as an engineering nicety but as part of the cost of every GPU hour. Whoever builds verification into the platform and prices it honestly is doing the work that makes the rest of the buildout bankable. That is where the value in AI infrastructure quietly sits this week.

Get Data Center by email, free

Who is building AI's data centers, and who is paying for them. In our daily email, every weekday.

Loading META

Reader comments

0 comments

    More in Data Center

    All in this topic →

    Data Center

    Applied Digital put $97 million into ChronoScale on a note that matures December 31

    The Schedule 13D/A filed November 20 shows the funding behind ChronoScale's $200 million Super Micro GPU order: a $97 million no-interest note from majority owner Applied Digital, fully funded and maturing December 31. Conversion needs a stockholder consent still in the mail, so one balance sheet carries the timing risk.

    Data Center

    Trump threatens 'tremendous tax' on US companies that build data centers abroad

    No rate, timetable or legal authority accompanied the threat, which landed on Alphabet, Amazon, Microsoft, Meta and Oracle, the companies already building abroad, on a day the Nasdaq 100 fell 5.7%.

    Data Center

    Oracle slid to about $103, under Bernstein's $133 floor if its AI data-center business fails

    Bernstein's stress case says Oracle's databases, software and conventional cloud are worth $133 a share even if its AI data-center business fails. At $110.57 and falling, the slide suggests investors are pricing what that floor leaves out: years of negative cash flow between building the capacity and getting paid for it.

    Data Center

    Fermi falls 16% as a court lets its board rewrite bylaws by majority while shareholders need 70% to act

    The ruling leaves Fermi shareholders needing 70% of all outstanding shares to touch the board while directors can rewrite the same rules by simple majority. The co-founder who sued sold $21 million of stock the week before, and the five-day rally on Amazon's name rests on a lease that is not signed.

    Data Center

    Cameron County rejects SpaceX tax break on $24 billion Starbase campus

    The nonbinding rejection opens a fight over property taxes on a $24 billion campus, and it arrives in a week when SpaceX priced $76 billion of GPU debt.

    Data Center

    YTL Power denies its data centers moved Nvidia chips to China as data-center profit jumps nearly 133-fold

    The Malaysian utility filed its denial hours after quarterly results showed data-center profit up nearly 133-fold, the group's only growth engine. Investors marking the stock down about 20% in two weeks must weigh the company's word against a five-source account of Alibaba's chips running inside its Johor park.

    Subscribe to TheInference for free