The Graze
Eclectic enrichment
Beau's Library
Tech

A false smoke alarm nearly drowned Jane Street's New York data centre

The Graze · November 12, 2026
The facade of 1 New York Plaza, a tall beige office tower with a grid of square windows, photographed in late-afternoon light
Photo: Rhododendrites, CC BY-SA 4.0, via Wikimedia Commons (cropped)

A false smoke alarm at Jane Street's only New York data centre filled the sprinkler pipes with water and shut off the cooling. The room started to become a sauna. A systems engineer who lived nine minutes away ran in and pulled the power cords, cabinet by cabinet, before the heat could open the sprinkler heads.

1 New York Plazathe office tower that housed the machine room, on the 33rd floor
~10 × 10rows of cabinets in the in-house data centre
143.6°Fpeak temperature after the cooling stopped
150°Fthe threshold at which the sprinkler heads would have opened

How a dust cloud almost flooded a trading firm

The account comes from Michael Bacarella, who was on Jane Street's systems team and published it on November 10. It is a first-person memoir of an incident that took place before Hurricane Sandy hit New York in October 2012, not a failure at the firm this month. Jane Street, he writes, had long insisted on keeping its machine room next to the trading floor: first because it did not want servers stolen, and later so that a runaway trading system could be stopped by running across the hall and yanking the power cables. A partner who had served on a nuclear submarine had helped set that culture. As the firm grew, the room became a full data centre, with an online UPS, computer-room air conditioners, a diesel generator in the basement and about ten rows of cabinets, ten deep, packed into 1 New York Plaza.

Contractors were working in a prep room next door. Dust from the work set off the smoke alarm. In a machine room the sprinklers are usually dry until an alarm is confirmed: the pipes then fill with pressurised water, but the heads themselves do not open until heat melts a plug, typically around 150°F. The same alarm, Bacarella writes, was wired to shut down the CRAC units, so that air would not feed a fire. That is the trap. A room full of servers without cooling heats up fast. If it gets hot enough, the already-charged sprinklers will do exactly what they are designed to do, and drown the firm's only New York machines.

Hong Kong noticed first. Someone saw temperatures climbing on server after server; no single box was set to alert just because it was a little warm. Bacarella was about to get into the lift at his apartment, a nine-minute walk from the office, when the call came. He ran down Water Street, waited for the lift to the 33rd floor, badge-swiped through the office and the machine room, "stepped into what felt like a sauna and went row by row, cabinet by cabinet, yanking power cables out with both hands." The temperature monitor climbed to 141°F, then 142, 143, 143.5, and stopped at 143.6. The markets were already closed, so the downtime was short; the room was back up the next morning.

What the firm did next

The post-mortems, he writes, were "matter-of-fact, level-headed" rather than a hunt for someone to fire. The sprinklers would stay. Procedures and surveillance would tighten. The servers would remain in the office, next to the trading floor, "until Hurricane Sandy smacked us upside the head."

The other change was monitoring. The Hong Kong team had caught the overheating by luck. The next morning Bacarella wrote a small program in OCaml that queried the temperature sensors on every Linux server in each location, once a minute, averaged them, and alerted the systems team if the mean was high. CRAC telemetry and room sensors, he argued, were less reliable than asking the machines themselves.

Gigazine published a recap of the same account this week. The mechanical logic Bacarella describes is the standard design of a pre-action, or double-interlock, sprinkler system: water is held back until a smoke alarm is confirmed, and each head still needs heat before it sprays. In a data centre that design is meant to prevent a leak from destroying equipment. Wired to the cooling, it can create the heat that then opens the heads.

Our take

The near-miss is almost too neat. Jane Street kept its computers across the hall so that someone could always run in and pull the plugs. That is exactly what saved them, and it saved them from a fire system, not from a fire. The partner's submarine instinct, a red lever you can pull when the automation is wrong, turned out to be the right one, just not for the reason anyone had in mind.

There is a wider lesson for anyone who puts a high-powered machine room inside an ordinary office tower. Fire code and trading-risk culture can be individually sensible and jointly dangerous. Dust is enough to start the sequence. The useful response was not to rip out the sprinklers, which the firm declined to do, but to notice a building-wide temperature rise before it reached 150°F, and to have a person close enough to act. Bacarella's later monitor is the unglamorous version of that: ask the servers if they are getting hot, every minute, and treat a small, general increase as an emergency. It is the sort of boring safeguard that only gets written after the room has already felt like a sauna.

Sources

Share
Keep grazing

Latest on The Graze

Back to The Graze →
Shoppers in France bought 25.31 million bottles of Prosecco in the year to July, just ahead of 25.23 million bottles of Champagne, Circana…Food & Drink

Prosecco has outsold Champagne in French supermarkets

Shoppers in France bought 25.31 million bottles of Prosecco in the year to July, just ahead of 25.23 million bottles of Champagne, Circana figures show.

33m ago
Rolls-Royce Motor Cars unveiled a commission-only backgammon set whose Spirit of Ecstasy stays upright as the board opens, after more than…Style

Rolls-Royce's backgammon set has a Spirit of Ecstasy that stays upright at any angle

Rolls-Royce Motor Cars unveiled a commission-only backgammon set whose Spirit of Ecstasy stays upright as the board opens, after more than a year of development at Goodwood.

24h ago
A coyote was photographed on the southern slope of the Wallis Annenberg Wildlife Crossing on 28 October, the National Park Service said…News

Coyotes are already using the wildlife crossing over Los Angeles's 101 freeway

A coyote was photographed on the southern slope of the Wallis Annenberg Wildlife Crossing on 28 October, the National Park Service said, ahead of a 2 December opening.

24h ago
Andrea Bartzen was refused entry to MarieBelle's 25th-anniversary party on Broome Street on 16 November, Page Six reported; the organiser…Style

Palm Beach's 'fake Cartier heiress' was turned away from a New York party

Andrea Bartzen was refused entry to MarieBelle's 25th-anniversary party on Broome Street on 16 November, Page Six reported; the organiser said she was not invited.

24h ago
The California DMV barred driverless freeway operations after construction-zone incursions, and Waymo paused the rides in every market…Technology

Waymo took its robotaxis off freeways nationwide after three drove into a work zone

The California DMV barred driverless freeway operations after construction-zone incursions, and Waymo paused the rides in every market where it ran them.

25h ago
Built with Hyundai Card, the agent checks balances, picks the best of three quotes and sends the order to a separate Kinexys agent to…Finance

JPMorgan tests an AI agent that shops three banks for the best currency deal on its own

Built with Hyundai Card, the agent checks balances, picks the best of three quotes and sends the order to a separate Kinexys agent to execute, in a test environment for now; CNBC reports a wider phase is targeted for the first half of 2027.

26h ago
See all stories →