A false smoke alarm nearly drowned Jane Street's New York data centre

A false smoke alarm at Jane Street's only New York data centre filled the sprinkler pipes with water and shut off the cooling. The room started to become a sauna. A systems engineer who lived nine minutes away ran in and pulled the power cords, cabinet by cabinet, before the heat could open the sprinkler heads.
How a dust cloud almost flooded a trading firm
The account comes from Michael Bacarella, who was on Jane Street's systems team and published it on November 10. It is a first-person memoir of an incident that took place before Hurricane Sandy hit New York in October 2012, not a failure at the firm this month. Jane Street, he writes, had long insisted on keeping its machine room next to the trading floor: first because it did not want servers stolen, and later so that a runaway trading system could be stopped by running across the hall and yanking the power cables. A partner who had served on a nuclear submarine had helped set that culture. As the firm grew, the room became a full data centre, with an online UPS, computer-room air conditioners, a diesel generator in the basement and about ten rows of cabinets, ten deep, packed into 1 New York Plaza.
Contractors were working in a prep room next door. Dust from the work set off the smoke alarm. In a machine room the sprinklers are usually dry until an alarm is confirmed: the pipes then fill with pressurised water, but the heads themselves do not open until heat melts a plug, typically around 150°F. The same alarm, Bacarella writes, was wired to shut down the CRAC units, so that air would not feed a fire. That is the trap. A room full of servers without cooling heats up fast. If it gets hot enough, the already-charged sprinklers will do exactly what they are designed to do, and drown the firm's only New York machines.
Hong Kong noticed first. Someone saw temperatures climbing on server after server; no single box was set to alert just because it was a little warm. Bacarella was about to get into the lift at his apartment, a nine-minute walk from the office, when the call came. He ran down Water Street, waited for the lift to the 33rd floor, badge-swiped through the office and the machine room, "stepped into what felt like a sauna and went row by row, cabinet by cabinet, yanking power cables out with both hands." The temperature monitor climbed to 141°F, then 142, 143, 143.5, and stopped at 143.6. The markets were already closed, so the downtime was short; the room was back up the next morning.
What the firm did next
The post-mortems, he writes, were "matter-of-fact, level-headed" rather than a hunt for someone to fire. The sprinklers would stay. Procedures and surveillance would tighten. The servers would remain in the office, next to the trading floor, "until Hurricane Sandy smacked us upside the head."
The other change was monitoring. The Hong Kong team had caught the overheating by luck. The next morning Bacarella wrote a small program in OCaml that queried the temperature sensors on every Linux server in each location, once a minute, averaged them, and alerted the systems team if the mean was high. CRAC telemetry and room sensors, he argued, were less reliable than asking the machines themselves.
Gigazine published a recap of the same account this week. The mechanical logic Bacarella describes is the standard design of a pre-action, or double-interlock, sprinkler system: water is held back until a smoke alarm is confirmed, and each head still needs heat before it sprays. In a data centre that design is meant to prevent a leak from destroying equipment. Wired to the cooling, it can create the heat that then opens the heads.
Our take
The near-miss is almost too neat. Jane Street kept its computers across the hall so that someone could always run in and pull the plugs. That is exactly what saved them, and it saved them from a fire system, not from a fire. The partner's submarine instinct, a red lever you can pull when the automation is wrong, turned out to be the right one, just not for the reason anyone had in mind.
There is a wider lesson for anyone who puts a high-powered machine room inside an ordinary office tower. Fire code and trading-risk culture can be individually sensible and jointly dangerous. Dust is enough to start the sequence. The useful response was not to rip out the sprinklers, which the firm declined to do, but to notice a building-wide temperature rise before it reached 150°F, and to have a person close enough to act. Bacarella's later monitor is the unglamorous version of that: ask the servers if they are getting hot, every minute, and treat a small, general increase as an emergency. It is the sort of boring safeguard that only gets written after the room has already felt like a sauna.
Sources
- Michael Bacarella, How a false smoke alarm almost drowned Jane Street's only NYC datacenter (November 10, 2026)
- Gigazine, A data center nearly got flooded due to a sprinkler malfunction (November 12, 2026)
- Wikipedia, Fire sprinkler system, on pre-action systems, including their use in data centres to protect equipment from accidental discharge







