My understanding was it was a basic failure of the generators to start and transfer load over in response to line power being lost. The UPSes took over and signaled but the generators never started.
Paralleling is pretty simple and something they don't even have to think about - it's handled easily by the controllers. Basically, the controller starts the generator and regulates the speed/etc. to produce a matching waveform (three actually, but they're tied together in time)... once everything matches it closes the contacts and continues to regulate speed to keep things matched.
Amazon's report said that an "instability" in the generators kept two generators from paralleling and going online, and that the vendor (unnamed, but I have good reason to believe it is Cat) made a software change to the engine computers to allow for a wider range of operation.
Now, there's a bit you can read into that. At the data center I work at, we have two generators, and they parallel every time. It takes 14-16 seconds from power outage to full transfer. 5 of those seconds are the dead time mid-transfer, about 2-3 of those seconds are taken in paralleling, and the rest is in engine startup to a stable RPM. The engine startup is so fast, because we have battery warming pads and dual block heaters to keep the oil and coolant at the same elevated temperature, year round.
In my install, the paralleling is not necessary, as each generator is large enough to handle the entire critical load. This is Tier III/IV.
It sounds like (although they have not gone on the record on this matter) that Amazon is going for Tier I (zero redundancy). In that scenario, they would only have enough power to handle the critical load IF BOTH generators were paralleled. So there would be an enabling relay output from the computer to disable transfer until both generators are online.
Now, here's the thing about paralleling. It is real simple when the computer handles it, until it doesn't work. It may work 100 times in a row, and fail when you need it. And that's assuming they actually test it, and not just run engine exercises weekly.
I run a full transfer to generator, twice annually. But I also have hours of battery capacity. It also sounds like Amazon didn't have enough battery to leave time for hands on-site to react to the issue.
As for how to parallel. You aren't synchronizing three waveforms. The three waveforms will already be 120 degrees from each other, because of the windings. The first generator to reach RPM and have output voltage within the nominal range will close it's connection with the output bus.
The next generator will closely (but not exactly) match RPMs with the first. Because the RPM's are different, the phase angle between the two will rotate. As the angle crosses zero SLOWLY, the second generator will close it's connection to the output bus.
Now if there is an issue with EITHER engine maintaining the exact speed (in the middle of a severe weather event), paralleling will fail. It isn't that paralleling is a complex or unreliable function (it is done easily by computers all the time), but there are still so many things that can go wrong, and the computer will always err on the side of caution by not paralleling, because should the connection be closed out of phase, the results would be catastrophic (head on collision of tractor trailer kind of energy involved, or more).
My switchgear has an insulated wrench stored inside the panels for a manual transfer. You couldn't pay me enough to touch that thing.
per hour
$24,331.20 per day