Super Bowl Outage

[RustFarmer]

From today's Chicago Tribune ......

"S&C Electric Co. said the outage, was due to the demand for Superdome stadium power needs exceeding a "trip setting" for an electrical relay it supplied.

The device was specifically designed to prevent an outage if a cable connection to the stadium failed. Testing since the game has revealed that a higher power setting would have prevented the equipment from causing the disconnection, S&C said in a statement.

S&C doesn’t control the power settings on the equipment it provides.

S&C equipment is commonly used where high reliability is critical, he said, including data centers for United Parcel Service Inc., drug manufacturing centers and hospitals. The company also works with other stadiums throughout the U.S. and Canada."


Question for the Sparkies on the forum. Is the S&C "relay" a programmable circuit breaker that can be set to different trip settings? How many amps can these things handle? Anyone work with this size of equipment?

Thanks


http://www.chicagotribune.com/busin...-in-super-bowl-outage-20130208,0,361768.story

www.sandc.com

 

[Falcon67]

Superbowl MVP

 

[sparky36000]

These two devices basically perform the same function, but obviously that's where the similarity ends. I don't know if the relay/circuitbreaker at the Superbowl was anything like this or a smaller power breaker of some kind. The large power CB's are very adjustable and configurable unlike your shops simple thermal mag CB.

 

[frankush]

From what I understand, most of S&C's equipment is medium and high voltage. Anything 600V or less is considered low voltage to them. I'm pretty sure that things like lighting in the dome is probably 480V. When all those lights went out at once, they had to lose feeds to the panels that supply the transformers that supply the 480V panels. We're talking big equipment and high dollar items here. Somewhere along the line an engineer screwed up by specifying the wrong trip assembly.

High voltage equipment comes in a limited number of sizes. You change the trip assemblies to give you the ampacity you need. They have range limits for adjustment purposes, both high and low.

I'm a 600V or less sparky and deal with large equipment in that range often. Manufacturers will make a particular frame size breaker that is capable of handling a range of amperages at a specified voltage. You select the trip assembly to fit that particular breaker at whatever amperage you require. That is also done by the engineer that designs the system.

 

[Gary S]

We should look into doing this every year for the Nerfball bowl.

 

[BigGMC]

The issued statement may have been good for the masses but to those of us in the field, its a "DUH" statement.
Most average electricians aren't exposed to this kind of equipment. It's generally power company stuff, although large manufacturing plants often have thier own high voltage protection (as they usually own thier own substation, buying high voltage power at bulk prices).
I took interest in that superbowl event when I heard it was an "issue" in the substation... I laughed and wondered when the lawsuits would begin. Almost predictable how Etnergy stated it wasn't thier fault.

Rust - In my field, the term "relay" is thrown around alot - used to describe a simple single function overcurrent unit, to devices that offer dozens of protection features all from one single electronic unit which is totally programmable for nearly every application. Those units also come with three binders of instruction manuals..... not so simple.
Not knowing what device was in service (S&C makes a large array of equipment, many I'm familier with), hard to say what its current capacity is. That's only one of many makers of protection equipment.

S&C isn't the power company, only the provider of the equipment. They dont setup/program - either electronically or by hardwiring) the equipment, they sell it. The POCO's engineering staff or hired contracting group, (which seems to be the norm these days BTW) would be responsible for coming up with the proper settings based on load demand info recieved from the customer (superdome).

Bottom line is, so long as the device in question tested out ok (based on the intented settings and wasn't defective), somebody goofed on the calculations and set the thing too light.

So to recap, it's not Etnergy's fault, it's not S&C's fault, and it's certainly not The Superdome's fault....... I bet it was Obama's fault

edit: just refreshed the page and was able to see sparky36000's second attached pic showing a couple high voltage CB's - 345Kv perhaps?
Sparky is this a yard you work in or a pic off the 'net?

 

[sparky36000]

edit: just refreshed the page and was able to see sparky36000's second attached pic showing a couple high voltage CB's - 345Kv perhaps?
Sparky is this a yard you work in or a pic off the 'net?

No that's not something I've worked on, just a picture off the net. I let the utility guys like you handle that big stuff. I've done quite a bit of work at powerhouses, but 13.8kv is about as high as we get.

 

[Norcal]

From what I heard the voltage supplied by the PoCo was 24KV, this is a subject discussed on professional electrical forums too.

"Some of you may find this interesting . The main utility company feeder supplying the majority of the load was running at about 420 amps per phase ( at 24 kv ) prior to the loss of power . AFTER the lights went out , this same feeder still had about 90 amps of load . Another feeder had about 40 amps of load that went mostly unchanged throughout the ordeal ."

Here is a link with the relay mfg. S&C Electric blaming the PoCo & they in turn blaming a faulty relay.

http://hosted.ap.org/dynamic/storie...ME&TEMPLATE=DEFAULT&CTIME=2013-02-08-11-37-10

 

[Larwyn]

It has been my experience that once all the engineers and reporters are done, no one will ever know the true cause of a major outage. But many responsible asses will be be very thinly covered while the blame either dissolves or is absorbed by the innocent.
The fact is, those who know will likely never be asked.

 

[BigGMC]

It has been my experience that once all the engineers and reporters are done, no one will ever know the true cause of a major outage. But many responsible asses will be be very thinly covered while the blame is either dissolves or is absorbed by the innocent.
The fact is, those who know will likely never be asked.

Couldn't have said it better.... was wondering if you caught wind of this thread

 

[socapots]

Someone pushed the wong button while doing some trip testing?? Maybe on the wrong equipment.
Inadvertent outages cost big money. Up here the truth usually comes out like it should when these things happen. Then companies get fined by nerc

 

[BigGMC]

Ain't nobody trip testing equipment on sun night......

 

[Larwyn]

I was once called by dispatch to go assist the city in checking on a substation that was in the dark and that darkness had naturally spread to the community which it served. When I got to the station the gate was already open and there was a vehicle there from the owning utility (we provided the high voltage feed to the city utility).

Upon entering the gate I noticed broken bits of grey porcelain under a structure. As I drove by I realized that it was bits of a lightning arrestor under a 138 KV dead end structure.

In the control house I found the chief engineer for the utility in question preparing to attempt to close the breaker back in. When I asked him about the arrestor that was scattered across the yard his response was "Oh, I didn't see that".

Turns out that the very same engineer designed, ordered and approved test results for the lightening arrestors in question. They would have been the correct arrestor had they been on the low (24 KV) side of the transformer rather than on the incoming 138 KV line.

Of course I was not surprised to hear later that the official "cause" of the outage was reported to be a faulty lightening arrestor.

 

[rockwithjason]

Question for the Sparkies on the forum. Is the S&C "relay" a programmable circuit breaker that can be set to different trip settings? How many amps can these things handle? Anyone work with this size of equipment?
[/QUOTE]
.

at this voltage level the trip electronics are generally not onboard with the actual breaker. the breaker has the mechanicals and some auxilliary switches and the trip electronics are packaged as a "protective relay" it's mostly terminology.

 

[foolishpride]

I'm not familiar with the S&C relaying equipment. Wasn't aware that they had manufactured relaying equipment.

I see alot of the G.E. Multilin Feeder and Motor Management relays today.

The relay settings should be determined after doing a coordination study. Sounds as if someone either forgot to set the settings back, after recently testing the relay, or it was a relay failure. Because the Superdome had previously held events with no issues.

 

[foolishpride]

These two devices basically perform the same function, but obviously that's where the similarity ends. I don't know if the relay/circuitbreaker at the Superbowl was anything like this or a smaller power breaker of some kind. The large power CB's are very adjustable and configurable unlike your shops simple thermal mag CB.

These are SF6 Circuit Breakers, maybe 345 KV. (Could be 138 KV, but the bushings look taller.) The relays that control these breakers are adjustable, and configurable. These relays will be housed in a control house, inside the substation.

 

[Jagmandave]

In our paper it was reported that they had two of these set identically, one tripped and the other didn't....but they were set the same.

I think that's why the PoCo is blaming the switch/breaker/relay whatever you want to call it - mfr.

Like others have said, I doubt we'll ever know the real story.

What was the cost to the Superdome, other than whatever they paid the guys to reset the breaker and turn the lights back on?

 

[FJ 432]

Here's my guess.

I believe whatever failed was on site because I never heard whether anything outside of the Superdome went dark. Reports had indicated that the building only lost half of it's power. So that means the building was fed from two separate utility supplied transformers.

So... I suspect the half time was over and someone was disconnecting additonal power feeds that were set up temporarily for the half time show. Someone opened up the wrong breaker and turned off the power. Once the mistake was made and everyone was scrambling they realized the issue and closed the breaker. The lights normally take time to re energize and that was the biggest part of the delay. Human error.

My .02.

 

[LCD]

The cost was not to the Superdome but to the bookies who stood to loose millions if underdog Baltimore's team won. Let's see... How do I disrupt a game held indoors that would be in the dark and that depends on a massive amount of elec power to keep the lights on.

Hmmmmm..... lets try BOOM

 

[foolishpride]

Here's my guess.

I believe whatever failed was on site because I never heard whether anything outside of the Superdome went dark. Reports had indicated that the building only lost half of it's power. So that means the building was fed from two separate utility supplied transformers.

So... I suspect the half time was over and someone was disconnecting additonal power feeds that were set up temporarily for the half time show. Someone opened up the wrong breaker and turned off the power. Once the mistake was made and everyone was scrambling they realized the issue and closed the breaker. The lights normally take time to re energize and that was the biggest part of the delay. Human error.

My .02.

Here's my best guesstimate. I've attached a picture of the substation that sits Northwest of the Superdome. (I only assume that this substation feeds the stadium.)

The building inside the substation contains a 15 KV Switchgear section, containing 15 KV Vacuum Breakers. Two of these breakers feed a set of cables, that each feed into a Main 15 KV Breaker in the switchgear section inside the Superdome.

The 15 KV switchgear will have a Tie Breaker in the center, to tie both ends of the switchgear together, in case they lose one of the feeders. They lost one of the feeders, due to a relay failure, or incorrect setting. They then closed the tie breaker to get them back online.

I remember during the Superbowl, the Superdome officials talking about closing the Tie.

 

[Teken]

So silly question for those not in the know. Why out of the countless games played this night something happens?

Its not like they haven't experienced a high load show? Isn't this sort of stuff tested and validated to be ready to rock and roll?

Teken . . .

 

[Outlander]

Teken - the NFL felt it was a lopsided game and people would stop watching so they purposely stalled the whole thing. According to the conspiracy theory folks in my house over dinner anyways.

 

[Teken]

Teken - the NFL felt it was a lopsided game and people would stop watching so they purposely stalled the whole thing. According to the conspiracy theory folks in my house over dinner anyways.

Are you saying the whole black out event was staged?

Teken . . .

 

[Outlander]

Are you saying the whole black out event was staged?

Teken . . .

Made for interesting dinner conversation! One of my step-sons always finds the conspiricy in everything. I thought this was a good one!

 

[Modifieddriver]

So to recap, it's not Etnergy's fault, it's not S&C's fault, and it's certainly not The Superdome's fault....... I bet it was Obama's fault

QUOTE]


Obama is Teflon coated.

So it has to be George Bush's fault

 

[Larwyn]

Well, if they did have two identical relays with identical settings monitoring the same equipment it would be an unusual relaying scheme to me. But even so, the two relays would be very unlikely to operate in the same instant. It would be a "relay race" and only the relay which actually tripped the breaker would target and it would likely be the only one to retain a fault record if that feature is included in the relay in question.

The breakers are actually just big, dumb, remotely operated switches. But they can be and are operated by all manner of electromechanical, electronic/computerized fault detection devices monitoring everything from current and voltage to to breaker position etc.

Those SF6 breakers will automatically trip if the insulating gas (Sulfur Hexafluoride) pressure gets dangerously low. Better to trip while it is still safe to do so than wait. In that case they cannot be closed back in until the very expensive gas has been replenished.

 

[Playwme]

To be honest , I didn't even realise there was a blackout. I figured that was just the standard amount of time between plays.