Read a report that major power failure due to overloaded transmission lines at a major switching point.
Power out from Orange co California - San Diego - Yuma Az. Several million persons without power and possible not fully restored until Friday under a major heat wave.
This will probably affect the Surfline and Sunset routes as the backup batteries at signal points go dead. Crossing signal as well.
Metrolink reports that they are not having any delays. But metrolink reports Amtrak and Coaster are experiencing 30 - 40 minute delays.
Amtrak is unable to report train status and has no reports of trains in the area.
All San Diego transit websites are down so no information. It amazes me that they have no back up power to run the website. Maybe some heads will roll when this present crisis is resolved.
According to CAISO (California Independent System Operator) a 500kv AZ intertie line tripped out about 530pm dropping San Diego Power system offline as well as a bit of SoCal Edison. About 5000 MW load was lost. Current load is about 38K MW in CA.
A lot depends on the condition causing the trip in the first place, although with excessive heat comes overheated switchgear and transmission lines that short (across phases in the wind or to ground) or just collapse due to overheating. It’s what caused the last major NE outage 100K mw in '04 IIRC. A primary transmission line in northern Michigan was overloaded and shorted to ground. This caused a ripple effect which cascaded across the NE power grid for about 45 min before the system dropped offline.
First problem is restoring the main transmission supply. There can be problems.
This is unlikely to occur from Az as the Az grid is in difficulty from fires in the eastern part of the state.
Generally, an electrical utility such as San Diego Power will have other interconnection agreements or service connections that will allow them restore power from other sources to their grid as well as supplement their ability to get their own stations back on line. The problem is also getting their load split so they can bring their system back up in stages without triggering another collapse. This takes time especially in overheated conditions like this. They’re lucky. 5k MW is a lot but compared to other major outages this is relatively small as far as Ca is concerned.
However…
Arizona appears to be in trouble because of the forest fires and concurrent winds that are threatening their eastern transmission lines and affecting, as a result, their northwestern Texas interties and possibly the rest of their system as a result. How quickly they can import power from elsewhere out of the state without overloading their interties as well as their major transmission lines and restore/balance their load will be trick
The server hosting the website may well be up and running. Just like anything else, one weak link can make it look like there is lots wrong when the problem is just the weak link. One router whose backup power failed could be the problem.
Too, the existing links may just be “oversubscribed” under the circumstances, and unable to handle all of the traffic.
Charlie: Do not have any idea of battery capacities. Each location will probably be different depending on amount of battery capacity; load at each bungalow, etc. A bigger problem will be how many standby generators the signal maintainers can round up to power the various locations and maybe the lack of communications due to RR radios not working or battery back up dying?
Secondary problems –
Overloaded Cell phone service and probably some towers losing back up power.
Signal maintainers running out of time on the hours of service rules that apply to them.
Stalled trains also loosing crews due to hours of service. Any crossings may need to be approached with caution in case they are not working?
Don’t forget it also take electrical power to run the information superhighway - the communications companies may not have sufficient back up power to run their ENTIRE networks. When the MAIN power source for a entire area goes down - that is the cataclysmic event that really has not been planned for. In power failure situations you can plan for a failure at several individual facilities and have back up power for them to access the rest of the working network…but to have the entire area, including all the nodes of the network fail…catastrophic.
San Diego Gas and Electric’s website is up and running. Their outage map shows every circuit offline.
They have no estimate of restoration schedule beyond a general statement it will be sometime tomorrow.
CAISO power dispatch is running on verbal instruction currently which means that all automated switching systems are suspended as they work at restoring the San Diego area. This is a standard procedure, IIRC, when staged restoration of power to an affected area is being planned and is ready to be effected. It literally has to be done manually, circuit by circuit, in the control rooms, switchyards, the substations and on the poles.
It’s going to be a long hot night and day for SDGE and CAISO power dispatch, especially, as they work at putting the San Diego system back together. They’re even bringing in power from Mexico…didn’t realize they also had a southern intertie. US western interties must be at capacity for that to happen.
Signal maintainers must be pulling their hair out in that area. Hope the batteries hold out.
OK, folks, power has been coming back up across San Diego County since around ten this evening. The 500 KV transmission line from Az actually was back up relatively quickly, as the fault was, according to the SDG&E CEO, caused by a “human error” event at the Hassayampa switchyard in western AZ. Unfortunately, the voltage sag caused by the line interruption at 3:42 PM PDT caused a number of power plants in the area to drop off the grid, including the San Onofre nuclear plant, which was supplying about half of the capacity being used, or slightly more than half.
Power was out throughout the SDG&E service area, Imperial County (IID) and northern Baja California. The outage does not seem to have been heat related or caused, just coincidental to the end of a few warm days, which excepting the humidity and rain are not that unusual this time of year out here. (I’m just glad we didn’t have our usual Santa Ana conditions.)
For the moment, officials with MTD are saying that they will not be running the trolley system until most of the power network is restored. Amtrak and Coaster (NTCD) are saying that they will be running normal schedules Friday but expect delays, and bus service will replace any annulled trains. All schools will be closed tomorrow, pretty much.
Power at the homestead was restored nine hours and ten minutes after it went down. Still a lot to be done, though, it would seem.
Good to hear your grid is coming back up. I didn’t realize that you had a nuclear plant in the area…for some reason I thought it was further north.
San Onofre plant was probably base loaded which is why it was carrying half or more of the SDGE load. This is not uncommon for nuclear plants. I imagine it will take them some time to get their reactors up after an emergency shutdown. That will cause some delay for SDGE I expect. They appear to rely heavily on nuclear generated power.
The basic problem for the SD utility as with most utilities in such circumstances is to incrementally restore load to the system in a way that does not result in overloads leading to emergency trips leading to another cascade of the sort that happened yesterday. Major loads such as transit, manufacturing as well as residential/commercial airconditioning are the usual areas of difficulty at restoration. I’m guessing a good day by the time near normal system operation is resumed.
Got a friend whose husband works in the SD Transit System - System Maintenance. I think I’ll give him a call when the dust settles.
Interesting that the 500kv trip occurred at 345pm. CAISO’s load graph for yesterday shows a precipitious 5K MW load shed about 5 pm. They had declared a maintenance emergency at 416pm. The cascade must have taken a while to reach the point where GS’s started tripping offline.
Sounds also like the RR’s are breathing a sigh of relief.
Thanks for your comments and insights, Charlie - very informative ! (and from Chuck and others, too)
Not to quibble about a trivial point of history, but I believe that outage was initiated by trees falling across several of FirstEnergy’s transmission lines in Ohio, and from there it then went around the Great Lakes. Also, it was in August 2003 if memory serves (which it might not . . . ).
Yeah, there was a pretty fast dump of load due to the line interruption. The CALISO graph showed the drop at just past 3:30 PM PDT. I actually was not aware of the extent of the outage when I checked the graph the first time, as I figured it was a very local failure. When I saw that stepped loss in capacity and use, I knew we were in for a long outage.
I am a retired EE, though not a power weenie, but knowledgeable enough to get registered before they changed the exams so you don’t have to know squat about power to pass. I listened to a couple of the new conferences and almost laughed out loud when the “journalists” couldn’t get their minds around the concept that you had to bring a system back up incrementally. It was even worse when the SDG&E guy was telling them that it would be a while before they could get even the fossil-fuel plants back up and on line.
As I write this, the outage map still has eight dots on it, which is a lot better than the measles of last night. On any given day during the summer, there are nearly that many outages being worked, so it looks like they are almost back to normal. San Onofre will be coming on line within a day or so, they think, which should lighten the load on the transmission lines coming in.
Interestingly, the power peak on the CALISO plots is always right around four in the afternoon, and the demand cycle is nearly sinusoidal, except that there is a sundown deviation always, year round. The statewide capacity reserve was about 12 Gigawatts at the time of the outage. The last interesting historical item is the massive NIMBY opposition to SDG&am
Earlier in my working life I was a powerhouse operator (diesel and hydro) and later system operator. Got my start as a power engineer, 3rd class. I’m not an electrical engineer, rather a ticketted operating engineer interested in the ins and outs of power generation. The ISO’s make it easier to track events than was previously the case when I was operating. We did pretty much everything, even to synchonising equipment to close in and load it. We controlled s and voltageIt was an education…
Afternoons in the powerhouse in summertime could be interesting. Airconditioning and resistive loads such as stoves firing up for supper etc always bounced the load meters about 4pm as you say. We would make sure we had enough reserve spinning to take up load quickly if needed. Problem for us was potential transformer overheats resulting in trips. The other concern was lightening strikes.
We lost a high line (138 kv) to lightening once when I was working in the diesel plant. Three strikes in quick succession and then the transmission breakers cut out. At the time there were 6 engines on line. All we two (plant foreman and shift operator) could do was sit there and watch as 50MW tripped offline engine by engine. Nothing we could do at the board or manually made any difference. It was like grotesque ballet. By the time it was all done (about 10 minutes IIRC) we didn’t even have station service to operate the spillway regulating gates at the hydro down the hill. The hydros had gone and the battery bank we used to start them was in the tank at the time.
We had to get a couple of CAT sets (397’s) started in order to begin restarting the system. Ours was just a small one, about 100MW, but the effect was the same.
Like I say, it’s every system or plant operators worst nightmare. I feel for the guys at SDGE.
Sounds like it is coming up quicker than predicted.
Not being electrically inclined except that I am able to reliably plug something into an outlet.
What I don’t understand about these kinds of black outs is the cascading nature of them. Failure A occurs which creates Failure B and so on until the entire system has failed. I understand that the various generating systems are trying to protect themselves, but it seems that the procedures currently being used just spread the failure.
We’re both right. If I recall the inquiry report that followed that event and it was extensive.
The originating problem I believe was line sag caused by over capacity from a First Energy GS in (I thought) northern Michigan (perhaps it was Ohio). I know First Energy served there as well. It can be a problem as you know when utilities get in a competitive electrical market which is what ISO’s are all about. I had a millwright friend in one of the GS’s at Niagara Falls. Remember talking over the event with him. I’m pretty sure it was i
August 14, 2003 - apparently 4 separate sets of lines eventually overheated and sagged into trees below and “faulted” or “grounded” - see “The 2003 Northeast Blackout–5 Years Later” datelined August 13, 2008 by JR Minkel in Scientific American at:
As stated, it cascaded around the Great Lakes to bring down some Canadian systems as well. One other aspect not noted is that some of our political types here in the US - with their ‘half-vast’ knowledge, experience, and integrity, of course [swg] - initially attempted to cast aspersions on and blame our Candian friends for this event. Please accept my apologies - they were no credit to either their office or the concept of “truth” (as we’ve since found out to our great regret).
Paul North.
P.S. - Charlie, it appears we were posting almost simultaneously - I was 12 at the time of the 1965 Blackout. Back then I had a “great-uncle” who was a marine (shipboard) operating engineer for large iron-ore vessels for Bethlehem Steel Co. and interested in such things, and he would often visit generating plants with me and my father in tow, so I suspect that’s at least part of what got me interested too. (Back then a grizzled old guy with a Marine Engineers Beneficial Association union card, a kid, and an interest could get into almost anyplace informally.) On one trip, we visited one of the Beck plants at Niagara Falls, and someplace I have really poor grainy and dark B&W photos of the Niagara Junction Rwy.'s steeple-cab electrics ! Anyway, in neither event did the power go down where I was - in the Pennsylvania-Jersey-Maryland or P-J-M grid, the first “ISO” there ever was, and it has a pretty good&n
Basically what you get is like ripples back and forth on a pond’s surface; however what is happening is that as the generating capacity decreases (due to plant dropoff) and the load remains much the same, the remaining (and reduced number and output capacity) plants try to take up the load. Their protective relays drop out either instantly or on a time delay (relays can be set for either, or at least were in my day) to preserve their generating and transmission equipment from damage. These delays, dropoffs and reductions of generating capacity ricochet load back and forth across the grid to different GS’s at increasing speed until the whole system finally drops out. Voltage and amperage fluctuate in inverse proportion (even more in these in these situations) and from my experience it is difficult to track in real time as the events are relatively small to start and then cascade to the point system failure begins. Then it’s everyone for themselves which is why Beck2 stayed online in 04. Their operators cut off the grid quickly enough and were able to preserve their own plant service area as a result. It’s not always so much the operating procedures as it is the operation of protective equipment in conjunction with procedures. Sometimes I’m sure it’s dumb luck…
The other thing to remember is that the NA power grid is older and approaching a critical point in its service life. This mea
There was an electrical distribution problem in SOUTHERN CALIFORNIA! While the residents of that area seem to think that they are the center of the universe, SouCal is NOT, “SW USA”
Here in Sin City, in the sane part of the Southwestern US, the lights didn’t even flicker.
August 14, 2003 - apparently 4 separate sets of lines eventually overheated and sagged into trees below and “faulted” or “grounded” - see “The 2003 Northeast Blackout–5 Years Later” datelined August 13, 2008 by JR Minkel in Scientific American at:
As stated, it cascaded around the Great Lakes to bring down some Canadian systems as well. One other aspect not noted is that some of our political types here in the US - with their ‘half-vast’ knowledge, experience, and integrity, of course - initially attempted to cast aspersions on and blame our Candian friends for this event. Please accept my apologies - they were no credit to either their office or the concept of “truth” (as we’ve since found out to our great regret).
Paul North.
P.S. - Charlie, it appears we were posting almost simultaneously - I was 12 at the time of the 1965 Blackout. Back then I had a “great-uncle” who was a marine (shipboard) operating engineer for large iron-ore vessels for Bethlehem Steel Co. and interested in such things, and he would often visit generating plants with me and my father in tow, so I suspect that’s at least part of what got me interested too. (Back then a grizzled old guy with a Marine Engineers Beneficial Association union card, a kid, and an interest could get into almost anyplace informally.) On one trip, we visited one of the Beck plants at Niagara Falls, and someplace I have really poor grainy and dark B&W photos of the Niagara Junction Rwy.'s steeple-cab electrics ! Anyway, in neither event did the power go down where I was - in the
Chuck: I take full resopnsibility for the subject title. At the time I sent the post that was all my source knew that it was in the SW as he was in the dark and heard a radio report. MY POST WAS DELAYED IN SHOWING UP FOR SOME REASON.
Streak, in your defense, we are south and west down here, about as much as you can have of the combination. The Lost Wages slot machines didn’t blink because of the opening of the transmission line from the north coming by way of San Clemente. Things could have gotten out of hand without that happening.[:)]
With respect to having a network go down like we had yesterday, you all gotta remember that those pesky electrons are quick little suckers. Actually, it’s not so much the electrons as the electric fields and such.
The only way to really handle this kind of situation is to load shed within a couple of hundred milliseconds of the loss of a power source. Unfortunately, by the time a human operator or supervisory computer becomes aware of the situation, those milliseconds have gone away and the grid voltage is going to begin to sag precipitously because the loads are still demanding power. That rapid sag will initiate protection because, first and foremost, you have to protect the grid components, especially generation, or you are for sure up you know which creek. I have my doubts whether one could actually fail-safe a portion of the grid of the size we saw yesterday.
As I noted in an earlier post, the amazing ignorance illustrated in some of the press conference questions last night was just pathetic. You don’t just throw a switch and everything comes back up magically. The fact that the electric power grid works as well as it does obscures the public from the fact that it is a very complex system, the design and operation of which requires legions of educated, dedicated an