Friday, June 29, 2007

Power Outage Mystery

On June 27, 2007, one of the hottest days of the year, residents and businesses on the upper east side of Manhattan in New York City were affected by a Con Edison power outage. According to an AP story, "Consolidated Edison said the blackout affected 136,700 customers in all, or more than 500,000 people." This incident raises a number of questions regarding reliability and operation of New York's power plants and electric system.

Con Edison Restarts after Finding No Failure of Its Equipment

On the day the power went off, the AP quoted a Con Edison spokesman saying there had been a "transmission disturbance." Xinhua reported that
Con Edison Chairman and CEO Kevin Burke said Wednesday night that the blackout was caused when breakers opened at an Astoria substation and cut off power to stations servicing Yorkville and parts of the Bronx.

Why the breakers opened remains under investigation, but increased power demand due to the hot weather was not a factor, he said.

A June 28 Con Edison press release attributed the cause of the blackout to a "transmission disturbance."

Another, more detailed, explanation from Con Edison emerged in a June 29 New York Times story, It Was Lightning, Con Ed Says, That Caused the Lights to Go Out:
The utility’s managers have concluded that a bolt of lightning must have struck a small component of its network at or near its transmission substation in Astoria.... Although the lightning did not spark a fire, explosion or any damage, it caused the power system to respond as though there had been a significant mishap****

"It's a very low-probability event," said [Con Edison's senior vice president for electric operations] who has worked for Con Ed for 26 years. "I can't remember an event like this one that had this affect on transmission."

Local power failures caused by thunderstorms like the one that moved across the city on Wednesday usually result from wind and trees knocking down power lines.... This failure was unusual because after electricity was restored, no repairs were needed. ****

The best guess of the utility’s managers was that lightning struck a relay, a component that helps the system monitor how power is flowing through its lines. The high-voltage shock tricked the system into sensing a surge in power and reacting by shutting down some segments to protect others....

Within a fraction of a second, circuit breakers began cutting the flow of electricity to more than 135,000 customers on the Upper East Side and in the Bronx.... Each of those areas is supplied by one of two substations that sit across the street from each other in the southwest Bronx.

The substations shut down at 3:41 p.m. and did not start sending electricity to homes, stores and hospitals again until about 4:30, he said. In the meantime, Con Ed engineers searched for signs of damage but found no equipment that had been burned or blown up....
On June 30, a Newsday story similarly reported
A lightning strike near a Queens substation caused Wednesday's power outage that hit the Upper East Side and the Bronx, Con Edison said Friday.

Real-time lightning tracking data showed that detection instruments measured a lightning strike around the Astoria substation at 3:42 p.m. Wednesday, precisely at the time of the power loss, Con Ed said in a news release. The strike momentarily affected communication equipment that prompted circuit breakers on multiple transmission feeders to open, causing the service interruption, the utility said.
In sum, the public was informed about substations, equipment, relays, breakers, transmission lines, feeders, etc. involved in the shut down of service. But the Con Edison press release and the major news accounts do not directly mention that the incident depriving a half million people of electric service may also have involved the unscheduled shutdown of one or more power plants near the Astoria substation.

Did the Outage Also Involve the Sudden Shutdown of a Power Generation Plant?
Power plant shutdowns played a role in the California energy crisis, creating shortages that led to spot market price spikes. In July 2006, two power plants tripped shortly before the Queens outage events began, accompanied by a major price spike. New York outages and price spikes have been associated with power plant shutdowns or unusual grid events. For example, on August 2, 2006, the hottest day of the summer in New York City, NYISO operator messages indicate there were nine "large event reserve pickups," a possible indicator of abrupt power plant shutdowns, and a need to call on emergency resources, accompanied by spot market price spikes exceeding $1,800/MW . See Heat Pushes Power Prices Above Caps.

The NYISO has issued no public explanation regarding the state of the grid on June 27, 2007 event or the extreme price spike accompanying it. A NYISO Operator message indicates that on June 27, 2007 at15:43:31, "NYISO has initiated reserve pick-up." According to a NYISO Technical Bulletin, a "reserve pickup" may be needed after a generator has tripped off line:
The NYISO will initiate ten-minute operating reserve pickup if load exceeds current energy dispatch opportunities. This condition may be due to the loss of a generator**** The NYISO will terminate reserve pickup or maximum generation pickup when a sufficient level of energy has been reached.
In addition, the NYISO records indicate that three minutes after calling for spinning reserves to be used, additional electricity from the 1,160 MW Gilboa pumped storage plant was requested :
15:46:38 ISO REQUESTS GILBOA___3 OUT OF MERIT. COMMITED FOR ISO RELIABILITY AT START TIME 06/27/2007 15:00 FOR loss of con ed generation.
One of the features of the New York State Power Authority's Gilboa station is that it can increase production rapidly:
it stores water for emergency power production. If necessary, this project can be up and running within two minutes. It can "pinch hit" if another plant or line suddenly goes out of service.
The "reserve pickup" and request for Gilboa power suggests an outage of a large power plant or plants. Nothing in the NYISO operator messages mentions any "transmission disturbance" such as indicated in the Con Edison press release and news stories reporting Con Edison's explanation of a lightning strike that shut down transmission lines.

NYISO market data also indicate major anomalies during the June 27, 2007 outage. For example, real time spot market prices for wholesale electricity in the New York City zone soared from $276/MWH to $1,710/MWH from 3:50 to 3:55. The regular 5 minute report for 3:45 is missing. and instead there is a report at the exact moment the reserve pickup was initiated.

This huge price spike suggests there was an urgent need on the part of the NYISO grid managers to acquire a large amount of additional generation supply. If a large area of Manhattan were shut off due simply to a "transmission disturbance," however, the disconnection of customers would reduce electrical load temporarily and it is unlikely that there would be an urgent need for more power at far higher prices unless a significant generator that had been supplying power tripped off line.

The North American Electric Reliability Corporation (NERC) issued a press release on the day of the outage indicating
[t]he event involved the loss of generation, the loss of transmission lines over 100 kV, and the loss of service to customers.
This clearly indicates that one or more power generation plants tripped off line in the incident. The sequence of events is not known. For example, did a power plant outage precede or follow the tripping of Con Edison transmission lines? Did a power plant go off line unnecessarily after a relatively minor disturbance that should ordinarily result in just a flicker or momentary loss of service to customers?

NYISO Secrecy Rules Prevent Con Edison from Discussing Power Plant Tripping
Ostensibly, the NYISO is responsible for operation of the high voltage grid, but in practice Con Edison actually performs many functions in management of grid operations in the New York City area. If the power outage affecting consumers involved the loss of a power generation plant for any period of time, one might think Con Edison would say so. But Con Edison considers itself prohibited by NYISO rules from telling the public about unscheduled outages at merchant power plants.

For example, in a case involving the July 2007 Queens outages, Con Edison takes the position that it is prohibited from divulging information about tripped power plants, under NYISO tariffs approved by FERC. Only after PULP moved to compel disclosure, and being required to answer, did Con Edison reveal that there were two power plant outages approximately one-half hour before the unexplained secondary system fire considered to have begun the Queens event. Con Edison in its answer to PULP's discovery motion said it
cannot provide more detailed information on the generators that tripped. . . the Company is prohibited from disclosing generator outage information acquired as a transmission owner, to the extent such information is not made public by the NYISO.
The identities of the plants that tripped just before the 2006 Queens outage were not divulged, although Con Edison said one of them was "connected to the Astoria East substation...."

A History of Mystery
To conform with the PSC's restructuring vision, Con Edison sold most of its New York City area power plants to merchant power companies. (Con Edison was allowed to form a holding company, which now operates its own merchant power plants in other states, through an affiliate, Con Edison Development). In New York, Con Edison must now buy power from the merchant power companies for resale to retail customers. When power is scarce, for example, when a power plant is offline during periods of heavy load, the essentially deregulated market rates allowed by FERC may spike dramatically, even if costs to produce electricity do not go up.

Blackouts in New York since 2002 raise questions about merchant power plants tripping offline. These blackout incidents include:

1.The 2002 Reliant/Astoria Incident
A PSC Order considering reliability incentives mentions a 2002 incident where Con Edison and a merchant power generator, Reliant, traded accusations of misoperation:
On July 29, 2002, 9,251 Con Edison customers receiving power from the Queens radial system lost power up to five and one-half hours. The customer interruptions were attributed by Con Edison to an incident with a step-up transformer and its associated circuit breaker in the Reliant Energy’s Astoria Generating Station, and related impacts on Con Edison’s transmission and distribution systems (Reliant-Astoria incident).
Reliant, owner and operator of the Astoria Generating facility, acknowledges that its step-up transformer and associated circuit breaker (G5EN) failed. Reliant, however, argues that subsequent events, including the trip of Astoria Generating Units 2, 3, and 4, and the resulting customer outages in Queens, "were caused by Con Edison controlled relays, misoperations and the repeated closing of Astoria East breaker 4E into the G5EN."
Con Edison contends that it is not at fault for the outages associated with the Reliant-Astoria incident because it was Reliant’s equipment that caused two severe and simultaneous faults that exceeded the company’s design criteria for the electric distribution system.
The PSC decided that case without resolving the issue whether Con Edison or Reliant was responsible for the customer service outages.

2. The August 14, 2003 Blackout
The Joint U.S. - Canada Task Force Final Report on the widespread Northeast blackout of August 14, 2003 indicates at page 94 that in New York, some of the power plants sold off to new owners during the "restructuring" orchestrated by the New York Public Service Commission were set so they unnecessarily tripped at low disturbance levels, exacerbating the cascading outage:
In particular, it appears that some generators tripped to protect the units from conditions that did not justify their protection, and many others were set to trip in ways that were not coordinated with the region’s under-frequency load-shedding, rendering that UFLS scheme less effective. Both factors compromised successful islanding and precipitated the blackouts in Ontario and New York.
The Task Force Report at page 96 noted its frustration with information provided by the power generators:
Unfortunately, 40% of the generators that went off-line during or after the cascade did not provide useful information on the cause of tripping in their response to the NERC investigation data request. While the responses available offer significant and valid information, the investigation team will never be able to fully analyze and explain why so many generators tripped off-line so early in the cascade, contributing to the speed and extent of the blackout.
The Times said, "[a]ccording to the report, some were set to trip much sooner than protecting the generator required." The Task Force Recommendation for rigorous investigation into the relationship of industry restructuring and reliability was not fully heeded. Conferences were held, with numerous participants agreeing that restructuring had reduced cooperation and increased the risks of blackouts, but no agency made specific findings and recommendations. See What Happened to the "Independent Study" of the Effects of Electric Industry Restructuring on Reliability?

3. The July 17, 2006 Queens Outage

Just five days before the July 17, 2006 Queens outage, Federal and NYISO grid officials had warned that with major Con Edison transmission lines out of service, New York City was at risk of load shedding in the event of hot weather or further outages. See The Queens Blackout and Queens Power Outage Update. The hot weather came, and, as discussed above, further outages did too, including a power plant at the Astoria East substation, accompanied by a spot market price spike that roughly quadrupled prices for Long Island and New York City zones. Con Edison maintains that the power plants that tripped just prior to the beginning of the 2006 Queens outage are not related to the outage.

4. The June 27, 2007 Outage
Whether power plant tripping was involved in the June 27, 2007 New York City outage is not known. There are indications, however, that one or more power plants may have tripped off line in connection with this particular incident. These indications include
  • Con Edison reports no damage to any of its equipment
  • Con Edison mentions a power surge or disturbance
  • NYISO operator messages indicate a reserve pickup and loss of Con Edison generation
  • NYISO real time spot market prices spiked dramatically during the customer outages
  • NERC's press release mentions a loss of generation
More Scrutiny of Power Plant Operations Needed
In addition to the above incidents, minutes of the New York State Reliability Council reveal that in the month of July, 2006 “Indian Pt. 3 and Astoria Energy East each tripped twice at near full load.” Obviously, the tripping of merchant power plants at times of heavy load should be fully investigated by the PSC and other entities that may have jurisdiction regarding either reliability, reasonability of rates, or market conditions. The Joint U.S. - Canada Task Force report on the 2003 blackout indicated that power plants in New York tripped unnecessarily at low disturbance levels.

Additional scrutiny is particularly important because the NYISO market rules allow sellers of electricity to reap vary large financial rewards at times of scarcity. Such scarcity can be created by a power plant tripping off at times when demand for electricity is high.

Although the New York PSC has adopted a policy of "lightened regulation" over new electric companies that purchased power plants from the old utilities, the PSC retains supervisory authority with respect to matters such as enforcement, investigation, safety, reliability, and system improvement, and has the power to see that all electric companies, including the merchant power generation companies and the NYISO, perform their public service duties. As stated by the New York State Court of Appeals in Matter of Astoria Gas Turbine Power, LLC v. Tax Commission of City of New York, :
the PSC maintains "light regulation" over AGTP covering "matters such as enforcement, investigation, safety, reliability and system improvement . . . . This light regulation also gives the PSC authority to limit AGTP's power in the market and any actions in contravention of the public interest.
The reliable operation of all electric companies in the state is a very major matter affecting the public interest, requiring continued vigilant PSC supervision. This duty cannot simply be "divested" by the PSC and left to NERC or FERC, the utilities, or other entities with whom they may have contracted to actually operate the power plants.

Non Utility Operation and Management of Power Plants
There is a recent trend for an owner of a merchant power plant to be a Limited Liability Corporations (LLC) formed by financial investors, the main asset of which is the power plant. The LLC then contracts with a third party company for services, such as actual operation of the plants, fuel purchasing, and possibly marketing of the output. In some cases, the actual operator of the plant is not a utility company regulated by either the New York PSC or FERC.

For example, an Astoria merchant power plant built to supply energy to Con Edison customers is not actually operated by the lightly regulated utility owner, but by an independent third party company, North American Energy Services Company. Notwithstanding a name suggesting the local hemisphere, North American Energy Services Company is actually owned by a large Japanese corporation, according to a June 16, 2006 press release:
North American Energy Services Company (NAES), a broad-based provider of services to the power generation industry, announces that it assumed full responsibility for the operations and maintenance (O&M) of the Astoria Energy Facility (Astoria) on May 21, 2006. * * * * NAES is owned by ITOCHU International Inc., the U.S. affiliate of ITOCHU Corporation. With operations in over 80 countries covering a broad range of industries, ITOCHU is among the world's largest corporations.
ITOCHU Corporation describes NAES as "the world’s largest independent power plant operation and maintenance firm." In November 2006, ITOCHU bought a U.S. company that sells natural gas -- the fuel used for the Astoria power plant -- and announced that "we intend to expand the sales territory to include the U.S. East Coast, which constitutes an immense demand." Hypothetically, if sales of the affiliate were expanded to the New York area, NAES could be in a position to purchase natural gas for the power plant from its affiliate, possibly affecting the cost of electricity to Con Edison customers if the contract between Con Edison and the power plant owner provides for fuel cost adjustments. The affiliated gas marketer, though subject to FERC regulation, would be dealing with NAES, not a utility affiliate, and so might not be subject to any of FERC's rules regarding affiliate transactions.

In its recent Order 697 adopting regulations for market rate sales of wholesale electricity, FERC considered criteria for deciding if an energy manager of a power plant has become a utility subject to the agency's oversight and jurisdiction. NASUCA reply comments urged FERC to "adopt a rule that at a minimum encompasses the exercise of control over prices, bids, or output, including the ability to affect the cost of fuel and other inputs to generation." The NYISO made a similar argument for broad inclusion of the new energy asset managers as utilities. FERC, however, decided not to issue firm guidelines on the degree of control sufficient to deem that third party asset managers are utilities subject to FERC oversight, instead leaving the determination to a case by case assessment. As a result, it is possible that some third party operators of power plants may not be utilities subject to direct jurisdiction of either the PSC or FERC.

July 12, 2007 Update: Three Astoria Power Plants Tripped
The United States Department of Energy (DOE) requires electric utilities to file reports of major distubances immediately, including incidents that affect service to large numbers of customers and outages of major power plants. The DOE website publishes summaries after a three-month lag. The reports filed with DOE by Con Edison and NYISO clearly state that the June 27, 2007 outage events involved power plants that tripped off line, and identify the power plants that tripped.

The Con Edison report to DOE of the June 27 incident states the event began at 15:41. In addition to substation breakers tripping, "the generator units Astoria Unit 3, Astoria Unit 4, and the NYPA CC1/2 plant, which were all connected to the Astoria West yard, tripped offline:
Con Edison is examining data and equipment to help determine what caused the outage. Engineers are examining possible links between the outage and lightning strikes in the area. It appears that there were several strikes near a transmission substation in Queens at, or near, the time of the event.
The Astoria power plants that tripped are very "near" to the Astoria substation.

The NYISO report states the event began at 15:42. NYISO states that "Consolidated Edison is the appropriate party for the DOE to obtain additional analysis of this local area event."

The detailed sequence of grid disturbance events (typically recorded in thousanths of a second) is not contained in the NYISO and Con Edison reports.

A detailed timeline of events could reveal if one of the power plants tripped off before the Con Edison substation breakers tripped, or if the breakers tripped first, followed by tripping of the power plants.

According to a news story, Blackout Boosted Power Price 900%, a NYISO spokesman said that the price spike on June 27 would be "unnoticeable" to consumers and the ISO does not believe the spike resulted from price fixing - which has been suspected recently in New York's electricity system."

Overt price fixing to rig the wholesale electricity markets was made a federal crime by Congress in 2005. FERC then issued regulations interpreting the law to require "scienter" or specific intent to sustain any penalty. If New York generators independently decided to allow their plants to trip more easily, as suggested by the U.S./Canada Task Force
on the2003 blackout, and if they engage in FERC-approved hockey-stick bidding strategies, plants could trip unpredictably requiring emergency power at very rewarding rates that are not directly manipulated, as defined by FERC.

No comments: