Until mid-2001, the oil analysis program was not utilized as an integral part of equipment reliability as compared to the more established programs: vibration and Thermography. However, during a Component Cooling Water Pump inboard bearing failure, which consequently cost the station nearly $500,000 and numerous regulatory scrutiny, the station needed to take a renewed look at its root cause. When we asked the question, "What factors contributed to the failure of a journal bearing without wear metals present in the oil?" we were forced to consider past practices along with our practices. As we all have heard this story before, signs of bearing distress were noted nearly 4 years prior to the bearing failure. Eyebrows were raised and one of the root causes was deemed to be a lack of Program Management-Ownership and accountability.

In times like this, at most any plant, decisions made point to a re-evaluation of what and why issues like this occur. At SONGS we dug deep into the core of the oil analysis program and re-justified, and in some cases, developed new tools that would enable us to maximize equipment reliability and accountability.

SONGS developed extensive program metrics, performed several program audits, and implemented extensive cross-departmental training, by way of erasing the status quo and raising the bar to obtain a world-class oil program. The performance audit below will individually explore the performance criteria and its results in more detail.

Program Metrics
In our acute awareness of this need for change, we realized that in order to elevate our oil sampling and analysis program with a World Class standard, our team would need to focus our efforts in achieving excellence. But what does Excellence look like? And by what standard is it judged? At the time of the CCW pump bearing failure, our staff didn't truly understand how to make this new decision a reality. We needed to look outside SONGS and contracted EPRI to perform a cross-sectional dissection of our program, focusing on EPRI's 12-key elements for an oil analysis/lubrication program. To our surprise, we were somewhat lopsided (Figure 1), our strengths were consistent with Nuclear Culture, but our weaknesses were rooted in areas that cause the most impact to equipment reliability; Storage and Handling, Sampling Techniques, Contamination Control and Lubrication/Relubrication Practices all required improvements. We will explore these metrics in more detail below.

Storage and Handling:
Figure 2 documents our previous system of storage and handing in an uncontrolled environment. We had very little control of our lubricants cleanliness and its exposure to natural elements, as shown by this barrel of Mobil SHC 825 (expensive synthetic oil). You can also see a barrel of Fryquel behind this rust bucket. As an area of focus, Engineering and Maintenance developed a "get-well" plan, with an objective to ensure and implement as many cost-effective industry best practices in this area of storage and handling. Within a two-year period, we completely revamped our storage facility, but most importantly we established measures and protocol to eliminate lubricant cross-contamination, new oil pre-filtration, random new lubricant testing and secondary container upgrades. These initiatives nearly eliminated lubricant cross-contamination, with new oils having a particle count equal to or lower than ISO 15/12 for non-hydraulic oils and ISO 13/10 being a typical cleanliness for hydraulic oils. Both oil types typically maintain moisture levels below 80 ppm, as well. Above all, these efforts increased the site's overall awareness with respect to contamination control. Today, these accountability efforts have become an integral part of our team's standard of excellence.

Sampling Techniques:
If "Dead Men Tell No Tales," then dead samples do the same. In the past, our oil sampling process was not as telling and accurate at displaying abnormal conditions. Initially, we'd shut down the equipment to gather a sample that was not representative during its normal operation. This shutdown caused a decrease in wear particle density, breeding a change in suspension, thus altering the dynamics seen by the load supporting elements. Tight lipped, this dead sample from an inactive piece of equipment, withheld vital information for an accurate portray of the oil's true condition.

Today, the installation of live sample valves (Figure 3) has afforded our plant the accessibility to gather oil samples while the equipment is in service. This technique maximizes the data density of the segregated sample for condition-based analysis. The process enables our technicians to capture a real-time, highly repeatable snapshot of the oil without shutting down the equipment, minimizing sampling errors and disturbances to the component and at a fraction of the cost. This may all sound innovative, but how do you get your management on board for this 160-component renovation project? The following template can serve as an instructional tool when planning to implement this change process to your plant's lubrication program. Below, "At a Glance" provides a sample of the key elements that were essential to our successful implementation of the oil sample valve installation project. This 6-part reflection serves as one model of implementation used at the San Onofre Nuclear Generating Station. The cost analysis portion of the elements will be expanded below.

At a Glance:

• Cost Analysis
• Management Presentation
• Scope
• Feedback
• Work Process
• Part Set

Cost Analysis:
An initial cost analysis was performed based on the previous 5-year oil sample/change orders and was divided into two equipment classes: large components, 4.16 kV and small components, >480 V. The component pieces (266 valves and fittings) are estimated to cost $15 per installation for a one-time project cost of roughly $4,000. The estimated annual Maintenance man-hours of our older oil sampling process neared 3,500. Additionally, the cost associated with purchasing, consumption and disposal of our oil products was taken into consideration for this estimation. Miscellaneous costs: additional clean-up time, drain plug redrilling and tapping, and maintenance disposal time.

This estimate should be considered conservative. In today's work environment, workers will not be displaced by this reduction in man-hours. We've found that a savings of this magnitude directly affects emergent work orders and overtime costs.

Contamination Control
As part of the Pro-Active storage and handling initiatives implemented, tightening existing contamination control measures was the next step to ensure maximum equipment reliability. Desiccant Breathers were installed on components in harsh environments, as determined by microscopic evaluations indicating contaminant ingression. New age bearing isolators were installed under the similar premise. Additionally, new sets of filter carts were purchased to clean the Reactor Coolant Pump Motor oil during refueling outages. The goal was to design a cart that could be easily moved from motor to motor (4 total per unit) when filtration was complete and satisfactory, but more importantly, to have the ability to clean up the oil where radiation exposure was reduced and minimized. As a result, our cleanliness targets were met (ISO 17/14) in a quarter of the time as compared to the older bulky filtration units, further reducing radiation exposure significantly. Similar high efficiency filter units are used on our 56 hydraulic valve units.

Case Study: High Water Content and Microbial Growth within SA1307MT039 (Dirty Lube Oil Tank):

A sludge sample was collected from the Bulk oil centrifuge bowls during routine maintenance and the technician questioned the abnormally large amount of sludge. This questioning attitude prompted Engineering to obtain an oil sample from both the clean and dirty lube oil tanks. The analysis indicated that tremendous amounts of contamination, including high acidity, particulates and wear metals were present. The samples were taken from a drain directly bottom center of the tank and not from the normal oil sample located nearly 1 ft from the tank's bottom location. The clean lube oil storage tank did not indicate any abnormalities, while the dirty lube oil tank indicated nearly 100% contaminated water and sediment, with a high microbial population. It was later determined that approximately 2000 gallons of contaminated oil/water was present in the tank and the next lube oil sample wasn't scheduled for another 6-months. Both the main turbine lube oil reservoir and feed water lube oil reservoir are transferred to the dirty lube oil storage tanks during every refueling outage. Consequently, if these reservoirs were sent to the dirty lube oil tank, they would have become contaminated with micro-bacteria and high levels of water, potentially resulting in rapid oxidation and oil cooler fouling. This loss of heat transfer effectiveness would require swapping coolers, which is a highly risky evolution, potentially sending the unit off-line. Since the need to perform the required oil analysis was determined, an effective corrective action plan was put into place by draining the tank, cleaning the interior and performing the API 5-year tank integrity inspection early. As a result of this awareness, accountability, and commitment to excellence, lube oil samples are now taken on an annual basis and from the bottom of the tanks to avoid potential oil contamination during a refueling outage.

If this problem would have gone undetected, it could have ultimately required:

◦Replacement and disposal of approximately 30,000 gallons of GST 32
◦Cleaning of the lube oil reservoir and storage tanks
◦Chemically cleaning the oil coolers to an additional factor for being tripped off-line requiring several days to mitigate this problem.

The minimum cost as described above would be $186,000 for 30,000 gallons of GST 32, $13,000 for oil disposal cost, and based on a similar event on Unit 3 in 1997, chemical cleaning of the oil coolers, cleaning of the MLO tank, 3MT179, and support MOs, an additional 2,290 man-hours or $103,450 plus approximately $5,000 in supplies. It is estimated that the minimum potential cost associated with this save is $310,000 excluding engineering, operations and other support organizational cost, which is estimated to raise this total to between $400,000 and $500,000.

Lubrication/Relubrication Practices:
Mixing of lubricants, grease and fluid had become a problem onsite prior to several recent changes put barriers in place so the craft would not make this human performance error. Additionally, very little guidance had been given in the past with regard to electric motor greasing. Motors were failing due to over greased bearings. Based on these two (2) issues mentioned above, Engineering developed a specific lubrication locker procedure, SO23-I-1.55, which eliminated cross-contamination of lubricants and use of the incorrect lubricants in the plant. Also, below was the Engineering Methodology for greasing Electric Motors. This Methodology was applied to all critical and non-critical electric motors, which resulted in updating nearly 202 Repetitive Maintenance Orders and showed nearly 3500 man-hours savings (annualized). See below for the electric motor greasing methodology applied.

The purpose of this write-up was to provide Maintenance and Engineering guidance on greasing frequency, and the amount of grease required during PMs.

A site wide issue exists that pointed to motors being over greased further aiding in a less optimum motor/bearing life. The intent of the directions provided below will minimize the potential of over-greasing antifriction bearing. Any grease frequency, amount or type change must be reviewed and approved by Reliability Engineering.

Prerequisites:

• Determine that the bearing is a greasable bearing. Sealed bearings are not greasable, all others needed to be greased; open, single-shielded and doubleshielded. It should be noted that Y-type sealed bearing can accept grease but grease needs to be applied slowly.

• Prior to greasing, the fill plug/zerk fitting, drain plug and grease gun must be wiped clean using a lint-free rag.

• Remove the drain plug ensuring a purge path. If possible, the drain should be cleaned with a metal rod, removing any hardened grease.

• Prior to regreasing, ensure motor is at a normal stable temperature. This allows the grease to heat up, decreasing the greases viscous consistency further allowing the hot grease into the bearing more efficiently. If this step can not be met, it should be noted in the work plan as "motor greased cold" and brought to the attention of reliability engineering.

Post-Requisites:

• After greasing evolution is complete, run the motor until a stable operating temperature is reached. This typically occurs in about 1 to 2 hours. The drain plug should be removed so any excess grease is purged. If possible, the excess grease should be collected and visually inspected for wear debris, contamination and consistency.

• Before installing the drain plug, ensure it is clean and free of any debris.

Regreasing Intervals: For regreasing intervals, EPRI document NP-7502 (latest revision) "Electric Motor Predictive and Preventive Maintenance Guide" should be applied. Inputs needed to determine the proper greasing interval are as follows:

• Motor RPM
• Motor Horsepower
• Motor Load Configuration; belt driven or directly coupled
• Bearing Operating Temperature
• Motor Operation: continuous or standby/lay-up.

Note that greasing intervals for motors that are in standby/lay-up mode are typically 1.5 times the continuous regreasing interval.

Amount of Grease: Figure B-1 of EPRI document NP-7502 (latest revision) should be consulted. The amount of grease to be added to the bearing is correlated to the shaft diameter (or bearing Inner Diameter). Note: For motors in standby/lay-up modes, the amount of grease used should be divided by 2 (since the frequency of operation is at least 50% less).

Example: CEDM Motor/Fan Assembly:
The CEDM motors drive a dual stage fan unit at approximately 1800 rpm and supply approximately 40,000 SCFH airflow. The motor consist of 250 Hp, is directly driven and the bearing temperatures are assumed to be less than 140F based on the airflow supplied. The fan/motors are run continuously for approximately 50% of the outage. Further, the bearing type used is a double-shielded configuration. The motor shaft diameter is approximately 3 ½". Therefore, based on Table B-1 "Regreasing Intervals" indicates 24- 36 months multiplied by 2.0 (based on 50% demand) resulting in a regreasing interval of 48 to 72 months, or every 2 to 3 refueling outages. Lastly, based on the shaft diameter of approximately 3 ½" the grease PM should add approximately 2 ounces of grease. Typical grease guns dispense approximately 15 shots per ounce.

SONGS Today:
With the Oil Sampling and Analysis program at SONGS operating at a World Class level (Figure 4), the programs focus is continuous improvement. Development of Cost Avoidance Models, Auditing Vendor Lubrication Programs, on-site lubrication training development, and assisting others in the industry to implement an oil program similar to SONGS are several examples of the Oil Program's continuous improvement efforts underway.

For any additional information please contact Michael Bryson via email at Mike.Bryson@SCE.com›.

Presented at PdM-2008 Predictive Maintenance Technology Conference. www.maintenanceconference.com