Journey Toward Reliability Starts with Uptime Elements

Central Contra Costa Sanitary District (Central San) is an organization providing wastewater collection and treatment for over 481,600 residents and 3,000 businesses in Contra Costa County, California, located about 25 miles from San Francisco. Central San’s main facility is a treatment plant in Martinez, California, processing an average of 32 million gallons per day of municipal wastewater and providing up to three million gallons per day of recycled water to several businesses in the community.

In 2014, Central San started formalizing an asset management policy, then developing an asset management implementation plan. The plan was developed to align with the vision, mission and values of Central San, and translated to a strategic plan with goals that create the path to success.

Figure 1: Asset hierarchy

The first step toward improving reliability was to review and improve the data stored in the computerized maintenance management system (CMMS). The information in records was gathered to assign install dates, service life, asset cost, work area and asset type. To facilitate characterization of the assets and the plant’s processes, a process-based asset hierarchy was developed, as shown in Figure 1.

Attributes critical for improving the reliability program included installation date; warranty data; standard nameplate data; condition score; consequence of failure (COF); likelihood of failure (LOF); and business risk exposure (BRE).

Data integrity is vitally important, therefore, Central San added components to the CMMS, such as a color-coded quality assurance/quality control (QA/QC) panel, which summarizes the status of key areas of the work order that must be completed. Also added were fields for regulatory and safety compliance to identify the work order as such. A third improvement was adding a field for planner or geographic information system (GIS) updates. When these boxes are marked, the CMMS sends an e-mail to the appropriate group identifying changes for things, such as to update the standard operating procedures (SOP), the asset attribute, or remove and replace assets.

Concurrently with the CMMS implementation, and what would become the cornerstone of the maintenance strategy, Central San piloted a reliability-centered maintenance (RCM) approach. The pilot was led by Anthony “Mac” Smith, a pioneer in RCM. The maintenance division’s goal was to formulate a systematic approach to developing a comprehensive maintenance program based on asset criticality and consequence of failure. As was found through the pilot, RCM is a qualitative decision methodology that identifies the most effective preventative maintenance (PM) task for treatment plant equipment and systems. Additionally, RCM with a CMMS establishes a repeatable program with documented processes and procedures.

Prior to jumping into an analysis, the maintenance team spent a week with the consultant and his company discussing Central San’s current maintenance practices and issues, then learning classical RCM methodology. The discussion covered details of the seven step RCM systems analysis process and case histories. Included was the strategy for the deployment and use of RCM, experienced-centered maintenance (ECM) analysis, defect elimination (DE) and root cause analysis (RCA).

Figure 2: Reliability strategic framework

Plant maintenance developed the strategic framework, shown in Figure 2, which would become the road map of Central San’s efforts to optimize its maintenance program. After completing the RCM analysis, the organization piloted an experienced-centered maintenance (ECM) analysis on the primary process, ultraviolet (UV) system and boiler feedwater system, all with successful results.

The strategic framework identifies Central San’s approach moving forward to perform RCM analysis on the 20 percent of assets that are “eating our lunch,” optimize the PM program for the better behaved system (i.e., the 80 percent of assets) by performing an ECM analysis, and perform DE as the situation or opportunities arise. Also, at any time, or as needed for new or complex tasks, perform an internal audit for safety, quality and effectiveness (SQE) of the tasks. And finally, perform RCA on regulatory, safety, or high impact operational failures.

Figure 3: Top, smooth transition eliminates buildup points shown in bottom image

Identifying ways equipment can fail was well understood after completing the RCM pilots, so another proactive way to eliminate them was created. To address failures caused by the five sources identified by the Uptime® Elements, a DE program was developed. This program would be used across the board on both well behaved and bad actors, as needed. Potential triggers for defective elimination were discussed and used, such as failure before the asset’s useful life, repeat failures showing up on the bad actors list, when there’s more than one reactive work order with a priority of “1” or “2” on a specific asset between PM tasks, or simply as an optimization idea.

With an already strong PM program running, a logical stepping point was to expand the use of condition-based maintenance (CBM) and predictive maintenance (PdM) to complement the existing PM program by incorporating asset condition monitoring (ACM). It is fundamental to RCM to apply the most applicable and effective task that most often leads to the least intrusive choice of tasks. Existing CBM tasks were updated and new PdM methods were identified and implemented.

Another crucial element identified was for the maintenance staff to be actively engaged in all phases of the assets’ lifecycles to assure equipment maintainability and reliability. It became extremely important for maintenance to engage in the planning, design and construction phases of each of Central San’s capital projects, including participation with the selection of the design teams.

A major component of the new involvement with capital projects was a comprehensive asset handoff protocol to ensure assets and their documentation were properly transferred from the capital projects division to the relevant team once installation work was completed. This defined what information is reviewed and passed onto operations and maintenance (O&M), as well as when during the project phases. Some of the changes included detailed workflows, which identified key timelines for the development of asset nameplate data, PM tasks, spares, and reviewing training materials and agendas. The protocol also lent itself to using failure mode and effects analysis (FMEA) early on when staff is engaged in planning and design talks. Also key to the capital project review process was the use of 3-D modeling in planning and design. This helps with ensuring new equipment is serviceable and aids in identifying any obstacles by being able to virtually walk through the layout. This provides visualization of accessing and working on the equipment.

Data and document management was paramount to these efforts. Engineers, operators and maintenance staff needed to readily access information to work effectively and efficiently. A geoportal allows one to easily navigate back and forth between other databases and the CMMS to find assets and other important asset information.

Figure 4: Work order history

Figure 5: Asset Information

Figure 6: Project documents

• Work order history (Figure 4) allows access to history in the CMMS for both the current and previous system.
• Asset management information (Figure 5) includes records, such as job plans, condition assessments, O&M documents, etc.
• Project documents (Figure 6) list all project work done on the asset and includes associated documents.

This has proven to be an invaluable resource for many work groups as it compiles information from all the different databases and software systems Central San uses to allow easy access to virtually all the information associated with any given asset.

It is well understood that staff members are the key to success. To meet the challenges when it comes to recruiting and retaining skilled technicians who can maintain an industrial complex, the following steps were taken:

• The succession planning committee developed an action plan for each division.
• Training and development checklists were produced to identify knowledge, skills and abilities needed by staff for each craft shop. Each training and development checklist discusses expectations of the job, outlines cross-functioning, broad-based training, includes a development road map and provides a tracking document.
• Internal programs and resources were developed, which included a supervisor academy, mentorship program, leadership academy and several other development opportunities.

Figure 7: Repeat failures tracked in CMMS

Figure 8: Query report allows for easy review of any asset

Figure 9: Asset condition report gives latest values for COF, LOF, BRE, cost to date charged to asset and other important information 

Access to information was crucial for decision-making during implementation of the asset management program and it continues to be an extremely valuable resource in Central San’s day-to-day operations and maintenance. A variety of reports (Figures 7-9) are accessible through the CMMS and geoportal. The report list can be generated using a custom selected parameter.

One report is the query, shown in Figure 8. The query report provides a quick review of any asset for spare units, spare parts, PM tasks, PdM tasks, and if the asset ever had an RCM, PM optimization (PMO), RCA, or other type of assessment performed on it. It is a powerful tool that helps Central San continuously improve its maintenance program. The report has dynamic links to the associated documents.

Workflow monitoring can be done by supervisors and technicians in a variety of “in-boxes,” which can be configured to display on the user’s home screen in the CMMS.

Because of these and other initiatives, Central San has made tremendous gains in operational reliability and efficiency, which was recognized by Uptime magazine with the 2018 Best Reliability Engineering for Maintenance Program award. All objectives continue to be in support of and in alignment with the organization’s principal aim of its mission and vision: “To protect public health and the environment” and “to be a high performance organization that provides exceptional customer service and regulatory compliance at responsible rates.”