Measuring and Evaluating Workplace Safeguard Effectiveness

Evaluating Safeguard Effectiveness versus Safeguard Number

Recently, I worked with a company that had an impressive set of safety systems in place – fourteen to be exact. On a daily, weekly, and/or monthly basis, they use a variety of audits, inspections, briefings, work plans and permits, JHAs and FLRAs, orientations and meetings, and training in an effort to minimize goofs. Sounds pretty impressive, doesn’t it?

Unfortunately, evaluating safeguard effectiveness was not a focus at this location. The number of times each safeguard was used was being tracked. Measures for evaluating safeguard effectiveness had not been defined. Big problems were being captured, and some daily near miss errors were being snagged.

However, safeguard strengths and opportunities for improvement were not evident. Limited opportunities for improvement or learning could be identified. How much value does each type of safeguard REALLY return?

How do you measure the effectiveness of your key safeguards? For example, how effectively does each of your supervisors spend their thirty minutes or so of job preparation time each day? How effectively does each employee spend his or her twenty hours of computer-based learning each year?

How effective is the 40 or so hours you invest in process audits annually? Many people have not thought about how to evaluate their safeguards in this manner, in spite of the monies we invest.

EXPLORE MORE: How Effective is Your Process Improvement Work System?

Are Your Safeguards Process or Target Guards?

Some safeguards, like tailboard meetings, audits, training, orientations, and the investigation effort itself are repeatable processes. All processes produce outcomes. We just need to find the vital signs of effective process performance. To what degree can you define the outcomes for each of your key process safeguards that you desire?

For example, it takes a certain amount of time to conduct a tailboard meeting. Whenever one spends time, he or she also spends money. Tailboard meeting attendees are customers of that process. In turn, you can track their satisfaction. Finally, all processes contain some level of errors and/or defects that you can count. Plus, you can trend these errors and defects over time in rate form (defects or error per meeting).

Other safeguards, like personal protective equipment, equipment guards, and other types of ‘target guarding’ rely less on humans doing the right thing. This latter safeguard type is not a process. However, we should have measures in place to gauge their effectiveness as well. As you know, humans can find a way to defeat their potential effectiveness. How often do the humans you supervise defeat their ‘target guards’? Have you tried to trend such percentages over time?

Measuring Safeguard Effectiveness – ‘Target Guards’

I will start with the non-process, or target guarding, safeguards, as their effectiveness is relatively easier to measure than the process-based safeguards. Basically, your behavior-based observation / near miss / ‘at risk’ behavior monitoring processes should detecting those hopefully rare occasions when people bypass, or fail to use, target guards. Involve enough properly trained people in such auditing efforts, and you should be able to get a pretty good read on the relative effectiveness of these safeguard types. What is the likelihood of a given target guard being bypassed or compromised?

We can also look at the design of the target guards themselves. Best practice designs are out there and in use, whether we are aware of them or not. This post’s picture takes a humorous look at different ways to protect one’s head, but the effectiveness spectrum from turtle to helmet is still evident. Similarly, there are multiple ways to barricade an open hole, protect one’s hands, and provide effective fall restraint. How well do you design your target guards? To what degree do you use best in class target guards?

Three Ways to Measure Process-Based Safeguard Effectiveness

The process-based safeguards are not that much harder to measure. The problem is that we are simply not that familiar with how to measure the effectiveness of management-level work systems. All too often, we fail to even see these repeatable efforts as processes that we can measure with each cycle that occurs. Fortunately, there are three simple ways to regularly monitor these repeatable types of safeguards:

• Ask your internal and external customers to evaluate safeguard effectiveness via survey
• Trend the cost and time ratios for each safeguard process
• Capture and trend major defects and errors that occur with each process cycle

Measurement Option #1: Ask your internal and external customers via survey

This is the best way to gauge process safeguard effectiveness IF your culture is open and honest enough. Surveys can also be used to gauge target guard effectiveness. High performance companies survey both customer types on at least an annual basis to help determine how well their key management systems, including their different key safeguards, are working. How do you measure key work system effectiveness?

For example, it is not uncommon for one of the annual employee survey statements to focus on the effectiveness of regular workgroup meetings or training sessions that take place. With today’s technology, it is easy to also do spot check surveys. You can ask each employee to reply to a short 3-5 question ‘pop up box’ survey each time they log onto the system for their messages each day. We also always have the option to use ‘old school’ paper assessments.

EXPLORE MORE: How to Measure and Improve Your Customer Satisfaction Work System

Measurement Option #2: Trend the cost and time ratios for each process

All processes expend time and money as they cycle, whether we measure such values or not. In turn, we can calculate and trend the costs and times per cycle in terms of calendar hours, or cumulative people hours, over time.

I am amazed that more managers are not required to use lean six sigma methodologies to reduce these time and cost ratios given the enormous costs associated with team meetings, training, and town hall meetings. All work is a process, and all processes have an owner and vital signs. Why isn’t every process owner manager working to optimize the measures and processes they own?

DISCOVER MORE: How to Measure and Improve Your Measurement Work System

Measurement Option #3: Capture and trend major errors that occur with each process cycle

Do you track meeting defects for each meeting? Do you look at how often a leader fails to engage the team effectively when they talk to them prior to work? All processes should have a defined set of steps and specifications for effective execution. When we deviate from such standards, a defect occurs. What are the key performance expectations that process owners must manage as they execute their process-based safeguards on a daily, weekly, or monthly basis?

All processes produce both outcome and in-process measures. The problem is we often don’t capture and trend them. It is not enough to simply track the number of process cycles. One needs cost, customer satisfaction, and quality measures to understand the overall effectiveness of a given cycle. When it comes to process measurement, we must start somewhere.  The above three areas, coupled with general ‘at risk’ behavior tracking with good findings/day flows, are great places to start.

Evaluating Safeguard Effectiveness Reduces Human Error Rates

A strong relationship exists between safeguard effectiveness and human error rates exists. In fact, you should explore the relative strength of your existing safeguards. You might find some patterns and systemic reasons that help explain why certain problems keep coming back. Effective safeguards reduce human error rates!

We often fall into an organizational trance where we think that our investments in rule enforcement, training, process improvement, and leadership development keep us error free. A lack of tracking daily errors does not mean that one is error free however. Where is the proof? Which safeguards provide more value than others? Which safeguards need improvement?

If we don’t measure daily process errors, how do we really know? Safeguard absence and failure leads to process errors, but few process owners track such errors. We track and analyze the big problems, but we pay little attention to the small, daily errors that are the precursors to the big ones. This is one key way to find failed safeguards. Go out and look for them! One might argue that this is the purpose of audits, and I would agree. Sampling is not enough if we want to stop daily goofs for good.

How Error Free Do You Truly Want to Be?

One can evaluate safeguard effectiveness via surveys of many types, criteria-based observation ala Toastmasters, and sound root cause analysis of human error. Remember, most people do not measure the effectiveness of their existing safeguards. How much do you resemble the company that uses fourteen different safeguards to keep people safe, but doesn’t measure the effectiveness of any of them directly?

How error-free do you really want to be? Could you become even more error free, at a lower cost, if you improve the effectiveness of your existing safeguards?