Measuring Safeguard Effectiveness
By Kevin McManus, Chief Excellence Officer, Great Systems
After posting my article on evaluating safeguard effectiveness, I received several requests regarding HOW this can be accomplished. How are you measuring safeguard effectiveness?
Over the past few weeks, as I have thought about this question and explored it further in additional TapRooT® root cause analysis courses, it became quite clear to me that people in general are struggling to perform such evaluations. Fortunately, my national Baldrige Examiner background and my own experiences designing and improving measurement systems has provided me with some ideas that you can use for just this purpose.
Are Your Safeguards Processes or Target Guards?
Some safeguards, like tailboard meetings, audits, training, orientations, and the investigation effort itself are repeatable processes. All processes produce outcomes – we just need to find the vital signs of effective process performance. To what degree have you defined the desired measureable outcomes for each of your key process safeguards?
For example, it takes a certain amount of time to conduct a tailboard meeting. Whenever one spends time, he or she also spends money. Tailboard meeting attendees are customers of that process, so their satisfaction can be tracked. Finally, all processes contain some level of errors and/or defects that can be counted. These errors and defects can also be trended over time in rate form (defects or error per meeting).
Other safeguards, like personal protective equipment, equipment guards, and other types of ‘target guarding’ rely less on humans doing the right thing. This latter safeguard type is not a process, but we should have measures in place to gauge their effectiveness as well. As you know, humans can find a way to defeat their potential effectiveness. How often do the humans you supervise defeat their ‘target guards’? Have you tried trending such percentages over time?
Measuring Safeguard Effectiveness – ‘Target Guards’
I will start with the non-process, or target guarding, safeguards, as their effectiveness is relatively easier to measure than the process-based safeguards. Basically, your behavior-based observation / near miss / ‘at risk’ behavior monitoring processes should detecting those hopefully rare occasions when people bypass, or fail to use, target guards. Involve enough properly trained people in such auditing efforts, and you should be able to get a pretty good read on the relative effectiveness of these safeguard types. What is the likelihood of a given target guard being bypassed or compromised?
We can also look at the design of the target guards themselves. Best practice designs are out there and in use, whether we are aware of them or not. The picture attached to this post takes a humorous look at different ways to protect one’s head, but the effectiveness spectrum from turtle to helmet is still evident. Similarly, there are multiple ways to barricade an open hole, protect one’s hands, and provide effective fall restraint. How well designed are your target guards? To what degree are you using best in class target guards?
Measuring Safeguard Effectiveness – Process-Based Safeguards
The process-based safeguards are not that much harder to measure. The problem is that we are simply not that familiar with measuring the effectiveness of management-level work systems. All too often, we fail to even see these repeatable efforts as processes that can be measured with each cycle that occurs. Fortunately, there are three simple ways to regularly monitor these repeatable types of safeguards:
- Ask your internal and external customers to evaluate safeguard effectiveness via survey
- Trend the cost and time ratios for each safeguard process
- Capture and trend major defects and errors that occur with each process cycle
Ask your internal and external customers via survey
This is the best way to gauge process safeguard effectiveness IF your culture is open and honest enough. Surveys can also be used to gauge target guard effectiveness. High performing companies survey both customer types on at least an annual basis to help determine how well their key management systems, including their different key safeguards, are working. How do you measure key work system effectiveness?
For example, it is not uncommon for one of the annual employee survey statements to focus on the effectiveness of regular workgroup meetings or training sessions that are provided. With today’s technology, it is easy to also do spot check surveys. Short 3-5 question ‘pop up box’ surveys can be used that each employee replies to when they log onto the system for their messages each day. We also always have the option of using ‘old school’ paper assessments (as with the example survey graphic).
Trend the cost and time ratios for each process
All processes expend time and money as they are cycled, whether we measure such values or not. In turn, the costs and times per cycle in terms of calendar hours, or cumulative people hours, can be calculated and trended over time.
I am amazed that more managers are not required to use lean six sigma methodologies to reduce these time and cost ratios given the enormous costs associated with team meetings, training, and town hall meetings. All work is a process, and all processes have an owner and vital signs. Why isn’t every process owner manager working to optimize the measures and processes they own?
Capture and trend major errors that occur with each process cycle
Do you track meeting defects for each meeting? Do you look at how often a leader fails to engage the team effectively when they talk to them prior to work? All processes should have a defined set of steps and specifications for effective execution. When we deviate from such standards, a defect occurs. What are the key performance expectations that process owners must manage as they execute their process-based safeguards on a daily, weekly, or monthly basis?
All processes produce both outcome and in-process measures. The problem is we often don’t capture and trend them. Simply tracking the number of process cycles is not enough. Cost, customer satisfaction, and quality measures are needed to understand the overall effectiveness of a given cycle. When it comes to measuring a process, we must start somewhere. The above three areas, coupled with general ‘at risk’ behavior tracking with good findings/day flows, are great places to start.
How are you measuring safeguard effectiveness? Keep improving!
If you want some more great ideas on measuring safeguard effectiveness, please check out my new book “Vital Signs – the Power of Meaningful Measurement.”