How to Measure Training Effectiveness Podcast

This week’s Real Life Work podcast looks at how to use ten training effectiveness measures to measure training effectiveness.

Top Ten Workplace Training Effectiveness Measures

Often, even the best organizations struggle to effectively measure workplace training. For example, multiple Baldrige recipients in recent years continue to rely on course participation and completion rates as primary training effectiveness indicators.

From an overall results and work systems design perspective, these companies are high performers. Why don’t they measure training differently?

Some do. They simply do not want to use their limited application space to show results they can more effectively, and completely, share on site.

However, only a small percentage of workplaces can use such an excuse. Most don’t measure training effectiveness at all. Some struggle to provide evidence of any formal training.

Organizations often measure training cost, training hours, and completion rates. Unfortunately, they rarely combine these metrics for the purpose of work systems effectiveness review.

For example, how do training investments affect process error, equipment failure, and workplace injury rates? How do we choose the best training investment if we need performance gains in a specific area?

In this podcast, I share a progressive list of possible training effectiveness metrics. Keep in mind those at the top of the list may seem familiar, but they are also the weakest gauges.

Measure #1: Total Training Cost and Time

Because we budget training time as an expense, most organizations can show you how much time and money they invest annually in training. Plus, they may divide this total by the total number of staff to give you a training cost or training time per staff member ratio.

My concern relates to HOW we prove these investments are worth it. There are many different ways to develop job skills these days. Often, low cost, simple methods provide much greater return than fancy classes, speakers, and handouts.

Measure #2: Participation Rates

Participation rates look at how many people attend the training versus who was supposed to be there. For this metric to work, one must have a list of courses each job position requires.

Ideally, a competency matrix dataset exists to define course requirements for each job role. If it does, use the course-specific participation rate data to create competency participation rate index.

Measure #3: Completion Rates

Completion rates are a step above participation rates in that they show who finished a course. Yes, there are cases where many more people attempt certain courses versus complete them.

Additionally, good course designs require course work outside of the classroom. In turn, such courses are not complete until the work outside of the classroom is done.

Measure #4: Smile Sheets

Smile sheets are another term for end of course surveys. Donald Kirkpatrick refers to this ‘reaction’ metric as a Level One training effectiveness evaluator.

As there are four levels in Kirkpatrick’s levels of training effectiveness, this means end-of-course surveys are something, but not good enough.

Also, it is common to include one or more statements with a training focus on the periodic culture survey most companies conduct. One common example is ‘I receive the training I need to do my job effectively.” It is also common to inquire about the general effectiveness of training people attend.

With the time that passes since the training, it is likely these ‘learning’ indicators are superior to smile sheet results.

Measure #5: End-of-Course Tests

This is the classic formal education measure. Did you pass the test? To evaluate training, we can give such tests at the end of an entire course or course segment.

The effectiveness of this measure largely depends on test design. For example, a ‘hands on’ skill demonstration test is a more effective indicator of skill acquisition than a simple True – False test.

Measure #6: Post-Training Position Observation Checklist (POC) Scores

Now, we move into the above average training effectiveness metrics. The Position Observation Checklist (POC) is a best practice I learned from national Malcolm Baldrige Award recipient Pal’s Sudden Service many years ago. Since then, it has served both me and my customers well at multiple locations.

Both the trainee and their supervisor use the Position Observation Checklist (POC) to assess skills on the job post-training. A typical checklist has five areas of performance, with an average of five skills per area. Each person evaluates skill application on a 100 point scale for each area.

This tool serves as a Level Three ‘behavior’ training effectiveness indicator. CLICK HERE to download an example Incident Investigator Position Observation Checklist!

Measure #7: 90-day Post-Training POC Score

This ‘behavior’ measure is more effective than the previous one simply because it demonstrates longer skill retention and effective application. Typically, both checklist assessments are used.

Comparison made between the two assessments help show which skills people tend to retain better than others over time.

At this point in training work system design and measurement maturity, it is likely that a certificate process exists to help integrate different skill sets, needs, and training results. Both end-of-course and 90-day assessments of a skill demonstration nature are common with such certifications.

Measure #8: Reduction in Process Errors and Waste

The final three types of training effectiveness measures reflect correlation between the training and work process performance. In a proactive process improvement system, each process owner captures key errors and failures daily at the process level.

In turn, they can connect daily process results with the staff who support that process.

Begin with a simple ratio that looks at average POC scores (post-course and 90-day) versus daily process error and failure rates. For the best results, track errors at the process non-conformance level.

Process-specific scrap and rework reduction rates also work. Avoid the use of lagging, aggregate measures (such as the site safety recordable rate).

For equipment failures, look for downtime duration and frequency relationships, in addition to the non-conformance that results in process downtime. Avoid the use of downtime percentages in favor of downtime minutes and frequencies.

Measure #9: Improved Workforce Engagement Scores and Retention Rates

An additional level of results correlation exists when effective training satisfies key workforce engagement factors. Such satisfaction helps reduce turnover and increase staff retention.

Plus, the reduction of daily errors and failures in the workplace creates a more positive attitude towards work in general.

If you are fortunate enough to have datasets that allow it, look for a variety of correlations between the different inputs and outcomes I present. The design of your annual work culture survey should help support training effectiveness data capture.

Plus, try to find ways to practice lean data capture for your training completion information. For example, QR code scans – one from the trainee’s security badge and one code from the post-course test – provide a quick way to update a training database.

POPULAR CONTENT: How to Increase Workforce Engagement

Measure #10: A Training Work System Effectiveness Index

The ultimate gauge of training system effectiveness is an index that constitutes multiple effectiveness measures. Plus, the use of such an index allows results ‘credit’ to be given for foundational course completion efforts. A typical index would include metrics from essentially each option in this list.

As with all process measurement, it begins with effective dataset construction. What training data do you need to capture for each course someone attends? What type of training profile should you build for each staff member?

How will you set up a similar course effectiveness profile for each course? How do you want to progressively improve your approach to training effectiveness measurement over time?

Would You Like to Kaizen Your Training Work System?

Effective training is a passion of mine. I try to improve both training content and delivery with every course I teach.

When I serve in a training capacity, my ultimate goal is sustained, and effective, skill application on the job. What are your goals as a trainer?

Please take a look at my ‘Teach Your Teams’ performance improvement workbooks. I created each one over time as my own work demands drove the need.

Most of my past employers did not have huge (or much of any) training budgets. Often, they had tight profit margins and smaller businesses.

In turn, I did my own research and created my own ‘real life work’ training products. Currently, you can find full video and audio content for three of these workbooks on my Great Systems YouTube channel.