Stop Starting with

Smiles and Begin with Business

Outcomes: Results-Based

Integrated Design by Betsy Allen

Repeated studies over the past seven decades have challenged the impact of training. Concepts trekking from the classroom to the job range from a tenuous 10% to a flighty 50%. Surprise,surprise; when tragedy hits; training's budget is one of the first to go! Yet, knowing that competitive advantage is dependent on human productivity, innovation and pursuit of excellence, there must be a better way. There is.

The American Society for Training and Development's 2003 State of the Industry Report (ASTD) revealed that three challenges for the performance improvement industry include:

controlling costs, evaluating training and understanding return on investment (ROI).

The integrated model described here addresses this serious need and applies an inductively logical, visible, repeatable process to design training with the business need in mind.

Over 85% of benchmark organizations surveyed by ASTD in its 2002 State of the Industry Report, evaluate training with Feel Good Smile Sheets (feedback forms distributed to participants at the end of training). What do most Smile Sheets do? They evaluate how well the participant felt about the instructor. How well they bonded. Not, whether the trainee will change behavior.

You want behavior change? Escape from this mediocre majority; read this article and take steps towards realizing breakthrough performance.

One of the foremost authorities on training assessment is Donald Kirkpatrick1. He introduced his assessment model in 1968 and that model remains the standard for talking about training evaluation. In simple form, Kirkpatrick's model has the following four levels:

Level One Did they like it?

Level Two Did they learn it?

Level Three Did they use it?

Level Four Did they make a difference?

Level One - Did they like it?

The Smile Sheets mentioned earlier would be Level One evaluation in Kirkpatrick's model.  While the real purpose of Level One is tracking customer satisfaction to make it right this time (when possible) and to improve it next time. Most Level One users stop with this as the only evaluation of the training. The participant's immediate assessment of how well they liked the training, the instructor, and the content, is a limited, biased measure.

Level Two - Did they learn it?

Just because you said it, doesn't mean they learned it. Assessing the degree of learning takes us to the second level. In most cases, this shows up as a pencil and paper test. This testing approach is used in less than 25% of corporate training programs according to ASTD.

Originating in the traditional school system, this basic theory is also flawed and biased to reward good testers, not good transfer. It assumes that if I can recall cognitively what was presented in the course, I'll understand it's importance and will apply the learning on the job.

In an article titled "One-Size-Fits-All Education Doesn't Work2," educator and author Alfie Kohn says: "It's easier to quantify how many semicolons are used correctly in an essay than how many wonderful ideas it contains. Those who make a fetish of specific, measurable standards end up dumbing down the learning." Likewise, Linda McNeil, Rice University says, "measurable outcomes may be the least significant result of learning," which supports this view.

You see paper and pencil tests tell us nothing of higher order critical and problem solving skills essential to on-the-spot workplace application.

Abandon Level Two or Revise? Revise it. Move Level Two evaluation away from pencil and paper test and create authentic (real world) forms of assessment. Content learned and tested in context allows us to test for learning, not just to pass. This newer approach is based on the belief that it is more important for the students to know what they know, feel successful in the training, andbe able to apply what they have learned than it is to have a "cognitive score." For example. there is not a doctor on the planet who will give a pharmaceutical representative a multiple choice test.

Yet, most will verbally challenge the rep and expect rapid responses. So assess the rep verbally. How well can they spontaneously articulate study results which match the doctor's needs.

When Level Two assessments are designed with processes to observe the actual application of the new knowledge, Voila! You get a realistic view of potential transfer of the desired change from the learned place to the workplace.

These approaches do not usually result in a score, although they can. The more standard form and format are observation forms and self-assessment instruments that provide feedback on a continuum of skills. Position it as learning for life, not just to pass. Supporting the concept that the development of skills is an ongoing process is essential. We want people to know where they are now and how to move on the continuum. Provide managers with this type of assessment after a training program and they will have a clear indication of how to support the employee in their continued development.

Level Three - Did they use it?

Unfortunately, less than 10% of corporate training is evaluated at this level. There are two common approaches to Level Three evaluations. One is through the use of participant surveys (after the class) that ask them the degree to which they are using the information from the training course (weak, yet it is evidence). The second approach is through observation of participants after the training. Both of these approaches will provide helpful information - the differences between the two approaches are reliability of the data and the costs.

Participant surveys depend solely on self-reported data, thus the design of the survey instrument is very important. The instrument has to ask the questions and provide answer options that are clear in terms of the behavior that you seek to evaluate. If participants are asked to rank how often they use the behavior, define the rankings clearly. While dependent on truthful reporting, this method is much more cost effective than interviews and/or observations.

Kick it up a notch with random observation checklists (yes or no) by the managers and watch the degree of buy-in with respect to the impact of the training grow.

Level Four - Did it make a difference?

The ultimate assessment in Kirkpatrick's model attempts to assess the difference the training made in terms of the business or organizational outcomes. As an example, a company might have surveyed their customers and found that the level of customer satisfaction was 75 %.

Leadership wants more. A number of strategies are put in place to increase the satisfaction.

Training of customer service representatives was one of the strategies. A subsequent survey of customer satisfaction found that the measure had moved from 75% to 92%. Did training make a difference?

Kirkpatrick points out that it is very difficult to determine the degree to which any one strategy is responsible for the movement of such a metric in a complex system. This might be why less than 5% of the companies employ Level Four measures. He suggests that training professionals should look for evidence that the training had impact on the business rather than trying to prove it. The costs of Level Four evaluation strategies, where proof is the target, are

significant because of the complexity of multiple influencers.

One powerful aspect of Level Four is that it calls for the training team to clarify the business outcome with the stakeholders before training design. Most corporate training is designed and delivered without any consideration of a business measure. In those cases, it is most likely that the intervention won't meet the potential impact on the business.

A Results-Based Integrated Approach

A comprehensive and integrated design approach provides the best results. This approach starts with Level Four and works in reverse through the levels.

The first step in the approach to a Result-Based Design (RBD) is to really understand the business problem or opportunity that the training is to address. Where's the gap between the performance we're getting and the performance we need to reach the business goals? Is there also an internal or external environmental issue which must be addressed for the behavior to change? Define the clear picture of the business purpose. Define the measure. Then you can identify the metrics to monitor impact. You can isolate behavior gaps. You can design the right content for the right people at the right time.

Most importantly, you can identify the behavior that you want and that behavioral change can also be designed into the training. Baseline data can be measured before and after training.

An "after training" Level Three survey or observational checklist can ask specific questions with clearly identified behavioral descriptors to collect clean data.

Use self-assessment instruments at the earliest possible moment to accelerate learning. From that Level Two assessment, participants can look at the desired outcomes and determine if they are performing as expected. The training team and peers can also observe and provide feedback. The formula is simple: Participants know the business purpose. Participants get to practice behaviors. Participants are motivated. When you have a motivated learner, let them show what they know and practice applying the new knowledge in a way that is meaningful.

Our job is to create a learning environment where the participants can be successful. If the training is longer than an hour then gather Level One participant feedback to modify the training so participants being successful.

RBD enables trainers to adjust the training as it is delivered and make participants as successful as possible. This gives the participants the feedback they need to stay focused on the right things and direct their own learning. The "after training" data will be available to determine if the training is being applied and if it is producing the desired results. With the combination of data, ROI can be estimated. Stop starting with smiles and begin with business outcomes instead.

1 Donald Kirkpatrick, Evaluating Training Programs, Berrett-Koehler Publishers, 1998. 

2 Alfie Kohn, "One-Size-Fits-All Education Doesn't Work," The Boston Globe, June 10, 2001.

Copyright ©2007, The Bob Pike Group. All rights reserved. • www.BobPikeGroup.com

The Bob Pike Group

14530 Martin Drive

Eden Prairie, MN 55344

(952) 829-1954 or (800) 383-9210

www.bobpikegroup.com