How do you measure the value of training?

So how do you measure the value of training? Ideally of course we would like to see the impact on the business, what is the return on investment in training.

In practice that is challenging: imagine investing 20k euro in a training session for a class full of PMs who run tens or hundreds of millions worth in projects. Even if only one every three of them saves a one-digit percentage point off their budget in avoided or well managed risks, the ROI is huge… but hard to measure. How can we know that the success was actually due to the class? Could it have happened anyway? Maybe it is due to new tools which are being used, or simply thanks to a better choice of PM. How does one measure the impact of a successfully managed risk anyway? Since we don’t have the alternative event to compare against?

A commonly talked about (and less commonly used) approach is the Kirkpatrick model. ”The Kirkpatrick model defines four levels of evaluation – Reaction, Learning, Application and Impact, with level 1 evaluation commonly referred to within the training industry as Happy Sheet” (for a related interesting paper by Elearnity (UK) click here).

However, there is a different approach that we may choose to take. Training is a support activity to change. If there is no change expected from the participants, then there is no need to send them to a training session. So an important question to ask, when evaluating the success of investment in training, is: “Has the mindset shifted?”

Following up on mindset shifts is crucial in the development of social and political movements, so I was curious of what they think about it at the Monitor Institute (part of Deloitte Consulting). They suggest a set of “metrics for every aspect of movement-building.” Then, they create categories/areas (community organizing, engagement, leadership, alliance building, campaigns, communications and framing, media…). “In each area they suggest telling a story of progress using two types of metrics: transactional and transformational. They define them as follows:

Transactions involve the quantifiable markers both internal (e.g., how much funding, how many members, etc.) and external to the organization (e.g., voter turnout, policies passed, etc). While the data is not always easy to collect (especially with transient or mobile groups), such measures tend to be easier to track because they are more tangible. But transactions only tell part of the story and tend to skip over the richness of experience and momentum that can be precursors to big change.

Transformations, on the other hand, are the vital but sometimes “invisible” work. They show how people, organizations, and movements have been altered through the collective efforts. Taking the transformation further, they can show how societal and political views have shifted or been impacted by movement building” (see more here).

So if we apply this logic to the mindset shift which we are fostering within a corporation, we could redefine the transactional and the transformational criteria (T&T), and see if we can use them to find out if training is actually enabling the expected mindset shift. Let’s give it a try:

Assume we want to shift the corporate mindset from the existing ad-hoc project management approach to a structured PMI style approach, and that we have set up a program of classes through which our PMs are all going.

Step 1: Create categories/areas:

These could be: management support, engagement… (Or we could even use the Kirkpatrick model: Reaction, Learning, Application and Impact).

Step 2: Tell a story in each area using T&T:

• Transactional:
  • Management Support: trend of approved funding for classes, certifications, PMI membership, and PM community building activities.
  • Engagement: number of participants in class, number of PMP certified project managers…
• Transformational:
  • What has changed?
  • Are PMs behaving differently?
  • Which benchmarks are being used?
  • Are best practices shared?

However we look at it, it is not going to be easy.

There is a trap in which we risk falling. “Happy Sheets”, or feedback forms, tend to be as easy to retrieve, track, and draw nice little graphs from, as they are of limited use. Limited to the “customer satisfaction” aspects, but telling us nothing about the transformative impact of the training efforts, and certainly nothing about the ROI.

Any thoughts? Ideas?

Cheers!

Max