Learning Measurement Part 1
For many businesses, our behaviors as consumers—our clicks, our scans, our searches—drive decisions about how, where, and when to place certain information in front of us. And that placement, determined by our own behaviors, can result in higher sales and greater revenue per advertising dollar for many of those businesses. Data and the analytics horsepower that yield these kind of insights are quickly becoming table stakes for the way most organizations interact with and engage with customers. A business strategy without a data strategy is often an indicator of a company’s naiveté,1 and many are becoming data-driven businesses. Why, then, do learning leaders often continue to struggle with the age-old battle of evaluating the value of learning to the business? Why do they continue to find it so difficult to justify increased investment? The answer: They are thinking about the problem all wrong.
In fact, BersinTM, Deloitte Consulting LLP’s State of Learning Measurement report showed 95 percent of L&D organizations surveyed do not excel at using data to align learning with the business, run the L&D function, or increase the effectiveness of learning methods.2 Furthermore, 22 percent of L&D organizations surveyed rarely or never track progress towards strategic initiatives,3 meaning some enterprises are not even looking at the numbers. This is a problem.
In what Thomas Friedman has called the “age of accelerations,” business leaders are desperate to keep the business sharp and attuned — to continuously grow and evolve just as the world around them does. Learning has never been more essential to enabling businesses to compete.4 As Friedman states in his interview with Cathy Engelbert and John Hagel, “…the future of companies is to be hiring people and constantly training people to be prepared for a job that has not been invented yet. If you, as a company, are not providing both the resources and the opportunity for lifelong learning, [you’re sunk].”5 In other words, as the world around us changes faster, enabling relevant and constant learning opportunities will be a critical differentiator.
As learning moves from formal experiences, like e-learning courses and in-person training, to on-demand learning embedded in the flow of work, like online videos and team messaging channels, the volume of learning data points will continue to grow6 and the importance of what we measure will only accelerate. Where today many L&D functions look only at information within learning and talent systems, the increase in informal and social learning experiences will likely demand the collection and measurement of data points across multiple systems and functions. Amid the growing noise of data, it might be easy for the deafening roar to overwhelm. But if one listens intently, a symphony for how to effectively orchestrate the L&D function in service of the business becomes clear.
Learning data doesn’t just live in your LMS—and knowing where to look and what to measure matters.
Outside the learning function, look to colleagues in marketing for inspiration. It is nearly impossible to imagine a new advertising campaign launching without testing customer reception of the message, or a new product hitting the shelves without promotion from key influencers. If we apply marketing rigor to the learning function—evaluating effectiveness, analyzing the target, and applying the insights to inform the business strategy—we can imagine a world in which employees are treated with the same consideration as customers. We can imagine strategically selecting, designing, and deploying learning experiences that prepare employees for the work of the future and enable us to track and validate the business impact.
For example, a large company wanted to evaluate and visualize the impact of learning on strategic business priorities. Its first step was to develop a learning measurement and evaluation charter that defined the factors critical for capturing learning data: a consistent framework, prioritized metrics, standardized data, a data governance processes, tools, and technology. Building upon the factors identified, a series of visual dashboards were developed to illustrate the impact and demonstrate the value of the learning function to the business. For example, one dashboard visualized training effectiveness by demonstrating the change in key performance goals before and after the training. For added flexibility and scalability, the dashboards could be easily modified to include additional metrics.
As a result, the organization is now able to generate reports aligned to business outcomes and to evaluate the impact of learning on key business priorities. Overall, the measurement and visualization of learning’s business impact enabled the organization to adapt learning experiences for its employees and optimize its investment in learning.
Gaining a clear line of sight: alignment and outcomes
Currently, many learning functions measure data points that are incomplete, focusing primarily on efficiency and effectiveness measures: how long it takes to develop a program, the cost to develop, total enrollments, and learner satisfaction. It is not to say that these measures are unimportant, merely that knowing these data points provide only part of the picture.
Although many companies are failing at effectively measuring L&D and leveraging its power to drive business outcomes, a few have stood out as shining examples. In fact, one global company will soon be able to save $75 million annually as a result of an L&D transformation that assessed current learning activity and investment and rationalized the learning curricula and vendors to focus investments on business-critical programs.
Capturing line-of-sight metrics
Eliminating wasteful expenditures on non-effective training and optimizing spend on learning solutions that can deliver the greatest value to the business demands broadening the scope beyond efficiency and effectiveness to capture line-of-sight metrics. Two key dimensions of line-of-sight metrics are alignment and outcomes.
Alignment refers to measures, actions, or governance that assist the learning function in meeting the needs of the business. In other words, alignment is the process of identifying and measuring the learning solution that addresses the skills, behaviors, and knowledge that most directly impact business outcomes. Alignment is moving beyond the number of minutes, cost per capita, or passing rate; alignment is asking did the behaviors, skills, and knowledge that most directly impact the business change as a result of this learning?
For example, imagine a technology company concerned about its ability to develop new products. In order to address this issue, it deploys an Innovation Lab focused on creativity and ideation. It examines the rate of prototype generation among employees who participated in the lab before and after the learning experience.
Similar to alignment, outcomes refer to measures that consider the effect of learning initiatives on business results. For example, imagine a manufacturing line manager interested in a learning solution to reduce the number of product defects on the assembly line. In this case, outcome measures might include the cost savings as a result of reduced completion time or improved production quality. These types of measures equip the line manager with the data and information necessary to serve as an advocate for learning and a leader of the business.
Making the jump
Transitioning from functional experts to business advisers is not easy, but the following four principles can help the L&D function think about how to begin:
- Focus on Impact: Prioritize the evaluation of employee segments whose skills, behaviors, and actions have the greatest impact on the business and define baseline performance outcomes at the learner, program, and function levels.
- Simplify: Counter low survey participation and reduce reliance on post-learning self-assessments with more reliable insights generated through statistical analysis.
- Align: Bring together the required business functions to develop and manage a business-driven data strategy.
- Be User Centric: Engage end users in the design process to confirm tools and reports are meeting business needs.
Amid the growing sets of data across all business units, these four principles can enable the learning function to drive business value. The path to becoming an L&D function that uses learning measures to make data-driven decisions that maximize business outcomes can be challenging, but the L&D functions that master it are able to sit confidently at the table as unequivocal learning experts and leaders of the business.
We’ll continue this discussion with part two in this series, focusing on ways to analyze your aggregated learning and business data to evaluate the impact of the learning experience.
1 DalleMule, L., & Davenport, T. (2017 June). “What’s Your Data Strategy.” Harvard Business Review, May – June 2017 Issue. https://hbr.org/2017/05/whats-your-data-strategy
2 The State of Learning Measurement, Bersin by Deloitte / Todd Tauber and Wendy Wang-Audia, 2015.
3 The State of Learning Measurement, Bersin by Deloitte / Todd Tauber and Wendy Wang-Audia, 2015.
4 Engelbert, C. E., & Hagel, J. H. (July 31, 2017). Radically open: Tom Friedman on jobs, learning, and the future of work. Deloitte Review, issue 21, 95-107.
5 Engelbert, C. E., & Hagel, J. H. (July 31, 2017). Radically open: Tom Friedman on jobs, learning, and the future of work. Deloitte Review, issue 21, 95-107.
6 Learning Trends 2015, Bersin by Deloitte.