What We Do

Critical Metrics for Big Business: Case Study 1

< Back

A large organization conducted a training evaluation survey and needed to quickly make sense of the key themes in the open-ended responses to several questions. By using a combination of advanced coding software and trained live-coders, Critical Metrics was able to code thousands of comments in a relatively short amount of time.

The survey had been administered both before and after the administration of training, which allowed for a comparison of comment themes over time. Analysis of the comments brought to light improvements in training that had been made, underscored problem areas that still needed to be addressed, and revealed new themes.

Critical Metrics reported findings for the overall organization, and provided a breakdown by organizational division and as well as employee position. This gave the organization insight into the themes for the overall organization as well as by key positions and divisions, thus facilitating potential interventions to future training efforts at global and local level.

Armed with the new information and insight, the company was better prepared to organize an improved training program for employees for the coming year.

Critical Metrics for Big Business: Case Study 2

Training Evaluation Case Study

Context: Over the last several years, a global hospitality brand had heavily invested in brand-related training and needed to evaluate the effectiveness of their efforts. The training was rolled out to over 1600 locations through a combination of methods, including computer-based, video-based, and group-led training tools. End users of training were not employees of the hotel chain itself. Instead, most were employed by local franchisees and management companies. Additionally, this employee population was hard to reach given hotel scheduling and the 24-hour nature of the hospitality world. These factors made traditional survey efforts a challenging method of data collection.

The client was interested in understanding how its brand-related training efforts were being used and understood at each location and how this, in turn, impacted important business outcomes like customer satisfaction and quality assurance. Critical Metrics was retained to find out if and how brand-provided training tools and resources were being used. A second goal was to identify barriers to training usage. Most importantly, Critical Metrics needed to demonstrate the impact of using training resources on brand-related knowledge, behavior, and guest experience and quality.

What We Measured: A team of Critical Metrics researchers was trained in the culture of the hotel brand and experienced most of the training they were to evaluate. Using an ethnographic approach, the research team visited over 75 locations based on a sampling plan stratified across geography, ownership structure, and hotel performance–although researchers were blind to sites’ performance standings. Over a five week period, over a dozen researchers conducted 121 focus groups with over 750 employees and 90 managerial interviews to gather quantitative and qualitative data on training usage, knowledge, and behavior.

Some of the metrics we collected included:

• Observations of brand-related behavior (each researcher spent a night and a day at the property as a guest and completed a behavioral checklist)
• Self-reported brand-related behaviors measured during focus groups
• Usage of various training tools and resources from manager (interviews and observations) and employee perspectives (questionnaires)
• Managerial and employee perceptions of the effectiveness of training tools and resources (questionnaires)
• Employee brand-related knowledge (actual knowledge tests)
• Other contextual factors about training (managerial encouragement, communication, managerial and employee understanding of training available, future training needs, etc.)

Results from these interviews and focus groups were aggregated to the property level and linked to property level guest loyalty metrics and quality data from internal audits.

What We Found: Greater training usage was related to greater brand-related knowledge, increased brand-related behaviors (see Figure 1), and higher quality assurance ratings (see Figure 2). Increased brand-related behaviors were associated with higher guest satisfaction and loyalty ratings (see Figure 3). Overall, the use of brand-related training was associated with a more positive guest experience and higher overall service quality. Although the results revealed that training usage lead to positive outcomes for the client, usage in general was low. Time was the top barrier to training usage. Increasing training usage and re-engaging managers around training is now a priority for the client.