jump to navigation

Wasting Time On Meaningless Data May 28, 2014

Posted by Tim Rodgers in baseball, Management & leadership, Process engineering.
Tags: , ,
add a comment

One of the things I love about baseball is that it’s amenable to assessing and studying performance using numbers, and that appeals to my analytical side. Since I started following the game as a boy there’s been a lot of questioning of the traditional measures of player performance, particularly with the emergence of sabermetrics as a serious area of study. Older metrics such as pitcher wins and batter RBIs have been challenged as inadequate and sometimes misleading because they don’t reflect the isolated contribution of a single player. Just because they’re easy to measure doesn’t mean that they contribute to a better understanding.

One of the breakthroughs was when people started looking for ways of associating the performance of a player with the objectives of the team, which is to win games. The team wins games when they score more runs than the other team, so a player who significantly improves the likelihood of scoring runs or preventing runs is valuable. That may seem obvious, but the older, traditional metrics don’t reflect that. It’s a complicated empirical model, and the sabermetric community has been experimenting with new metrics based on data that has never been systematically collected before. As a devoted fan with an analytical disposition I’ve been watching this with great interest.

All of this reminds me of organizations that collect and report data that are disconnected from or unrelated to the real drivers of success. I’ve written about this before, arguing that if the data isn’t being used to manage the business, then the business should step back and re-visit their metrics. More data and more metrics doesn’t always lead to better insight and decision-making.

Here are some key questions that should be asked when evaluating business metrics:

1. Can you control it? Can you “move the needle?” If you can’t control it, you can’t manage it. This is worth knowing. It may still be worth tracking because it provides some interesting background information that could be used as input for future decisions.

2. Is there a correlation between this metric and business success? If your deliberate action helps to move the needle in the right direction, is the business significantly better off? Conversely, if your lack of action leads to no change, or a change in the wrong direction, is the business worse off? I realize that we’re talking about complex interactions with imperfect models, but if there isn’t some kind of causation and correlation, then the metric has no value.

Finally, a word about balanced scorecards. I’m all for simplicity and focus in our communications. We should seek a minimum set of metrics that describes status and progress. In our desire to provide a complete picture we run the risk of ambiguity and confusion.

Metrics should be regularly reviewed for their effectiveness. If they’re not helping to manage the business, we should find new ones.

 

Change Management and “Moneyball” (Movie Version) December 1, 2013

Posted by Tim Rodgers in baseball, Communication, Management & leadership, strategy.
Tags: , , , , ,
add a comment

The other day I watched the movie “Moneyball” again and was reminded of a few important characteristics of successful change management. Brad Pitt stars as Billy Beane, the general manager of the Oakland A’s baseball team, an organization struggling with a limited budget to develop, attract, and retain players.

Some background:

At the beginning of the movie we learn that before the 2002 season the A’s have lost three of their best players who have signed more lucrative contracts elsewhere. Beane is trying to figure out how to replace these players, and more generally put together a winning team within the financial constraints imposed by ownership. After a chance encounter with a low-level analyst from a rival organization, Beane realizes that he cannot compete if he builds a team using the traditional ways of assigning value to players. Almost out of desperation, he decides on an unconventional strategy based on the emerging science of sabermetrics. He immediately faces resistance from his experienced staff, specifically the field manager and scouts who are unconvinced and in some cases actively working against the strategy.

Ultimately it’s fairly happy ending: despite public criticism of Beane’s decisions and early disappointments on the field, the A’s have a successful season. At one point they win 20 straight games, setting a new league record, and they make the playoffs, but lose in the first round. Beane is offered a significant raise to leave the A’s and join the Boston Red Sox  where he would have the opportunity to apply the same principles with a much larger budget. Beane declines the offer, but the unconventional strategy has been seemingly validated.

The movie focuses Beane’s underdog status and uphill battle during the season, and I’m sure some of the real-life events have been changed for dramatic effect. Regardless of whether they actually happened or not, there are several scenes that illustrate elements of successful change management.

1. A clear explanation of the new direction. In the movie, Beane leads a meeting of his senior staff to discuss plans for acquiring players for the upcoming season. This looks like Beane’s first opportunity to apply his new strategy, but he misses an important chance to align with his team. It’s clear that he’s the boss with the final authority, and it’s not necessary for everyone in the room to agree, but Beane could have taken the time to explain the new direction and acknowledge the objections. In later scenes, Beane acknowledges this mistake to his field manager who has been undermining the strategy through his tactical decisions, and fires a senior staff member who has been especially vocal in opposition.

The lesson: the team may not agree with the change, but they should be very clear about why change is needed. Team members should have the opportunity to raise objections, but once the direction has been set, their only choices are to support the change or leave the team.

2. Removing options to force compliance. Beane is frustrated by opposition from his field manager who gives more playing time to players whose skills are not highly valued in Beane’s new system. Beane stops short of giving a direct order to the manager to be make decisions that are more consistent with the strategy, and instead Beane trades these players to other teams, effectively removing those undesirable options. This is a variation of what is sometimes called “burning the boats,” from the Spanish conquest of the Aztec empire. You can’t go back to the old way of doing things because that way is no longer an option. As Beane replaces players, his manager has fewer opportunities to not follow the strategy.

The lesson: this seems like passive-aggressive behavior from both parties, but I can see how it can be effective. My preference would be to reinforce the desired change rather than take away choices, but if the old way is very well established you need to help people move on and not be tempted to return.

3. Giving it a chance to work. The A’s get off to a slow start and pressure builds on Beane to abandon the new strategy. In one scene he meets with the team’s owner and assures the owner that the plan is sound and things will get better. It eventually does, despite all the skepticism and opposition, and the movie audience gets the underdog story they were promised.

The lesson: even the best ideas take time. It’s absolutely critical to set expectations with stakeholders to help them understand how and when they will detect whether the change is working. Impatience is one of the biggest causes of failure when it comes to change management. 

Common Fallacies That Cause People to Doubt Statistics October 23, 2012

Posted by Tim Rodgers in baseball, Process engineering, Quality.
Tags: , , , , ,
add a comment

Lately I’ve been reviewing some old text books and work files as part of my preparation for the ASQ Six Sigma Black Belt certification exam in March. It’s interesting, and I think often amusing, to contrast the principles of inferential statistics and probability theory with they ways they’re used in the real-world. I think people tend to underestimate how easy it is to misuse statistical methods, or at least apply them incorrectly, and this can lead them to undervalue all statistical analysis, regardless of whether or not the methods were applied correctly.

I see this in baseball and political commentary all the time, particularly in the way people selectively or incorrectly use numbers to defend their point of view, while at the same time mocking those people who use numbers (correctly or not) to defend a different point of view.

Here are a few of the more-common mistakes that I’ve seen in the workplace:

1. Conclusions based on small sample sizes or selective sampling. Yes, we often have to make do with less data than we’d like, but that makes it especially important to put confidence intervals around our conclusions and stay open-minded about the possibility of a completely different version of reality. Also, a sample is supposed to represent the larger population, and we have to beware of sampling bias that excludes relevant members of the population and skews any findings based on that sample. Otherwise the findings are meaningful only for a subset of the population.

2. Unknown or uncontrolled measurement variability. We often assume that our measurement processes are completely trustworthy without considering the possible effects of variability due to equipment or people. If the variance of the measurement process exceeds the variance of the underlying processes that we’re trying to measure, we can’t possibly know what’s really going on.

3. Confusing independent vs. dependent events. There is no such thing as “the law of averages.” If you flip a coin 10 times and it comes up heads every time, the probability of a heads coming up on the 11th flip is still 50%. The results of those previous coin flips do not exert any influence whatsoever on future outcomes, assuming each coin flip is considered a single event. That being said, the event “eleven consecutive coin flips of heads” is an extremely unlikely event. If you take a large enough sample size, the sample statistics will approximate the population statistics (50% heads and 50% tails for an honest coin), sometimes simplistically referred to as “regression to the mean.”

4. Seeing a trend where none exists. This is usually the result of prediction bias, where we start with a conclusion and look for data to support it, and sometimes leading to selection bias, where we exclude data that doesn’t fit the expected behavior. Often we’re so eager for signs of improvement that we accept as proof a single data point that’s in the right direction. This is why it’s important to apply hypothesis tests to determine whether the before and after samples represent statistically significant differences. It’s also why we should never fiddle with a process that varies randomly but operates within control limits.

5. Correlation does not imply causation. You may be able to draw a regression line through a scatter plot, but that doesn’t necessarily mean there’s a cause-and-effect relationship between the two variables. This is where we have to use engineering judgment or even common sense. Earlier this year the Atlanta Braves baseball team lost 16 consecutive games that were played on a Monday. No one has been able to explain how winning or losing a baseball game could possibly be caused by the day of the week. A related logical fallacy is post hoc, ergo propter hoc (after it, therefore because of it). Chronological sequence does not imply causation, either.

Baseball and Measuring Individual Performance October 4, 2012

Posted by Tim Rodgers in baseball, Management & leadership.
Tags: , , , ,
add a comment

In early October 2012 one of the biggest current controversies in baseball is the question of who should be the American League’s Most Valuable Player: third-baseman Miguel Cabrera of the Detroit Tigers or center-fielder Mike Trout of the Los Angeles Angels. Both have had outstanding seasons by any measure, and Cabrera has received worthy praise for being the first player since Carl Yastrzemski in 1967 to win the hitter’s Triple Crown: leading the league in batting average, home runs, and runs-batted-in (RBIs).

For many baseball writers, commentators, and fans, this Triple Crown achievement is the strongest argument for Cabrera as league MVP. On the other side is the growing movement of the sabermetrics community, which for over 20 years has challenged the conventional wisdom about what constitutes a good season for an individual player, and how we compare the performance of different players. One of their issues with Cabrera and the Triple Crown is the importance given to RBIs. If a batter gets a hit (or in some cases even an out) that enables a baserunner to score, they get an RBI. If a batter gets the same hit in a different situation where no baserunner scores, there’s no RBI. The point is that what the hitter did is the same in each case. RBIs are not a measure of the hitter’s isolated performance because it depends on what other people have accomplished (getting on-base), or will accomplish (scoring a run after the hitter does his thing).

This suggests that any good hitter would get roughly the same number of RBIs if they had the same opportunities to bat with runners on base. Or, conversely, Cabrera would have significantly fewer RBIs if he were on a different team that did not put as many people on base.

BTW, some people have argued that any high RBI total is evidence that the batter is a “clutch hitter” who somehow performs better in high-impact situations. Unfortunately for those folks, there’s absolutely no evidence to support the idea that the “clutch hitter” exists. When you examine any player’s performance over an extended period (large sample size), there’s no statistical difference between how they hit with runners in scoring position vs. how they hit with no one on base.

Another example: pitcher won-lost record. Certainly a pitcher who doesn’t give up runs is valuable, but whether his team wins or loses the game depends on how many runs the team scores. As with RBIs, the won-lost record of a pitcher is not a good measure of his isolated performance, although certainly the team will ultimately be measured by their wins over the course of the season.

This is interesting to me, not just as a baseball fan. As managers we’re often responsible for measuring the performance of individuals and teams. I wrote about this in an earlier post in 2009 (see Individual Performance Measures). Team performance can be judged by examining their accomplishments and contributions to strategic business goals. Individual performance is harder because it’s harder to isolate and measure the unique contribution of one person without considering the context and environment, yet at most companies the compensation and bonus plans are tied to individual performance.

We have to determine what this person did to enable team success or avoid team failure. We need to take into account the interrelated nature of work and the limited power and influence of one person, since there are few jobs where one person has full control over the outcomes. On an individual level, we need to match performance measures to the person’s assignments, which makes them binary: Did this person accomplish this goal, or not? The challenge for the manager is to understand context, and differentiate between isolated good performance and average performance under favorable circumstances.

Read This: “Management Lessons From Major League Baseball” April 27, 2012

Posted by Tim Rodgers in baseball, Management & leadership.
Tags: , ,
add a comment

Here’s another example of how sports, specifically Major League Baseball, provides insights to effective management (from Julie Moreland on the Fast Company site):

http://www.fastcompany.com/1835194/management-lessons-from-major-league-baseball?partner=rss

“1. Job fit matters as much as ability.” Possibly more so. Smart and resourceful managers know how to put the right person in the right job (see Looking for the Square Hole).

“2. A common sense of purpose is more valuable than a massive payroll.” If employees are treated as mercenaries and interchangeable parts, they won’t put in the extra effort and long hours that accompany a shared and inspirational vision.

“3. Effective leadership requires striking a balance between micro-managing and passivity.” Finding and maintaining that balance is one of the toughest challenges for a manager, who must be attentive and responsible for the performance of the team while giving the team opportunities to learn from their mistakes and exceed expectations.

What’s So Wrong About Managing By the Numbers? March 21, 2012

Posted by Tim Rodgers in baseball, Management & leadership.
Tags: , , ,
add a comment

I think many people have an innate bias against the idea that performance can be quantified and businesses can be managed “by the numbers.” I can only guess, but maybe this derives from a fear that this would leave no room for subjective and non-quantifiable considerations, and thereby reduce all management to reading a measuring stick. When faced with unfavorable or ambiguous numbers, these people reject outright the whole idea of objective measures and ridicule another failed effort; or at least become passive aggressive, ignoring the numbers and going back to what they “know works best.”

Businesses are complex systems and it shouldn’t be surprising that it can be extremely difficult to build a model that relates some number of controllable measures to “success” (in whatever way that may be defined). We would like to be able to say that if these numbers move in the right direction and exceed certain goals, then we have achieved success. The value in using numbers is that they are inherently objective.

But more than that, we we would like everyone in the organization to understand how their actions directly contribute to moving the needle in the right direction. That’s why divisions and departments and functions create lower-level performance metrics. In theory, if everyone meets their measurable goals, then we’re all pulling in the same direction. (In practice, the cumulative effect may be multiplied, not just added.)

If we don’t see the expected success at any level in the organization, that’s not a reason to reject the idea of a quantitative model to guide management. If managing by the numbers doesn’t work, isn’t that because the model of the business isn’t well characterized or understood? Shouldn’t we try different measures, analyzing the results and tweaking until we get a better model? What’s wrong with trying to express business performance on a less subjective basis?

Each team in U.S. Major League Baseball is scheduled to play 162 games during the regular season (not including playoffs). Regular position players who avoid serious injury can expect to have 400 to 700 plate appearances during the season, and starting pitchers can expect 20 to 30 starts with as many as 100-120 pitches per start.  Unlike most other professional sports, baseball’s relatively large number of events and statistically significant sample sizes makes the game amenable to using numbers to measure, compare, and predict performance.

Over the first 100 years or so of baseball history, team owners, general managers, field managers, players, and fans used measures that reflected a poor and incomplete understanding of what contributes to the desired goals of scoring runs, preventing runs, and ultimately winning games. The recent history of baseball has witnessed the growing popularity of more rigorous statistical analysis (sabermetrics) which now makes it possible to estimate the financial value of a player and the contribution of a single in-game decision to the probability of winning that game.

The problem wasn’t that baseball couldn’t be analyzed and managed with numbers, the problem was that the wrong numbers were being used.

Read This: “Bring Back the Organization Man” March 19, 2012

Posted by Tim Rodgers in baseball, job search, Management & leadership.
Tags: , , , , , , ,
add a comment

Last week I discovered an excellent article by Peter Cappelli from the HBR Blog Network:

http://blogs.hbr.org/cs/2012/03/bring_back_the_organization_ma.html?awid=5825209905911292232-3271

This is great stuff. I strongly encourage you to read this for yourself, but here’s what I got out of it:

1. Companies are tending to hire external candidates to fill positions, which takes less time than developing internal candidates and allows them to aggregate the specific skills and experiences needed at any given time (“plug ‘n play”), and replace workers and re-assemble a different team as business requirements change (“‘just-in-time’ workforce”). Unfortunately this kind of short-term thinking is leading to retention problems as employees realize they will have to leave the company in order to achieve their career goals. In response to the turnover, companies reduce their spending on training and development, which reinforces the cycle.

2. Hiring from the external pool is inherently unpredictable: “The supply of skills in specific areas is uncertain, so the quality and price jumps around a lot.” There’s also a question of whether an external candidate is a good fit for the company’s culture, something that can’t be easily assessed during an interview process.

3. Many employees fear losing their jobs to external candidates who have specialized skills that a company needs at a given point in time, and this fear leads to sub-optimal performance.

4. Mr. Cappelli suggests a return to the old “Organization Man” model where companies invested in the training and development of their employees. The internal pool of employees is a more reliable and predictable (and cheaper?) source of talent to meet the changing and unknowable future needs of the business. When employees see that the business is committed to retention and professional growth, their anxiety level is reduced and performance improves.

I agree, but I worry about how many companies are willing to make that investment, particularly if they are focused on short-term financial and performance goals.

There is a staffing model that assumes that a high-performing team can be assembled from available people, including current employees, external hires, temporary contractors, and outsource partners. I think of this as the “Hollywood model” where a team comes together to achieve a specific goal (make a movie), then disbands as a formal group when the goal is achieved. Some members of the team may re-aggregate in the future to make another movie, depending on their performance, availability and price tag.

This may sound attractive as a customized approach, matching the goal and the skills required to the team members, but it assumes that success is just a matter of assembling a bunch of specialists and mercenaries.The whole is not necessarily greater than or even equal to the sum of the parts, and may in fact be less. There are many possible reasons for this: no loyalty, poor interpersonal compatibility, no commitment to a “greater good,” no incentive to help teammates.

For another example, note how many Major League Baseball teams have achieved success on the field simply by acquiring expensive free agents instead of developing talent from within. Success in baseball more often comes when a team complements their home-grown players by selectively filling key positions with external hires who don’t disrupt the clubhouse culture.

Measure of a Manager October 15, 2011

Posted by Tim Rodgers in baseball, Management & leadership.
Tags: , , , , ,
1 comment so far

What does a manager do, exactly, and how should their performance be measured? I don’t think that’s an easy question to answer. It’s tempting to say that a manager has done well when the team they’re responsible for achieves some kind of success that’s consistent with the higher-level objectives for the business. But, how much of that success can be directly attributed to the decisions and deliberate actions of the manager? Might the tam have been equally — or, even more — successful under the leadership of a different manager, or even without a manager at all? Did the team succeed despite the meddling of a poor manager?

I’m a baseball fan, so bear with me. At this writing there are only four teams left in the Major League Baseball playoffs. One of those four teams will win the World Series, which is clearly a successful outcome. Is the best manager for any given year the one who leads their team to a World Series championship? The only thing we can say for sure is that this was the combination of team-and-manager who won it all, and there’s no way to assess their respective contributions to the success. Each year since 1983 the Baseball Writers Association of America selects a Manager of the Year for each league, and it’s rarely awarded to the manager whose team won the most games (or won the World Series). One manager actually won the award with an overachieving team that had a losing record (Joe Girardi of the 2006 Florida Marlins). He now manages the New York Yankees, a team with a higher payroll and higher expectations.

The point is that it’s hard to measure a manager. Managers aren’t just there to execute HR processes, communicate directives from above, and keep an eye on expenses. But, they also can’t be expected to exercise complete control over the performance of the individuals they manage, and they typically don’t have the authority to upgrade their team by bringing in new people with better skills. When managers blame their team when things go wrong, someone needs to ask: What did you do to help achieve success?

Here’s how I look at a manager’s performance:

1. Context and circumstances have a lot to do with a manager’s success. Each manager has resources at their disposal, particularly people and budget, but how does the manager apply those resources? For example, do they distribute the responsibilities and assignments among the members of their team to match their skills and maximize performance?

2. At HP we used to say that how you get the work done is just as important as what gets done. Does the manager collaborate effectively with peers and other partners? Do they have positive influence beyond their positional authority?

3. A manager must have the judgment to know when to step in and when to stay out of the way. When did the manager step in, and what was the impact of their decisions? What happened when the manager decided not to get involved?

4. Any manager can cut costs, but that doesn’t necessarily improve operational efficiency. Did the manager introduce new processes in an effort to increase the team’s performance? What happened as a result?

5. It’s often underrated, but the manager should provide coaching and career development opportunities to their team. Does the manager prepare people for more challenging and valuable roles in the organization?

6. Finally, a manager should contribute to strategic planning. Has the manager demonstrated an understanding of what drives the business; and customers, competitors, and technology trends? How did the manager apply that knowledge in their leadership of the team?

A good manager has a positive impact on their team’s performance, giving them a better chance of achieving their objectives than they would otherwise. They may not be able to control the actions of individuals, but they have a lot of control over the environment where those individuals work.

Measuring Performance May 14, 2009

Posted by Tim Rodgers in baseball, Management & leadership.
Tags: , ,
1 comment so far

I’m a huge baseball fan. One of the things that I have always enjoyed about being a fan is the challenge of measuring and comparing individual performance. For years the only measures that were commonly available were batting average, home runs, and runs-batted-in (for batters), and wins-losses and earned run average (for pitchers). It was better than nothing, but it certainly didn’t provide a very complete or accurate view of a player’s overall performance, and certainly didn’t help you understand their value to the team’s objective, which is to win games.

There has always been a lot of “conventional wisdom” imbedded in the game that strongly influences players, managers, sportswriters, broadcasters, and fans. It takes the form of deeply-held biases that are typically accepted without question and subconsciously guides their behavior.

Starting around the mid-1970s, a group of dedicated baseball fans and self-admitted statistical nerds realized that there are enough measurable events in a baseball game, and enough baseball games in a season, to collect a lot of data and draw statistically-significant conclusions. They didn’t blindly accept the conventional wisdom, they posed questions and gathered data and discussed the results. Sometimes the conventional wisdom was supported by data, other times it was found to be completely unsupported. This scientific approach has become known as sabermetrics, defined by their pioneer Bill James as “the search for objective knowledge about baseball.”

It’s a search that never ends, because there will always be questions without data, and even with data it may never be possible to completely describe performance or contribution with numbers. But … that doesn’t mean we shouldn’t use objective, quantitative measures to help understand performance, even if that leads to more questions.

I believe the same concepts apply in the workplace.

Managers should strive to establish objective, quantitative metrics for team performance, ensuring that those measures are aligned with the interests of the larger enterprise so that optimizing team performance directly contributes to the success of the business. Ideally, those metrics should be completely owned by the team, meaning that the results can be unequivocally attributed to the team’s performance alone.

Managers must put mechanisms in place to make it easy to routinely measure the data, and publish the results for wide distribution.

Performance should be measured over a period of time to establish a baseline before setting goals, but now I’m getting into process control principles and I’m going to save that for another post.

Individual performance must be aligned with the team performance metrics. In other words, if an individual is performing well, then that must be reflected in the team’s metrics, otherwise either the metrics are wrong or the performance isn’t contributing to the success of the team (or the larger enterprise).

Last point (for now): the process doesn’t end. Managers must use the data and the analysis to ask new questions, including whether these are the right metrics at all. The numbers will never tell the whole story, but there is no story at all without the numbers. The purpose is not to reduce all human effort to a set of charts and graphs. The purpose is to use the charts and graphs to challenge assumptions and test hypotheses and continuously improve.

%d bloggers like this: