jump to navigation

Quality, Rework, and Throughput March 3, 2014

Posted by Tim Rodgers in Process engineering, Quality, Supply chain.
Tags: , , , , , , ,
add a comment

Some years ago when I was managing a software quality department I got into a heated conversation with one of my colleagues about our testing. “If your team would just stop finding defects, we can wrap up this project.” I had to point out that it wasn’t my team that introduced the defects in the first place. Of course no one deliberately does that. Software engineers want to write new code, not fix their (or anyone else’s) mistakes.

Quality isn’t only important for external customers and end-users. Internal operations should also improve quality as a way of focusing limited resources on value-added activities. In manufacturing, repair and rework are part of the “hidden factory” that reduces throughput and can prevent the plant from running profitably.

This isn’t hypothetical. At the China factory where I worked my goal was to improve end-of-line yield to the point where we could eliminate a single rework station for each production line. With more than 30 lines and 2 shifts that added up to some significant savings as well as an increase in capacity.

We’re human, mistakes will be made, and the complexity of our designs virtually guarantees that there will be unexpected outputs and interactions when we put those designs to the test in a service environment. However, we shouldn’t accept the frictional cost of fixing defects as an inherent inefficiency in the value delivery system. A defect found today is worth more than a defect found tomorrow, but a defect prevented by better design of products and processes has a far greater impact.


Higher Value for Higher Priced Employees November 22, 2013

Posted by Tim Rodgers in International management, Product design, strategy, Supply chain.
Tags: , , , ,

You can complain about it, but offshoring is not going away. Businesses will always look to reduce cost, and wherever there’s a significant difference in labor cost, that difference is going to attract interest. I’ve spent almost my entire career working at companies that have moved their supply chain and production factories to locations that have lower labor cost. For manufactured goods this savings must be weighed against other expenses to determine whether there’s a net gain, such as shipping costs and finished goods in-transit. For knowledge work where there’s virtually zero cost to instantly move the output from one part of the world to another (such as software), the advantage is even greater.

You can complain about it, but if you want to justify a higher cost of labor in one part of the world, you have to demonstrate that this labor provides higher value. The added cost must be offset by some benefit, ideally something that can be quantified. It’s important to distinguish between sources of higher value that are fundamental and relatively stable vs. those that can be eroded over time.

Here are some examples:

1. “We know how to do it here, they don’t know how to do it there.” Your design team, and factory, and supply base may be well-established in one location, but you’re wrong if you think that can’t be replicated somewhere else. There are smart, well-educated people all over the world, and it’s easier than ever to access their skills, especially for knowledge work. There will be training, start-up, and switching costs, and those will have to be evaluated against the steady-state labor cost savings, but it’s not impossible.

2. Cost of quality. This is related to #1 above. You may be able to produce output at a different location with lower labor cost, but does the quality of that output lead to additional expenses later, such as rework, field repair, and loss of customer loyalty? These can be addressed with specific improvement plans, depending on the causes of poor quality, and are not necessarily permanent conditions. As above, the costs to improve or maintain quality at any location should be compared with the labor savings.

3. Geography. This is an example of a more fundamental difference that may justify higher labor cost. Many businesses benefit from close physical proximity to their customers, enabling them to respond quickly to changes in market demand and mix without the burden of a long finished goods pipeline from their production sites. A hybrid approach is late-point differentiation where platforms are built ahead at low cost and later customized depending on the specific order. Another benefit of geography is co-design, where frequent, real-time interaction with customers leads to a better fit to their requirements. Some companies will overcome this one by using available technology to communicate with remote teams, or performing rapid prototyping locally to verify the design before shifting volume production elsewhere.

Note that geography can also be an overriding factor when there are political or economic barriers, such as regulatory or “local content” requirements.

My point is that if you insist on doing the work in a location with higher labor cost, you can’t assume that the corresponding value will always be worth the higher cost. Your survival as a business depends on your ability to identify, develop, exploit, and maintain a source of competitive advantage. Your choices about labor cost and geographic location should support your strategy to maintain competitive advantage, and that strategy should be regularly reviewed and updated to make sure you’re getting the value your paying for.

Joining a Software Project In-Progress July 11, 2012

Posted by Tim Rodgers in Management & leadership, Product design, Project management, Quality.
Tags: , , ,
add a comment

Yesterday a question on LinkedIn caught my eye: “What are 3 key metrics you use to assess [the] health of a given project? … Do you have your own checklist to use when being added to [an] already running project?” The person who posed the question identified themselves as a software developer, so my answer assumed a software project.

If I’m limited to only three metrics, here’s my checklist:

1. Schedule status: how far behind is the project vs. the original plan? You can’t change the past, but the actions you take from this point forward depend on how far behind you are now.

2. Scope creep and requirements stability: what is the weekly trend in requirements changes? What you’re looking for here is convergence on a fixed set of requirements. On-going turmoil and failure to converge is a red flag for both schedule and quality.

3. Defect management: what is the weekly trend in new, open, and closed defects? Is the team keeping up with defect fixing, or is there a growing backlog that will eventually overwhelm them?

With this information a project manager who joins a software development project in-progress should have a good idea of the current status. Certainly there are other things that would be worth knowing, for example the “validated completeness” of individual functional requirements, as determined by test results.

New Product Introduction (NPI) Transitions June 21, 2012

Posted by Tim Rodgers in Project management, Quality, Supply chain.
Tags: , , , , ,
add a comment

Whether you’re working on a hardware product or a software release, it’s obviously a good idea to build some prototypes and evaluate the cost, performance, and reliability of the design before offering it for sale. Many organizations follow some kind of formal development lifecycle, which is essentially a structured process with well-defined and documented phases, each phase having a set of activities assigned to different functional groups that are supposed to be completed, and exit criteria that are supposed to be met before moving on to the next phase. Senior management is naturally interested in monitoring the progress of the development project, and it’s convenient to use the transition from one phase to another as a time for a checkpoint meeting.

A checkpoint meeting can be a simple project update, but organizations with a formal lifecycle will claim that a checkpoint is actually a phase gate, meaning that the project team must present evidence that they have completed all the required tasks and satisfied all the exit criteria. If those criteria are not met, the project team cannot move on to the next phase.

Unfortunately the reality for most organizations is that except for some scolding from the senior managers there are typically few consequences for failing to satisfy the requirements of a phase gate. Schedule commitments tend to overwhelm process fidelity. The project team may seek a waiver for some of the exit criteria, either promising to complete them at a later date TBD, or insisting that their project is special and therefore exempt from those “rules.” The quality sticklers who believe in blind adherence to the documented process get mad, and everybody gets a little more cynical about processes in general.

Phase reviews are not just a date on the calendar. Phase transitions are supposed to signal a change in the behavior of the design team and the supporting functional groups. In the world of hardware development one of the most important transitions is from qualifying the design to qualifying the factory production processes. It’s generally a bad idea to make design changes while trying to stabilize the manufacturing line and reduce variability. It may go by different names, but there should be a checkpoint where the project team demonstrates that they have verified that the design meets the committed customer and business requirements. There’s a later checkpoint where the team demonstrates that they have verified that they can manufacture that design at the required level of production units, all of which meet the committed customer and business requirements.

There’s a similar phase transition in the world of software development, specifically for teams that don’t follow an agile/scrum model. It’s often called the functionally-complete checkpoint, where all of the code supporting the required functionality of the software has been checked in and verified. This is an important checkpoint for the software quality team because it doesn’t make sense to run the full suite of system tests beforehand.

Two additional recommendations regarding development lifecycles and phase transitions:

1. Keep it simple. Often the descriptions of phase activities and exit criteria are burdened with a lot of detailed requirements that really aren’t critical to the transition. They are all probably important things that need to happen at some point in the project, but if they’re not truly required to be completed before moving to the next phase, just track them like all other project tasks.

2. I believe in the product development lifecycle because I believe that signaling a change in how the team is supposed to behave is more effective than trying to put all the project tasks on a Gannt chart and track their completion. If the phase transitions are not strictly enforced, then the “checkpoint meeting” is just a project update. Don’t fool yourself that it means anything else.


How Do You Plan to Catch Up? June 12, 2012

Posted by Tim Rodgers in Management & leadership, Project management.
Tags: , , , , , ,
add a comment

When a project falls behind schedule — as it almost inevitably does — there’s a tendency for project managers and their teams to enter a state of denial and convince themselves that they can catch up by “working harder.” Maybe they remembered an earlier stretch when they weren’t working especially hard, maybe thinking there were still many weeks or months of budgeted time. Perhaps they encountered a barrier that couldn’t be easily overcome right away, and decided to “work around it” and come back to it later. Senior managers who are responsible for providing oversight to the project teams may accept this “working harder” plan, either because they’re sympathetic or don’t want to be bothered with the details.

How often does that actually succeed? Senior management shouldn’t accept an explanation that the project team will simply “work harder.” They should require the project team to explain how they will work differently. Doing things the same way is not going to help the team catch up; there needs to be a change in the scope of the project or how the work is done, for example adding resources, re-assigning tasks, or re-arranging tasks on the Gannt chart.

When I managed software testing we would track the number of defects found and fixed each week over the course of the project. The goal was always to have zero open defects by the time of the scheduled software release date. From historical data we had a rough idea of how many defects one developer could fix each week, so at any point in the project we could extrapolate an end date based on the number of open defects. If the extrapolated date was later than the scheduled date, we knew we were behind schedule and had find a way to catch up. In a case like this “working harder” means somehow increasing the rate at which the work (defect fixing) gets done, because continuing at the current rate was not going to reach the goal.

We used a variety of strategies, depending on the software project. Most often we would reduce the scope of the remaining work by de-rating defects and removing them from the fix queue, essentially accepting a lower level of quality. Sometimes we would add resources (developers) to fix defects, but anyone who has read “The Mythical Man-Month” from Fred Brooks knows that adding resources to a late software project just makes it later. We would finesse that problem by changing priorities and re-assigning tasks within the existing team instead of adding people (it didn’t always work).

Regardless, project managers shouldn’t be allowed to wave their hands and offer vague plans when projects fall behind. The catch-up plan must explain how things will be done differently, and it must be explicitly reviewed and approved by senior management to ensure a successful project completed without further surprises.

Designing a Software Quality System May 30, 2012

Posted by Tim Rodgers in Product design, Quality.
Tags: , , ,

Last time I wrote about considerations in designing a quality system for a hardware product, a system that’s based on engineering analysis of the design and doesn’t rely solely on end-of-line testing or audits of finished goods.  This post examines the differences and unique elements associated with a software quality system.

Hardware development relies on early prototyping, an iterative process that uses measurements and test results to verify that a product based on the design meets the performance specifications, and that the manufacturing processes can repeatedly and consistently deliver many products that meet those specs. Software delivery doesn’t rely on a manufacturing process (especially now that almost no one uses CDs anymore), so software development focuses on a single, final release of the code that meets the specs.

Agile or scrum development is very much in vogue right now (more on that later), but the older waterfall development model is still widely used. In the waterfall model an integrated, fully-functional build is delivered to the quality group who use system-level “black box” testing to check the performance and reliability. Defects are prioritized and fixed, and another build is released for another round of testing that includes verifying the defect fixes. The iterations continue, gradually converging on a level of quality that’s acceptable, or running out of time, whichever comes first. Black box testing is the equivalent of end-of-line testing in the hardware manufacturing world; it’s better than nothing, but it’s late in the process and makes it hard to isolate the cause of any test failures found.

As with hardware development, a better quality system includes early testing and verification of the building blocks of the software, usually called modules or subsystems. Assuming the modules have been architected and designed with well-defined interfaces, it’s possible to test and validate these intermediate deliverables before they’re integrated with the rest of the software using white-box unit tests or test harnesses. Many software teams still rely on the integrated build to indirectly test subsystems instead of taking the time to create specialized unit tests, but that time has to be balanced against the time required to debug system test failures to figure out where the defect is.

I’m a big fan of the agile software development model because (when practiced correctly) it allows you to verify the quality in stages before new functionality is added to the build, instead of waiting until the software is finished before starting testing. This means that if there are N stages planned for an agile project, you could theoretically deliver the software at stage N-1 when you inevitably run out of time. The software at that point would be missing some lower priority functionality, but it would have been fully tested and qualified along the way, thereby avoiding the last-minute scramble that often characterizes the waterfall method.

Regardless of the development model, testing of functional modules or subsystems is preferable to relying solely on system testing near the end of the program. An alternative to subsystem testing is to build and system test “prototypes” of software with partial functionality, ultimately leading to a functionally-complete (FC) checkpoint. Further development after the FC checkpoint can focus on robustness, defect fixing, and generally extending the basic functionality to include other use cases.

It’s a little harder to do an engineering analysis of software failure modes using FMEA or other processes, mainly because it’s harder to predict the impact of other software and operating systems. Hardware is tangible with clear interfaces and behaves fairly predictably according to well-established engineering principles. Software problems often arise because of unexpected interactions with its environment. Nevertheless, stress testing should be performed  to determine how the software recovers from error conditions, and assess possible timing and resource conflicts.

Whether software or hardware, the ideal quality system emphasizes: (1) design for quality, (2) testing activities early in the lifecycle, and (3) qualification of sub-units (parts, sub-assemblies, software modules) before integration.

Keeping Specs Updated April 16, 2012

Posted by Tim Rodgers in Communication, Process engineering, Product design, Project management, Quality.
Tags: , , , , , ,
add a comment

Some time ago I listened to the manager of a firmware development group explaining his team’s reluctance to keep requirements documents up-to-date for our product’s user interface. Basically he didn’t think it was worth the time, especially when the logical flow and the look of the UI were still moving targets. “Can’t we just do the coding first and update the specs later?”

I’ve written before about my love-hate relationship with process (see Process Conformance). I understand why people hate process. Often they think that it inhibits their creativity and makes them less flexible and agile. Generally, processes should be an enabler for greater efficiency and reliable execution, not a nuisance. When people follow the process there’s a lower cost for communication and alignment. Keeping requirements up-to-date sounds like a lot of extra work, but those documents are vital as a tool to make sure everyone understands the specs and aren’t working on an older version. The time it takes to update the specs should be balanced against the time and effort that would be wasted by not updating them or trying to communicate changes informally.

If the design team thinks they’re spending too much time updating specs, maybe they should be spending more time understanding why these changes are happening in the first place.

Quality and Productivity August 15, 2009

Posted by Tim Rodgers in Management & leadership, Process engineering, Quality.
Tags: , , , ,
add a comment

When I was managing a software quality department at Hewlett-Packard some years ago, my counterparts on the software development side of the house used to routinely complain about the number of defects found during testing. You would think their unhappiness would be directed at their own teams for introducing the defects in the first place, and that might have opened up some opportunities for collaboration around defect prevention, design reviews, code reviews, and unit testing. Instead, they seemed to be annoyed that the testing uncovered the defect, as if somehow they wouldn’t exist if they weren’t reported.

There is a legitimate discussion about the severity of defects, making sure they’re properly associated with real failure modes and customer requirements. James Bach has written a great deal on the subject of “good enough quality” in the world of software design, and I would refer the reader to his excellent site at http://www.satisfice.com

However, I think engineering organizations often miss the point when it comes to quality. It’s not about satisfying those pesky QA people who keep reporting defects. Ultimately, it’s about productivity. When design engineers are fixing their own mistakes, they’re taking time away from true value-added activities, like new features or higher functionality. It’s no different in the factory: scrap costs real money and wastes material and labor, even if rework is possible. It’s also demoralizing. I’ve never met a design engineer who prefers to spend their time on defects.

I wouldn’t try to sell the idea of quality in terms of some ideal perfect product that everyone can admire. Yes, quality has been defined as “meeting or exceeding customer requirements,” but it’s also about identifying and eliminating waste in the value-delivery chain, thereby improving productivity and enabling teams to achieve more. Isn’t that the ultimate reason to care about quality?

Success Criteria: Software Department July 30, 2009

Posted by Tim Rodgers in Management & leadership, Process engineering, strategy.
Tags: , , ,
add a comment

My philosophy of effective leadership hinges on the idea that teams need quantitative performance measures that are aligned with higher-level business objectives. At the most basic level, each team within a business consumes resources (e.g. salaries, other labor costs, direct expenses) while making some kind of positive contribution (adding value) to the overall enterprise. If the team can add more value while consuming the same or fewer resources over the same period, then that team has clearly helped the business become more successful. It’s fairly easy to measure the expenses associated with a team, so the challenge is measuring the value added, ideally in an unambiguous way that the team can directly control through their own actions, and aligned with higher-level objectives and strategies.

Here’s an example of how that works, from my recent experience leading a software development department. This team was responsible for design and delivery of embedded software as a component in a capital equipment product used in semiconductor manufacturing. At the time, the software was not sold as a separate component, and therefore did not directly contribute revenue, although we believed that the software contributed to the overall customer-perceived value of the product. In addition, software could be delivered and easily installed on previously-sold products through existing service contracts, providing an opportunity to add incremental post-sales value.

The software team’s contribution to the business was therefore based on our ability to deliver value to the customer, either as software-specific features or support for other high-level product features and specifications, and to deliver that value in a timely fashion by meeting new product introduction schedules and enabling/driving rapid deployment into the installed base. That translated into four high-level success measures:

  • Value-added software: support for new product features that meet customer needs, both at the basic “hygiene” level needed to maintain market share, and at the “wow” level needed to justify premium pricing.
    • Key performance indicator (KPI): Estimated dollar contribution of software. This is obviously a subjective estimate, but I really believed that our customer-facing sales and support organization could help us here.
    • Needed: Better understanding of customer needs to target high-value features.
  • On-time delivery: software available on-time to meet announced dates for new product availability and customer evaluation schedules
    • KPI: Actual checkpoints vs. plan. It’s especially important here to get an early indication of delay vs. plan while there’s still time to catch up.
    • Needed: Disciplined development processes, an architecture that supports design of incremental features, and high quality to ensure smooth customer qualification
  • Fully deployed software: rapid introduction of new features, replacing old software
    • KPI: Penetration, percentage of installed base using new software after introduction
    • Needed: Tighter collaboration with field support to drive change orders
  • Quality
    • KPI: Escapes, defects reported by customers, especially those that prevent the customer from qualifying and deploying value-added features
    • Needed: Relentless dedication to zero defects and elimination of variability

Obviously the value contribution will be different for other teams in other businesses, but I think this approach can be generalized. What product or service does the team deliver? How do others value that product or service? What can the team do to add more value?

On the cost side, I wanted to look at two things:

  • Operating expenses: those monthly reports from finance on how much money the department spent vs. budget
  • Operational efficiency: how much time is wasted on non-value added activities?
    • KPI: Percent of time spent by the team fixing their own mistakes, specifically defects found and fixed internally
    • Needed: Value chain analysis to identify sources and eliminate waste

I think these six measures were sufficient to express the contribution of the software team, track changes over time, and, maybe most importantly, ensure that proposed actions and changes would lead to improvements that benefit the business.

What Problem Are You Trying To Solve? June 20, 2009

Posted by Tim Rodgers in Organizational dynamics, strategy.
Tags: , , ,
add a comment

Late last week I met with someone who was interested in my experiences managing outsourcing of software engineering services. We talked about the difficulties that organizations typically encounter when they attempt outsourcing, and the fact that many are ultimately disappointed. It occurred to me that there’s a fundamental question that probably doesn’t get answered (or asked) in many of these situations: What problem are we trying to solve? In this case outsourcing is a means to achieve some higher goal, which might be something like lower labor rates, or access to technical capabilities not present within the current organization, or something else entirely. The point is that without a clear understanding of what outsourcing is supposed to do for the organization, it’s pretty unlikely that people and processes and measures will be aligned. If you don’t know where you’re going, you’re sure to get there.

%d bloggers like this: