Strategy

A Framework for Visibility Effectiveness

This article, probably the longest one on the website, addresses four critical needs of anyone working with supply chain visibility. The first is to formalize how a visibility solution impacts the business. Second, how visibility impacts specific business decisions (i.e. along what dimensions will the solution deliver change and therefore value). Third, how to evaluate the fitness of a supply chain visibility solution to a given business problem. Fourth, how to compare competing visibility solutions side-by-side. This is a long article, but represents a culmination of years of academic and practitioner experience, as well as my own work to build a cohesive framework for these topics. Enjoy!

 

How to Measure Supply Chain Visibility Impact on a Business?

A major concern for supply chain leaders is the ability to measure supply chain visibility impact on the business. The proposal here is to treat supply chain visibility as a vector of influence on the supply chain operation, which in turn is a vector of influence on the business outcomes. This approach is shown graphically below.

Supply Chain Visibility

 

 

What this implies is that four specific components can be measured separately:

  1. One can measure the extent to which supply chain operations impact the business outcomes. This is the line of enquiry pursued by Gartner in their annual top 25 supply chain report and also by Supply Chain Insights in their Supply Chain Index. To some extent it is also used in the Supply Chain Council’s Benchmarking process.
  2. One can measure the supply chain operation directly. This is the purpose of tools such as the Supply Chain Council’s SCOR Metric methodology.
  3. One can measure the extent to which supply chain visibility improves supply chain operations. Not much research has been done in this area, but Johansson and Melin’s study in 2008 is an example. It’s probably best to use the SCOR Metrics methodology, looking for impacts to level three metrics and add in statistical control groups to isolate causality.
  4. One can measure the supply chain visibility effectiveness directly. This is, at present, an area with no published research. It is this area that is discussed in the proposed “supply chain visibility scorecard” below.

Supply Chain Visibility

 

How Supply Chain Visibility Impacts Business Decisions

Even if supply chain visibility does not have a direct impact on a business as a whole, it certainly does impact specific decisions. From a business value perspective, it is crucial that visibility be tied to specific decisions. The visibility solution must be interruptive, resulting in a different— and better— business decision. Otherwise, there is no value created by visibility. There are really only two ways a decision can be improved by supply chain visibility:

  • Improve the information available to the decision mechanism (faster, more accurate, more complete, and so on)
  • Improve the decision making mechanics itself (change the individual or group who makes it, change the criteria, change the frequency, enforce consistency, and so on)

 

The graphic below shows how this works in practice. The two vectors for improving a key supply chain decision are shown on the left, and the ways in which the decision is different, and therefore better, are shown on the right.

Supply Chain Visibility

 

 

 

Scorecarding a Visibility Solution

Supply Chain Visibility

The following metrics and methodology are referred to as the “supply chain visibility scorecard”, and they represent a novel approach to measuring the performance of supply chain visibility directly without conflating the metrics with the impact visibility is having on the overall supply chain operation. The scorecard is decomposed in to four primary metrics which directly reflect on the performance of the four steps within supply chain visibility. These metrics are therefore diagnostic tools for assessing the health of the visibility sub-processes. The four metric categories under the visibility fit scorecard are discussed in detail here, and then the scorecard evaluation process is presented.

The first metric of supply chain visibility performance is “sensitivity”. This metric covers the first step in the visibility process, i.e. the capture of data. A highly sensitive visibility process is one which very successfully captures supply chain data, and conversely an insensitive visibility process will not capture all the necessary data. The metric of sensitivity decomposes into these kinds of sub-metrics:

  • Accuracy & Bias
  • Completeness
  • Timeliness
  • Redundancy
  • Depth of Detail

The second metric of supply chain visibility is “accessibility”. This metric quantifies how integrated the visibility solution makes its data model. High accessibility implies that a business user may start from any point and find the data they need. It also implies that users can navigate from one object to another object in multiple paths (if they are not a hindrance to usability) and that such navigation is low-cost and fast. Accessibility can be well quantified. Look at the diagram of two data models below and then consider these example metrics of accessibility:

  • Do all data objects connect?
  • What is the min, average, median, and max node count between any two objects?
  • What is the average effort to move through an intermediary node?
  • What is the average time to move through an intermediary node?

Supply Chain Visibility

The design on the right has more data elements, which would make it stronger in the sensitivity category. But it is measurably less accessible: it has longer average connections between objects and making those connections is slower and more expensive. Accessibility directly measures the effectiveness of the second step in a supply chain visibility process: the integration of captured data. To the extent that this step is well executed, the accessibility of the supply chain visibility solution should be high.

The third performance metric of supply chain visibility is “intelligence”. The category “intelligence” refers to the effectiveness of the routines used to process data and render it into relevant information. It measures the third step in the visibility process, the creation of intelligence. In many ways, the intelligence behind a visibility solution is the hardest to measure. In general the intelligence of the visibility solution is quantifiable by these sub-metrics:

  • Ability to recognize an event or state as needing or not needing intervention
  • Ease of updating from users to improve the recognition of important business events
  • The ability to learn or develop independently or through implied performance feedback

The fourth performance metric of the supply chain visibility scorecard is “Decision-Relevance”. This category is a measure of how well the visibility solution integrates into business decisions. The decision may be frequent and transactional in nature (like selecting from a list of approved vendors or ship-dates) or may be strategic and infrequent, such as when planning a logistics network after a merger or acquisition. The decision relevance metric directly quantifies the effectiveness of the fourth step of supply chain visibility: the interruption of decisions. The decision relevance can be quantified using these kinds of sub-metrics:

  • Is the visibility process or solution required for the decision maker?
  • Which party (visibility solution or human decision maker) starts the process of making a decision?
  • Does the visibility solution offer one or more suggested actions?
  • Can the visibility solution execute any actions selected by the decision maker?
  • Can the visibility solution fully automate the decision?

The last metric of supply chain visibility scorecard is simply the solution cost. This category could be expanded to offer a more subtle view on costs, such as direct operating costs vs. fixed asset costs. Unlike the first four metrics described above, the cost of a visibility solution doesn’t connect to only one of its process steps but to the overall solution.

Using the Metrics to Evaluate Solution Fit

The metrics described above help breakdown the visibility performance so that each process step can be measured. To be effective, the supply chain visibility scorecard provides three more tools: (1) quantitative scales for the metrics, (2) guidance on how to aggregate the individual metrics in to a single performance or fitness grade, and (3) methodology for how to conduct comparisons of multiple visibility options by using the scorecard.  Regarding the first point, an associated grading scale is described below for the first four metrics; with “cost” assumed to be standard enough it doesn’t require a novel quantification approach. The grading scales are examined in detail now, while points two and three are addressed afterwards.

Grading scale for supply chain visibility sensitivity

Score Description
0 No data is captured to support the target business decision
1 Some relevant data is captured, but it is incomplete
2 All data is captured but the accuracy of the data is unknown or known to be low
3 Data is complete and consistently biased (i.e. low quality but predictable)
4 All data needed to support the decision is captured, complete, consistent, and measurably high in accuracy.

 

Grading scale for supply chain visibility accessibility

Score Description
0 Data remains in the capturing systems with no attempt to integrate the data for later use
1 Data remains in the capturing systems, but processes allow them to be manually integrated for ad-hoc tasks
2 The solution integrates all the decision-relevant data, but not all of it is retrievable by decision makers.
3 Data is integrated and available to the decision maker, but not using the methods they prefer.
4 All relevant data is integrated and accessible by any relevant path the decision maker could use.
5 All relevant data is integrated, accessible, and the approach to integrating data is easily adapted
6 All relevant data is integrated, accessible, and the integration approach is self-updating when confronting new data types or sources

 

Grading scale for supply chain visibility intelligence

Score Description
0 There is no automated recognition from the solution that a business decision is needed
1 Sometimes there is recognition from the solution that a business decision is needed
2 The solution always knows that the business decision is needed
3 The solution’s approach to recognizing the need for a business decision is easily updated by users
4 The solution’s approach to recognizing the need for a business decision is self-updating

 

Grading scale for supply chain visibility decision relevance

Score Description
0 The solution has no explicit input to this business decision.
1 The solution is a required information source for the decision maker. A user decides how and when to make the decision.
2 The solution is a required information source for the decision maker. The solution decides when the decision is taken and the user decides everything else
3 The solution offers a set of action alternatives based on the event, or
4 narrows the selection down to a few, or
5 suggests one action, and
6 executes that suggestion if the human approves, or
7 allows the human a restricted time to veto before automatic execution, or
8 executes automatically, then necessarily informs the human, or
9 informs the human only if asked, or
10 The solution decides everything and acts autonomously,  with no notice given to humans except for debugging

 

The scales provided above may not be perfect, but they provide formal guidance about how to quantify what might otherwise be qualitatively graded. In particular, these scales are useful when teams of supply chain professionals must work together to evaluate, critique, or compare visibility solutions. In that sense any shortcomings in the proposed scales can be overcome by the team collectively agreeing on new grading guidelines. If it is agreed that visibility is better when it is highly sensitive, and that sensitivity should be quantifiable, the team can then promulgate and adopt guidelines on their grading approach as they see fit. The grading scale for decision relevance is perhaps the most robust because it has roots in human-systems interaction and design research (Parasuraman, Sheridan, and Wickens, 2000).

Once a team accepts these grading scales or adopts their own, the next step is to agree on how the individual metric grades are combined to achieve an overall fitness score for the visibility process. Many projects to build or improve supply chain visibility get off-track because they focus on functionality as a benefit in itself. This is simply not an accurate understanding of why visibility adds value. Features of a visibility system or process are only valuable to the degree that they fit into the targeted business decision. As an example, if a visibility process delivers beautiful visualizations of the meta-data, such as by plotting flows of materials and capital onto a map, this is an interesting feature. But if the targeted business decisions don’t have use for the feature, then it’s not going to add value to the company. The degree to which a visibility process meets the targeted business needs is what is being labeled the “fitness”, i.e. how good of a fit exists between the needs of the decision-making process and the output offered by the visibility process. On the high end would be a one hundred percent fit, where the visibility solution literally fully automates the decision at or above levels possible by a human being. At the other end zero percent fitness, where the system adds nothing meaningful to the decision making process.

Later researchers may introduce more nuanced formula, but for now it’s proposed to take a naïve view and assume that each of the four metric scores should simply be added and the sum divided by the maximum possible score so as to have the fitness percent score. Decision relevance is the most important metric, since it measures the most important step in the visibility process: decision interruption. But it doesn’t seem necessary to add a weighting to this metric because the associated grading scale goes to 10 points compared to the others which go to four or six, so it is roughly double the value of any one of the other three metrics already. In short, it’s suggested to use the grading scales to assign a score to each metric, then sum those scores and divide by 24. This gives a percentage of visibility fitness to the targeted business decision, indicating how well tuned the steps of the visibility process are compared to their ideal standards.

Ideally the scorecard could be used to compare visibility fitness across multiple solutions or options. This is where the cost metric comes in to play, as it factors in to the efficiency of the visibility process as a whole. What is suggested here is a specialized kind of cost-benefit analysis. From the total solution perspective, visibility efficiency would be the percentage of fitness achieved for a given investment by the company. When multiple options for how to deploy or improve supply chain visibility are available to a company, the company can evaluate the fitness percentage for each option and then plot fitness against cost to identify the efficiency frontier. This approach draws on non-parametric data envelopment analysis techniques, but the output is extremely easy to understand even by persons not directly or deeply involved in the evaluation. As an example, here is a fictive case study which traces 10 visibility options as they are graded at the process-step level, then for overall fitness, and finally contrasted against one another in terms of efficiency.

 

Example scorecarding of ten visibility solutions

Solution Sensitivity Accessibility Intelligence Decision Relevance
A 2 2 3 1
B 3 5 0 5
C 1 3 2 3
D 2 4 2 4
E 3 5 2 3
F 4 2 1 5
G 2 2 3 2
H 2 5 3 2
I 4 4 0 2
J 3 3 2 3
K 3 3 1 2

 

Example fitness percent of the ten visibility solutions

Solution Sensitivity Accessibility Intelligence Decision Relevance Fitness
A 2 2 3 1 33%
B 3 5 0 5 54%
C 1 3 2 3 38%
D 2 4 2 4 50%
E 3 5 2 3 54%
F 4 2 1 5 50%
G 2 2 3 2 38%
H 2 5 3 2 50%
I 4 4 0 2 42%
J 3 3 2 3 46%
K 3 3 1 2 38%

 

Fitness and cost of the ten visibility solutions

Solution Sensitivity Accessibility Intelligence Decision Relevance Fitness Solution Cost in Thousands of USD
A 2 2 3 1 33% 850
B 3 5 0 5 54% 1,120
C 1 3 2 3 38% 775
D 2 4 2 4 50% 915
E 3 5 2 3 54% 1,300
F 4 2 1 5 50% 1,050
G 2 2 3 2 38% 925
H 2 5 3 2 50% 890
I 4 4 0 2 42% 790
J 3 3 2 3 46% 860
K 3 3 1 2 38% 900

 

 

Tradeoff Frontier

The last graphic shows how the ten visibility options create a best-tradeoff frontier regarding fitness and cost, also known as the efficiency frontier. Obviously it is of limited use to compare high and low cost options in terms of cost alone. But this graphic and the associated approach allows for a simple and visual display of tradeoffs which occur between cost and visibility fitness. The dashed line shows the efficiency frontier (a concept borrowed from the data envelopment analysis domain). Solution options on this frontier represent the best tradeoff between cost and visibility fitness. Solution options which are enveloped by this line are not as efficient, and hence they should not be considered further. Another way to say this is that the solution options not on the frontier will always be dominated by another option. In this example, option B shows such domination. For the same cost, one could go with option H and have higher visibility fitness. Alternatively, one could achieve the same fitness as option B but at a lower cost by selecting option C. In either dimension (cost or fitness) the option B is dominated by another solution, therefore it should never be selected. Although this same measurement (fitness / cost) can be shown in a tabular view, the graphical view is simple to make and easy for decision makers, stakeholders, or other interested parties to understand. It goes a long way towards depoliticizing the removal of certain visibility options during real projects. But in most situations there will still be multiple solutions which form the fitness-cost frontier, and then it’s a question of organizational priorities to decide what level of cost and fitness is right for the company. This is valuable because it converts the problem of evaluating differing visibility solutions from a puzzle, dilemma, or paradox in to a tradeoff. See the figure below, derived from work on problems as strategic tensions in organizations (De Wit and Meyer, 2005), for a comparison on how these kinds of problems are evaluated. By reclassifying the problem as such, it (a) eliminates dominated solutions and (b) allows the organization to have fruitful discussions on where they should be on the tradeoff line, while knowing there is fundamentally no perfect answer to a tradeoff problem.

Strategy

Procedurally, here is the suggested way to use the supply chain visibility fitness scorecard. First, list the targeted business decisions which the visibility solution or solutions are targeting. If multiple visibility solutions are being compared they must be targeting the same business decisions. Otherwise comparisons are not informative, in the sense that the “appleness of an orange” or “orangeness of an apple” is not informative. The diagram below shows a mocked up scorecard where the evaluator or evaluation team can list out the decisions being targeted in the column on the far left. Just listing those decisions (and agreeing on them among the group involved) should drive the process towards better results because it focuses on the expected impact of the visibility rather than the visibility features in isolation. Create one of these pages for each visibility option being evaluated, but as described previously they must evaluate based on the same targeted decisions. At the bottom of the page is a field for the total cost of this solution, and the overall fitness score. Using one sheet per solution, evaluate the four metrics for each decision and then add their scores and divide by 24 to get the fitness percentage and place it in the column on the far right. After all options are evaluated this way, plot the solutions in terms of overall fitness percent on one axis and the cost of the solution on the other axis. Finally, draw a line to indicate the efficiency frontier, and then eliminate any solutions not on the frontier. Those options which remain can only be selected in terms of organizational priorities, as they represent the best possible tradeoff in terms of effectiveness for the investment at different investment levels.

Supply Chain Visibility Scorecard

To summarize, the supply chain visibility scorecard is a novel framework for evaluating the effectiveness of supply chain visibility, including the four steps in the visibility process. It fills an important area of assessment which has not been well studied up to now in the theory of supply chain visibility. It also has the benefit of providing decomposed metrics that align to the supply chain visibility process steps, which supports fine tuning of existing or proposed visibility solutions because the evaluator knows what part of the visibility process is contributing to a higher or lower fitness score. In the presence of multiple visibility solutions, or possible variations on a single solution, evaluators can introduce the dimension of total solution cost and compare it to the visibility fitness percentage in order to remove dominated options. In particular, this is useful during technology or strategic partner selection processes or periodic supply chain improvement projects. By adopting the data envelopment analysis methodology of plotting the fitness compared to the cost, an easily understood graph is available to share with colleagues, which tends to increase transparency about why certain options are favored over others. In a concise format, the steps to using the visibility scorecard are described below.

Steps to Use the Supply Chain Visibility Scorecard:

  1. For each visibility solution option, create a scorecard
  2. Add the list of business decisions which should be improved by supply chain visibility, and ensure the decisions listed are the same for all options being evaluated
  3. After studying the solution design, and using the grading guidelines provided, give each business decision a score for each category. The grading guide can be changed to provide more or less weight on certain areas, as long as the same guidelines are used by all evaluators and for all solution options.
  4. Sum the scores by business decision and divide the sum by 24. This is the “fitness” for the solution compared to the ideal support for the business decision
  5. Average the decision fitness percentages and add to the solution the expected costs
  6. Plot the relationship between fitness percentage and solution costs and then eliminate any options which are strongly dominated by the efficiency frontier.
  7. The remaining options represent the frontier of tradeoffs between fitness and the solution cost. Differentiating between these options requires assessing the organizations priorities.

 

Before closing the topic of the supply chain visibility scorecard, we should note that any visibility project can have external considerations in addition to the fitness to cost ratio. For example, there may be two visibility solutions under consideration for a company, but one of the options involves an external provider with bad financial health. The risk that the partner goes bankrupt and stops the project is not included in the visibility scorecard. This doesn’t mean that the financial health of the provider isn’t important, it just means that the scorecard is only targeted as specific visibility results achieved by the visibility solution. Professionals who are steering key visibility projects in their organizations can and should continue to look at the larger business environment for risks, synergies, conflicts of interests, and so forth.

Leave a Reply