We Need to Talk: the Role of Dialogue in Future Information Exchanges

Supply chain visibility, when this article was written in late 2010, faced several serious challenges to implement. One of those challenges is the need to exchange data while ensuring a common meaning for the data elements. This is part of the visibility framework (see link here), which means any supply chain visibility practice must overcome this challenge. In the following article, I give an idea of how this challenge was being solved in 2010, and the limitations current methods involve. At the end of the article, I suggest a paradigm for future data exchanges which, I believe, offers net benefits over our current model.
In most visibility solutions, there are many original sources of data. These sources are predominated by manual entry at some stage but include degrees of automated data capture, such as temperature, time, or GPS coordinates. Generally speaking, interfacing via XML or EDI is the best-practice for getting data from system X to a central visibility platform. Let’s get concrete and look at a quick example: a request for a spare part may be keyed into a sales system by a call-center operator. The sales system saves the order to its database. Either as a timed routine or via an event trigger an EAI or other data transformation tool takes the new data and formulates an interface file according to pre-defined rules, usually called an interface map, schema, or specification. The interface file is then transmitted outside of the call-center’s firewall through a pre-determined protocol and delivered to another system on the side of the visibility platform. There are many transmission protocols, but they all have to be predefined. Once the file arrives at the visibility platform’s server it is processed through one or more layers of information mapping. For example, one layer may simulate acceptance to inspect for security risks, and then pass the file onward for actual import attempting. Generally, just as there was an EAI layer at the call-center, there must be some system which actively relocates data from outside the firewall to inside the firewall and prepares it for consumption. Finally, the file is consumed by the visibility platform, which causes the incoming data to update existing database tables. Based on these updates new actions may be triggered, including the creation of outbound interfaces to other systems.
If this sounds complex, that is because it kind of is. The process above relies on pre-determined specifications to allow two systems to exchange data. Some of the predetermination is about security and identity: if we accept incoming data we want a way to verify the sender. But the largest part of the predetermination is to enable consistent data ontology, or in less fancy language, for the data to have the same meaning between systems. Let’s go back to the example above. It’s absolutely mission-critical that the part number being ordered is understood by both systems to be the same thing. I don’t mean to say that one system may say “ABC” and the other could receive “123”, which would be a transmission or processing error. I mean to say that the definition of “ABC” is clear to both systems. Often times, this poses significant challenges because a parts supplier may call its item “Part ABC” and have four different colors of the part. The reseller of the part keeps the part number unique for each part-and-color combination, so they use “Part ABC-Black” or “Part ABC-Red”. Before the two systems can begin exchanging data, they need a common data ontology, a kind of data dictionary, which both systems will use. In reality, no one changes their system to use the data dictionary of another company, which is why the intermediary layer is needed. It performs data translation and formatting to enable cross-system communication.

Okay, so how often is this really done? A lot actually. Mid-sized retailer point-of-sale systems can be transmitting data to a visibility platform at rates of 10,000 – 20,000 transactions per second. Each transaction is being pushed through a workflow that involves translation into and out of a predefined interface schema. Even with streamlined technology, these data volumes pose challenges. I don’t mean speed challenges, such as the delay of uploading we may experience on home internet connections. One of the greatest barriers in the deployment of new supply chain visibility platforms is the change-management of getting all the data sources to learn and correctly implement the interface paradigm.
At face value, our best practices may not seem so bad. An IT team from the anchor organization (see my post on supply chain governance) approaches their partner and proposes an interface schema. The schema includes the expected format of data, parameters for making contact and confirming identity, and finally the schedule or trigger for initiating a transfer of data. The recipient’s technical team then puts the schema in place and the two teams test it for quality. It could be so simple…

The challenge is that diversity of partners causes a rapid proliferation of interface schemas, which increases cost, risk of failure, and operationally resource requirements. Because of these problems, only the most powerful organization in the supply chain can really enforce its preferred data interface schema. In supply chains that lack a sufficiently powerful member, or for organizations that participate in different supply chains as both a weaker member and a stronger member, this becomes a disaster. Lastly, this approach causes supply chain IT professionals a low-grade annoyance regardless of what company they are in, because they can sense that there is a better and as yet un-found way of conducting data exchange. So let’s go back to the retailer POS volumes above. What if the visibility platform is supporting not just one but hundreds of retailers? Now the stakes are very high to ensure that there isn’t schema proliferation.

One way the supply chain field has is combating interface schema proliferation is to introduce “standard” schemas. An EDI 315, from ocean carriers, is an example of an industry consortium adopting a standard schema in order to reduce individual development costs and risks. But these kinds of schemas have largely failed to live up to their full purpose. The EDI 315s, as a good example, are “standard” in construction but not in ontology. What I mean to say is that I can decode any EDI 315 message using off-the-shelf software. But the “meaning” of the data isn’t self-evident. I see a PO number, but am not sure how the PO should relate to destination ID, for example. Is it many-to-many, one-to-many, etc? Are there values which must be globally unique? This kind of information is still left to individual exchangers of data to map and define. And it is challenging… Imagine now that you have 1,000 business partners in 50 countries around the world. They speak 15 languages and are diverse in terms of technological sophistication. Some are using Microsoft technology; others use open-systems architectures. Some may not even have in-house IT capacity at all. Imagine the effort necessary to rollout a data interface schema. At a minimum, we’d expect one day of work per partner being onboarded and to assure the quality of the new interface (regardless of if it’s the partner’s time or the anchor organization’s time). So, at least 1,000 days of work, or about 5 staff-years. At a cost of $100,000 USD per year for a typical technical lead for interfacing, this represents a collective investment of $500,000 USD to achieve connectivity to a central platform. These numbers are realistic. In some cases they may even be optimistic, such as if the data exchange is complex, or covers data that isn’t yet being captured, or if the platform is also sending data back to the partners as part of validation or tactical execution.

There is one more aspect which confounds this process. In the example above, it’s expectable that innovation in the underlying technology along with changing business needs would force another rollout within, say, five years. Upgrades to interfaces must be undertaken jointly, since it usually requires a change to the interface in order to have value for the business. The expectable need to upgrade interfaces once they exist is a reason people resist getting into them to begin with. It becomes a commitment that must be renewed via expense in hardware and resources.
In summary, the current data exchange methods:

  • Tend towards industry-wide protocols for data file structure and transmission
    • Require elaborate, predetermined definitions of the data elements to be made one-on-one between each sender and receiver pair
    • Usually necessitate a data translation layer on each side of the transmission to put data into the predefined format and then take it out again
    • Are at least moderately costly to rollout, because of the human effort involved in design and testing, and because of the need for hardware and software to execute
    • Allow for only minimal local upgrading… generally, the sender and recipient have to upgrade together to get advantages of new technology or business processes

Dialogue: another paradigm
Let’s go outside the box for a second… how would a call-center clerk get the spare-part order to the wholesaler if the system interface didn’t exist? Probably by making a phone call. While on the phone, the clerk and the wholesaler’s staff would have to discuss the order and resolve the difference in item number, along with any other ambiguities. For example, the call-center clerk might say: “The part has to arrive by tomorrow at 12:00”. And the wholesaler may reply with “12:00 noon or midnight, and is that in the time-zone of delivery, your time zone, or my time zone?”. These kind of iterative question and answers are what we’d call dialogue. It’s one mechanism humans use to exchange information and assure the recipient has the same (or about the same) definitions for the terms as we do. Dialogue is filled with heuristics to ensure transfer of data as well as data meaning, including the ability to question each other and redundancy in content. Human speech is about 65% redundancy, in one form or another. For example, this statement: “I’m so sick, my stomach is killing me” has a redundancy in it. We add redundancy in order to ensure that the meaning is transmitted correctly. A redundant phrase allows the recipient to miss or misunderstand part of the message but still have complete meaning. It’s pretty clever, basically, and it’s a good model for interfacing data.

Most current interfacing methods are static, in the sense that we pre-define the schema and its underlying data ontology once and then use it thousands and thousands of times. So, thinking through the efforts, it’s a matter of investing time up front to allow fast, simple processing over the incredible volume of transactions that will take place. In this way, a predefined schema allows systems to communicate millions of time more efficiently than if they had to conduct human-like dialogue. An analogy might be the difference between using taxis and using trains to enable transportation of people. Trains take a long time to setup, and are relatively difficult to transform later. But, they are certainly more effective at moving large masses of people if the flow is going to fit the train’s route design. Trains, therefore, are akin to our current best practices around data interfacing. Taxis, on the other hand, offer extreme flexibility. They are on-demand, granular services which can be re-configured to meet changing needs. They could potentially be used to bring hundreds of people from point to point, like a train, but at much higher costs. In this way, taxis are similar to a dialogue-based approach to data exchange. If attempting the same transport task, there will always be a clear winner between taxi and train because of their inherent attributes. Data exchange may not have such an obvious break-point between the two potential methods, but many times one will be clearly superior for a data transfer task.

I think there is a potential hybrid approach that is available for data exchange. It would involve a human-like dialogue at the beginning which may in fact be a mini-dialogue to confirm that a previous paradigm is being employed, followed by the schema-leveraging that typifies current interfaces. The dialogue that begins the exchange would cover important aspects such as:

  • Who the two sides are (the identity problem)
    • How they are able to present data, meaning an offer to be able to present data in formats X, Y, and Z
    • The ontology that the sending system is applying to the data, along with any available ontologies that might be useful for the recipient
    • A reason from the sender for why the data is being passed now, or likewise from the recipient if the exchange begins with a request
    • Agreement (commitment) to pass data is certain forms
    • Feedback on each other’s process, both as a “handshake” to confirm completeness but also to support self-improving dialogue models

One aspect of human dialogue that is useful but could be much more effective in machine exchanges is the ability to reference past dialogue in order to jump start the exchange of information. A phrase like “It went well” is acceptable in conversation because all the necessary data ontologies (what is the subject, how to define “well”, and when the action occurred) are being retrieved from memories of earlier dialogue. Our machines could do this much better, resulting in very small dialogues in preparation for exchanging large quantities of data.

What I’m really advocating here is a form of artificial intelligence: two agents, with limited but significant autonomy, pro-actively contacting each other to exchange data. This model of data exchange makes the assumption that most data sources will very soon have an attached artificial intelligence (AI) capable of this kind of work. The attachment between the AI agent and the data could be simply a program running on a server with IP access to the database. For example, I could build an AI agent on my laptop which then runs an ODBC connection to a Microsoft SQL server. Assuming today’s technology, this model would use a slower execution speed to exchange data between systems than the data interfacing we normally employ. But, in reality, we’re designing towards future hardware and anyway it’s not the execution speed which normally poses a bottleneck. Except for specific, high-volume situations, the occasional delay of 1-2 seconds while the AI agents figure out a new schema or ontology, would not pose a serious problem. In high-volume but low variability situations (such as with Point of Sale data), the agents would be adding a negligible dialogue to the front of the exchange because they would have developed already the requisite definitions.

This model gives us a few things we can’t achieve today; things we want and would find valuable. First, it allows new connections to be established with very little cost. Perhaps all that would be necessary would be the delivery of an agent’s IP address, something like passing along an email or phone number between people. From that “introduction”, the agents would then spend time running through their own routines for getting to know each other, essentially conducting mutual training. But, being away from humans or new software or hardware purchases, this procedure is extremely low cost and fast. So, the first reason to consider the dialogue paradigm is simply that it could be far less expensive to create new connections.

The second reason to consider this paradigm is that it allows for the exchange of infrequent, or ambiguous, data in ways which simply aren’t cost feasible today. Today’s paradigm requires formal definition for all data elements being passed. Future AI agent models might use dialogue, induction, and deductive reasoning to properly classify ambiguous or contradictory data. One of the many applications for this new capacity would be the ability to receive data that contradicts previous assumptions and then for the agent to actively contact the agents possibly affected to conduct verification or warning. Here is a concrete example: over thousands of iterations agent A receives data from Agent B about product purchases at a train ticket kiosk. In the data, there is always the same pattern of line-item costs summing to equal total credit card charge. Then, one exchange of data breaks the pattern. Agent A doesn’t notice the break in pattern because its ontology doesn’t include the need for the pattern to exist. But, when passing the data onward to an accounting system’s data exchange agent (agent C) the issue is discovered. Why is it discovered here? Perhaps agent A’s data ontology model is focused on relationships between data in other areas, such as transaction time and location. At any rate, it doesn’t include a relationship between line-level costs and total credit card charge. But, agent C’s ontology can be (and would be) different. If agent C discovers what it believes is a breach of consistency between ontology and actual data, it could alert a human expert. Another potential design is to have Agent C alert agent A that data passed was inconsistent, and explain the inconsistency. Agent A can either learn from the exchange, or not. But, as part of Agent A’s behavior it may take steps to contact agent B and request Agent B’s ontology. After all, Agent B is the source of the data and its ontology didn’t get broken by this supposedly bad data. This example demonstrates the kind of options we would have. Using human-like dialogue, but more exacting and powerful memory and computational power, this paradigm of data exchange has enormous potential.

Finally, a benefit of this paradigm is that it allows any single node to upgrade at will, with immediate benefit, and without much need to consider the other partners being communicated with. This is because the relationship between agents would be mostly self-organizing (and self-healing in cases of disruption). As long as agents speak the same “language”, and have sufficient capacity to train each other, they should be upgradeable independently. We see this, again, in human networks. A newly hired employee doesn’t know anything about the clients, the company terminology, etc. But, through a shared language, the other people teach the new-hire through the initial interactions. Just as an office needs a good mix of long-standing staff who have many experiences to draw on, and younger and more flexible staff, it could be possible that data exchange networks benefits from heterogeneous agents.
In summary, the AI agent model for data exchange would be:

  • self-organizing, allowing for uncoordinated changes to individual nodes
    • extremely low cost and low risk to add new nodes
    • capable of handling ambiguous or conflicting data
    • potentially self-improving
    • slower to process high volumes of the same schema (compared to predefined schemas as we have them today)
    • Open to different security risks, based on agents being tricked about identity of the other party, data being passed, or what constitutes “good” behavior, etc.
    • Would enable the sharing of more data (especially in terms of variety), but with a risk of mistaken data identity that is greater than current methods relying on predefinition

The Monday-Morning wrap-Up:
As with most of my articles, I’ll close with a look at how these concepts can be applied by decision makers in supply chain in their daily lives. With this article, more than others, the task is challenging since the discussion was looking at future potential paradigms. But, here are the concrete points to remember:

  • Data exchange represents a major effort for supply chain visibility practices. It is always necessary and rarely comes easy.
    • The best proven methods of data exchange in wide deployment are industry-standard interfaces with a custom data ontology (definition) agreed in advance between the sender and receiver.
    • It is very rare to find organizations who accept external data definitions in ways that over-ride their own. In reality, a translation layer (also called EAI) is used and must transform data from form and definition A to form and definition B.
    • Downsides to the current process include issues of startup time and cost, and with the need for “joint” decision making about node upgrades or changes.
    • Finally, alternate paradigms exist and will probably be enabled in the future as both hardware and artificial intelligence techniques are improved. These paradigms present significant advantages over existing methods.

Leave a Reply