In order to achieve significant or breakthrough improvements in most processes we need a radical new approach. Most existing methods only yield incremental results. With the economic pressures of the Nineties and in the current business climate, many businesses no longer have the luxury of spending large sums of money in order to use traditional means to fully study problems and work out complete solutions.

We expect that the 80/20 rule will apply to most business and industrial processes. That is, 80% of the cost, defects, errors or problems are generated by 20% of the process. This 20% contains the most complex portion of any process; this portion tends to be the least fully understood or addressed in any attempt to automate or systematize the process. Most attempts at automation of processes have dealt with the "easy" 80%, leaving the tough exceptions to human beings. To achieve a 5 or 10 times improvement in any process, we must address this remaining 20%.

How are these 20% handled today? They tend to be managed by the masters or experts in the specific domain. Many companies will acknowledge their need for expert support for their customers and within their own processes. They have not done much to help these people with tools because it has been too difficult or costly using traditional methods. We need to capture this expertise and build on it without impacting the current process. We also know that much of this expertise is being lost through downsizing activities.

The traditional way to capture knowledge was to have the domain master sit down with a systems analyst or knowledge engineer and, through a series of questions and answers, try to develop rules or logic to address their domain. This was an expensive process because it tied up the master and required at least two translations of the information gathered. It also resulted in false starts and required extensive reworking as several iterations of the knowledge base or program logic were needed when the real world testing revealed shortcomings.

The major cause of this rework was that the data was gathered in an artificial situation and the scenarios used to develop the rules did not include all of the possible variables. Lack of "common sense" was often cited as the symptom of this problem in such computerized solutions. Often, the solutions simulated this artificial environment and failed to address the root causes.

The Master's Apprentice (TM) uses a different approach to capture the knowledge. The domain master is placed in the real world decision-making situation and their reactions to the complete set of variables are captured to train rather than program the computer system. This knowledge is captured both through iterative prototypes in a repetitive object-oriented development cycle and through knowledge capture using both rules and "learning" neural networks. This approach accomplishes two things: first, the master assumes ownership of the training process and the results reflect on their ability to train the "apprentice"; secondly, the system will learn in the real environment and be capable of adjusting to subtle changes in that environment.

The key breakthrough in The Master's Apprentice is the idea of subsetting the problem the way a human expert would, and providing immediate feedback via prototypes that implement exactly the master's direction, to improve complete and accurate communication of the real requirements. This is how an apprentice is trained in most business processes. They learn how to handle all the various exceptions by consulting expertise when they actually encounter the situation.

The human master will usually have an initial impression of overall conditions possibly including things like sounds and smells (in a manufacturing process environment) that are not normally considered in automation projects. Based on this information, the master will observe another set of conditions followed by another with perhaps some intermediate actions along the way. In this way, the master will instinctively monitor the critical subset (e.g. 20%) of the inputs most appropriate to the current situation.

Traditional automation approaches would consider all the possibilities first, rather than decomposing the problem into obvious chunks like the human master does. Tackling the whole problem takes longer, costs more, and has a higher likelihood of major errors creeping into the final system. There are countless similar examples from other business processes. For example, in a sales process, the master might observe tone of voice, eye movement, and body language, in addition to the words spoken by the customer or prospect.

By subsetting the diagnosis, and applying simple rules before the more complex ones, the system can reach many conclusions much faster and with less involvement by the master. As the system becomes able to effectively handle the simpler diagnosis, more complex work can then be delegated, while the master builds their confidence in the system at the same time. This is the same as how a human apprentice learns their trade. Hence the name: The Master's Apprentice (TM).

Specific implementations of the system may use a combination of object-oriented analysis, design, and rapid prototypes, as well as expert rules and an adaptive hierarchy of probabalistic neural networks. This part of the system complexity is masked from the Master, just like the full complexity of a human apprentice is not apparent in the context of the specific job at hand.

By apprenticing with multiple masters, it is possible to make the system embody work practices and knowledge that no one in the immediate area has. This opens up the opportunity to create business change by deploying The Master's Apprentice to create improved work practices and offer new services where no expertise is currently available. As an example, the relationship marketing workbench deploys the sales process (or "funnel") to drive future engagements with prospective clients, while employing a set of rules to build the recommendations for additional services from the information gathered from the client during the process. Users of the system perform the new business processes and eventually gain expertise in these work practices from the system itself rather than needing to apprentice with a human master.

Copyright the authors and/or Keith Cowan