This paper discusses the unique problems faced by a vendor of so-called "commercial off-the-shelf" (COTS) or "shrink-wrapped" software attempting to create products which incorporate components supplied by others, and the special sort of trust required of those components.

A case study is presented, highlighting existing trust issues. Several existing trust models are reviewed briefly. The requirements of a new trust model which can meet the needs of the various parties are outlined.

Finally, a proposal is presented for a model based on an "auditable automatable contract" or AAC, which can lead to higher degrees of trust at an acceptable cost. This model is compared to other models used in other parts of the industry, and some key potential advantages and difficulties are discussed.

A vendor of so-called "commercial off-the-shelf" (COTS) or "shrink-wrapped" software faces a number of unique challenges, particularly when compared to the more common models of bespoke or contract development, and customisable vertical applications.

When creating a product, a feature set must be selected which will capture a sufficient market share, but there is no fixed specification. Individual features can be added or dropped as development proceeds and deadlines approach.

A substantial investment is made into the development before any return can be expected, but when the sales start to flow the profit margins are exceptionally high. Development is expensive, marketing is prohibitive, but if you hit a winner, the rewards are enormous.

The product is likely to be technically advanced and will be called upon to work correctly in many different environments in the hands of users ranging from guru to moronic, but the selling price is too low to allow for individual support or even bug-fixing. There is high technical and commercial risk and a real possibility that the initial vision cannot be fulfilled, but the work is exciting, staff become highly motivated and routinely deliver miracles, so it all seems worthwhile.

Given these difficulties, an enterprising vendor seeing a niche for a product will always be on the lookout for tools to reduce development costs and lower the risk profile. Even better, there may be a component which can be incorporated into the product, do a large part of the work, and reduce the development effort substantially.

The question is: how can one company develop and present such a component, so that another company can purchase and come to rely on that component? In other words, how can the component be trusted?

My company, POWERflex Corporation, has been placed in this position on more than one occasion. Our experiences are salutary.

POWERflex Corporation is the developer and vendor of a fourth generation language (4GL) called PFXplus. Developers who use PFXplus can create highly sophisticated, large-scale commercial applications for Windows, MS-DOS or Unix. We have several customers with applications of more than 1,000,000 lines of code.

Developers are increasingly interested in connecting their applications (written in the PFXplus language) with other systems and tools. We saw an opportunity to develop two such products: a Crystal driver and an ODBC driver.

A Crystal driver allows users to generate reports on PFXplus data files using Seagate Crystal Reports, a widely used Windows package. It was developed using source code supplied by Seagate. The source code was of less than ideal quality, and the documentation seriously deficient, but with some persistence we were able to create a driver of commercial quality which is now highly reliable.

An ODBC driver allows users to manipulate PFXplus data files using SQL and a wide range of Windows applications and tools. Writing an SQL query engine and handling all the details of the ODBC driver interface is a large development effort, beyond our resources. After some investigation, we found 3 vendors of ODBC development kits, and selected one.

An ODBC driver development kit includes a component which comprises an ODBC interface and an SQL query engine. This component is self-contained, accessed through an opaque interface and incorporated into the released product.

The project has been disastrous. Although the initial effort to develop the driver was quite small (less than 3 months), we almost immediately began to run into defects in the driver component. So far we have notified in excess of 30 defects to the supplier, of which at least 5 qualify as "show-stoppers." The supplier either refused to take us seriously, or was unable to correct the defects.

As part of our development process, we devised a suite of automated tests based on a simple scripting language. On several occasions we were dismayed to discover that a new release which corrected earlier problems, also introduced new defects or changed existing behaviour.

With considerable effort we were able to work around several of the defects, and when they eventually (8 months later) fixed some others, we released our product. A number of customers were able to use the software. However, for the first and hopefully the last time ever, we had customers complaining and even returning software because of defects which we could not rectify. We believe this has hurt our reputation significantly.

Clearly, we made a bad decision choosing this component and putting our trust in it. The question is: what should we have done? We still do not have the skills or resources to develop an ODBC driver from scratch, but we and our customers need one. Other companies have those skills, but we don’t know whether we can trust their software. How can we find out?

There are several models by which components can currently become trusted.

Buy your components from Microsoft, or IBM, or Oracle and take the reputation of the company as a guarantee that they will work correctly. The reasoning is that large companies have the resources and skills to build good components and get them right.

There is some truth in this, but also risks. Large vendors have so much market power and so little interest in your business that they may drop the product, ignore defects, change the specifications without notice or move into your market. More importantly, the components you need to use are often not available from these vendors.

Use components which are the market leaders, even if they come from smaller companies. The reasoning is that the product became market leader because it is the best, and the company is making enough money to fix any defects.

Actually, a product becomes market leader by being first, not best. Leading products provide "good enough" capability to the majority of users, but there is no guarantee that the specialised requirements of a component used in a specific context will be met. If the leading product has missing features, or unacceptable pricing or licensing conditions, this option too may be ruled out.

Devise a specification and test the component thoroughly against that specification before relying on it. Even better, test a number of candidates and choose the best match.

This is an expensive approach, but for companies with the expertise and resources to do the testing, does yield good results. It requires a large investment in learning about the component in order to test it, which will be wasted if the component is not selected.

Devise a specification and attach it to the contract with the component supplier. Effectively it means turning the relationship with the component supplier into one with a sub-contractor.

This is another high-cost approach, involving not only the cost of devising a specification but also lawyers and individual contract negotiations. It is highly effective in suitable circumstances.

Buy only components which have been independently certified. The idea is that a vendor with a component submits it to an independent testing firm, which conducts a series of tests and, assuming successful completion, issues a certificate.

This model has been successfully used in only a few very specialised situations, such as development of Windows device drivers, where the requirements are clearly specified and the owner of the specification (Microsoft) mandates the process. It is unlikely to be adopted widely because of the general lack of clear specifications and the expense involved.

Another problem which must not be overlooked is that the reputation of the testing firm becomes a major issue. There will always be a suspicion that the testing firm is being employed by the vendor, so will be biased towards the vendor’s interests.

Treat the original component as a "pre-release" version, and obtain a licence to acquire and use the source code. Then when problems surface, ignore the original vendor and fix them yourself.

This is a very effective strategy which we have used several times. Unfortunately, source code is not always available, or may be supplied at a price or on unacceptable licence terms. It is currently the only trust model we believe we can rely on.

A well-written manual provides some clues as to the quality of the company and its products, as well as defining the behaviour of the software. In our case, it was the quality of the manual which misled us into placing our trust in this component.

It may be possible to get references from other customers. Inevitably these will be chosen from the success stories rather than the failures, so may not be a reliable guide.

Evaluation software can be a guide, but "crippleware" is not a basis for trust, and evaluating the full software is equivalent to "Test It Yourself."

At best, these factors should be taken as supporting evidence and not the primary source of trust.

Clearly there are significant barriers to the use of components in creating COTS software, characterised by a high level of risk. To vendors already facing considerable technical and market risk, this will often be enough to prevent a project from being undertaken where use of third party components is involved.

Certainly there are situations where the availability of a market-leading component from a major player is available with certified behaviour and source code, but this is very much the exception! The only components we trust at present are those for which source code is available.

Something has to change if trusted components are ever to be widely used in developing COTS software.

First, it must be assumed that there is no fixed, predefined specification for the component. Different suppliers are free to devise their own feature sets in an attempt to attract market share. This does not preclude them from conforming to a published standard (such as the ODBC interface discussed above), but it is assumed that key features of each component are unique to the vendor.

It can be assumed that a prospective COTS vendor reviewing candidate components has a considerable amount of flexibility and can adapt to wide variation in specifications. This will always be within certain limits and there will always be certain key requirements which must be met. The challenge is to determine what the component does, estimate the amount of effort it will save versus the effort in connecting to it, and ensure that all key requirements are met.

Unfortunately it can also be assumed that some of those key requirements are unspoken and undocumented. For example, in the Windows environment there is an expectation that a component will conform to standard Windows interface behaviour; will work correctly in all versions of Windows; will work correctly with common third-party applications such as Microsoft Office or Crystal Reports; will degrade gracefully if deprived of resources such as memory or disk space; will never hang or crash; and so on. These things are hard to specify or to test.

It can also be assumed that once a purchaser selects a component, any room for flexibility drops sharply. It is highly likely that during the lifetime of the new product, defects in the component will be uncovered and corrected, and new features will be added. The purchaser needs to be certain that all behaviour relied upon in the component as originally evaluated and selected is preserved in future releases.

It is reasonable to view the process of selecting a component as akin to negotiating a contract. The supplier offers a set of features for a price. The purchaser selects the best match to its particular requirements. After a brief negotiation, a contract is signed. The supplier can then be held to its agreement, both morally and legally.

The problem then is how the supplier can disclose the behaviour of a component in sufficient detail that a prospective purchaser can understand what behaviour to expect, compare it against requirements and rely on that behaviour being delivered. At the same time, the supplier does not want to give away valuable proprietary information.

This problem is well known in the financial arena, and many solutions have been devised with names such as "audited financial statements" and "due diligence." Although imperfect, they are vastly better than what is now available for software components.

The concept of a contract is increasingly being applied to software too, but here the contract is about technical performance and enforcement is by technical rather than legal means.

The proposed model for conveying trust in a component is based on the disclosing verifiable information in the form of an auditable, automatable contract (AAC). This is a technical contract, rather than a legal contract, and it rests on two key points.

Assume that a way to create an AAC exists. A supplier with a vision for a new component sketches out the basic functional specification and builds a business case for the project. Once it receives approval, work is begun on the detailed functional specification, architecture, packaging, manual, marketing materials and AAC. As each stage of analysis and design is completed, the AAC is part of the deliverables. This ensures that the design contains only testable features.

As development proceeds and the code takes shape, from the first point at which it becomes executable, it is tested against the AAC. Since the AAC can be automated, this testing and verification can be performed on every iteration, after every build, at very low cost.

Finally, when the component is nearing readiness for shipping, the supplier again ensures that the completed component verifies against the completed AAC, and that the AAC accurately reflects the original vision. The AAC is made available to prospective purchasers who need the additional level of trust it provides. In time, an AAC could be a standard document attached to any trusted component.

The AAC should cover all essential functional and non-functional aspects of the component’s behaviour which the purchaser can rely on. The purchaser buys what the contract promises, no more and no less.

Functional requirements can be specified in several ways. It is usual to focus on specifying the behaviour of interfaces, with the most familiar being programming by contract based on pre-conditions and post-conditions and possibly augmented by invariants and progress conditions. Limitations in this approach have led to others less familiar, including histories and statement specifications.

Specification of non-functional requirements is less easy to do, and is typically expressed as a service level, which the component must meet or exceed. The service level may include guarantees of performance, resource usage, time and space complexity, throughput and latency, capacity, test case coverage and so on. All of these are verifiable at the time of purchase.

It may also include other guarantees such as availability, mean time between failures, mean time to repair and failure modes, which require ongoing monitoring if compliance is to be checked.

However, the AAC should not specify unnecessary or unenforceable requirements, or requirements requiring interpretation by the courts, or anything where the cost attributable to the specification outweighs the risk of it remaining unspecified. An AAC should be as simple as possible, but no simpler.

Currently, there is no tool or methodology which can create an AAC. There are several which go part of the way and many promising areas under investigation, so the prospects are good.

The best candidate for a human-readable specification which can be automated is currently the Object Constraint Language (OCL), supported by a specification written using the Universal Modelling Language (UML). The constraints in OCL refer to specifying required behaviour using programming by contract techniques. These constraints have been designed in such a way that they can be automatically converted into a convenient programming language for automated testing.

If OCL is not suitable, there are other candidates. Assertion Definition Language (ADL) is one; others have been devised as commercial products.

In any case, OCL alone is not sufficient to automate testing. Additional tools are required to formulate test plans, group and manage test scenarios, prepare and administer test data sets, administer tests, record and report results. At least some parts of these will have to be in the AAC.

Again, there are a number of tools which offer all or part of the required capability. They include TET and several commercial products. A literature and product review is needed to establish the availability and suitability of existing tools.

There are several obstacles to the concept of conveying trust in a component by attaching an AAC to it.

The first and most obvious is that currently there is no tool or methodology to create an AAC. Unless there are solid grounds for believing that one cannot be created, this should be treated as a challenge rather than a deterrent.

The second is that even once such a thing is created, perhaps no one will use it. This is a complex issue with many commercial and technical aspects to consider. At present it would seem that an AAC might be more valuable to a component purchaser than a supplier. A supplier might need encouragement by financial reward, technical assistance or favoured contract status. Alternatively, the prospect of being held legally liable for defects might provide some impetus.

In the longer term, the satisfaction of knowing that a completed component could be offered for sale and would result in contented customers, low support costs and much repeat business might be sufficient reward in itself.

The third and most difficult obstacle is to know what are the limits of an AAC, and what to do about required behaviour outside those limits. Most functional specifications lend themselves to being expressed in AAC terms, as do some of the performance-related non-functional requirements. Many service level requirements including some performance requirements are beyond the scope of automated testing.

Probably the best answer is that even an incomplete but reliable AAC would be very much better than what we have at present, and would certainly have allowed us to avoid the situation we found ourselves in with our ODBC project.

In summary, the concept of an Auditable Automatable Contract for software components provides an attractive and a feasible way for prospective purchasers to gain a level of trust in a component. This is a critical step before trusted components can be widely used, especially for customers such as the COTS vendor for whom the risks and potential costs of existing components are very high.

There are significant difficulties to be overcome, both in devising the technology and in getting it adopted, but good grounds for believing it is achievable.