Software delivery or use?

INDEX | ORGANISATION | TIME LINE | PEOPLE | STORY | RESEARCH METHOD | COMMENTARY | FINDINGS | HELP | BOTTOM
This section investigates why there were so many problems getting many of the users to effectively use the system. Were these problems because of the way the system was installed or were there other causes? Throughout the study data quality within the system was poor. Why was this?

    C.3.1 What affected the use of the system?
    C.3.2 Why was the quality of the data poor?


C.3.1 What affected the use of the system?

The attitudes of underwriters to using computers were widely regarded as one of the key reasons why the system was not used. In the UK, it did not seem to be an attitude to computers that was the problem but some underwriters did not see servicing as their business.

There was no lack of organisational commitment to the Multinational system but some senior individuals were not committed despite the use of prototyping. It is also true that system problems, of phase II particularly, caused people to lose faith in the system. However, the underlying reason for people not using the system was that it was not useful to them. In the case of the underwriters, the benefits were small and their lack of use meant that data quality was poor and this reduced the system's usefulness to other users.

Usage was also affected by the bad timing of the installation of phase I, insufficient consideration given to the resourcing of data loading and people's normal resistance to change. Training would have helped alleviate these problems but was very limited in phases I and II because there was a belief that prototyping reduced the need for training or a system manual. However, use of the system was encouraged by internal communications, which were generally adequate or good.

 C.3.1a User attitudes
 Organisation  
The attitude of the underwriters was widely cited as one of the reasons why the system was not effectively used by them. The difficulty of getting underwriters to use the system was raised in the IT Strategy in 1991 and the Post Implementation review highlighted that the problem had not really been addressed.

In the UK there is evidence that at least some underwriters were perfectly prepared to use computers. It was underwriters that wrote the systems used prior to the Multinational system and the underwriters adopted other aspects of Windows, such as email, introduced at the same time. Even Donald (who was regarded as one of the less enthusiastic underwriters concerning computerisation) said 'It is vogue to just moan about the computer and all the rest of it, but it's the easy thing to do, but most underwriters, I'd say, when it comes down to it will say that you've got to have one ...'. There was also a view that the various attempts at computerisation had left a poor image of IT in X but there is little evidence amongst the underwriters in the UK of this.

A more significant factor was the attitude of some underwriters to servicing the cases. Some of the underwriters in the UK believed in controlling their cases while others believed that getting the business was more important. This attitude may have been exacerbated by the decision to create Multinational Services; there was certainly a bit of a 'them and us' relationship between underwriters and Services. However, I think that the difference in attitude to the system exhibited by underwriters and Services was mostly due to the difference in the focus of their jobs.

 C.3.1b Commitment
 Individual  
Throughout the development of the Multinational system there does not seem to have been any lack of organisational commitment to the development. Nevertheless the importance of individual commitment in ensuring that the system was used is illustrated by the problems of America and Claims. Although a senior manager in America was involved in prototyping, and senior management on both sides of the Atlantic agreed on a plan for loading data, the data in America was not effectively loaded and the system had to be relaunched in 1995. Claims in the UK also seemed to suffer from no championship by an individual. There were still problems with Claims in the UK in 1995 although claims were being successfully downloaded in Holland because of the determination of an individual underwriter.

There was a view that one of the problems of phase II was that David was replaced by Jenny who was junior and less experienced. I am not convinced this was significant, both Jenny and Lucy from Services seemed to demonstrate commitment and enthusiasm for getting the system working.

 C.3.1c Prototyping methodology
 Method  
The prototyping methodology encouraged ownership by those that were heavily involved and explains the extraordinary loyalty to the system expressed by both Jenny and Lucy. Unfortunately this ownership did not extend to those who were less involved in the development such as the USA.

 C.3.1d System problems
 Belief  
The errors in the system meant that people, who were not confident with using it or were not convinced it was useful, found it easy to blame the system when they had problems using it even though there may have been human input problems. The number of bugs found in the early stages of version II were used by Jenny to alert X senior management to the system problems and probably made the reaction worse with some managers using the system problems as a scapegoat.

The launch of phase II was particularly catastrophic. It raised unrealistic expectations, reduced motivation and gave the underwriters a reason to be critical. Attitude to the system was contagious - underwriters saw others having problems and so were not minded to put using the system high on their list of priorities. In contrast, Services and Holland, who were more committed to using the system, were much more sanguine about the errors.

One of the advantages of the Post Implementation Review was that it seemed to focus the venom and vitriol but it also seemed to reawaken discontent so that even when most of the bugs had been fixed the users were still fed up with the system and management re-considered its long-term status.

 C.3.1e Functionality and usability
 Design  
One of the reasons why I believe the underwriters did not use phase I or II was that it was of limited usefulness to them. There is a marked contrast with the other facilities provided with Windows, which the underwriters did adopt.

The system was not useful as an information resource because much of the data they wanted to access was still held in manual files or was easier to access manually. In particular historical data about claims was not loaded. It was regarded as unlikely that the system would ever replace paper records. Very few reports were written for underwriters in phase I or II of the system and the absence of ad hoc reporting meant they could not write their own reports as they had with Q&A. Cases were easier to renew once they had been correctly entered into the system. However, loading the data was not easy because it was time-consuming to enter single bits of information about a programme. In phase I this was exacerbated by the speed of the system but phase II was still regarded as too slow by users.

The system was also limited in its usefulness to Services. The first installation of phase I was of no use to Services because it only contained quotes and clients. When the full version of phase I was installed it was not as useful as Q&A and this situation continued in phase II and had only begun to be addressed in phase III. The slowness of the system also impacted on Services because of the large number of screens that they had to use to process a single programme.

The problem with phase I and II was that they were designed from a management perspective but were not used because the data was incomplete. Underwriters were supposed to be entering some of the data but the system offered them little benefit. Services did use the system but found it less useful than Q&A.

 C.3.1f Data quality
 Method  
Without data quality the system was useless to management, for example in January 1995 they still could not use the system to report on their exposures to disasters such as the Japanese earthquake.

One of the reasons why the Claims aspects of the system were not used was because downloads could not match policy numbers because of the poor quality of the data in the Multinational system.

 C.3.1g Resource shortages
 Resources  
It is not clear that the resourcing implications of the introduction of the Multinational system were fully considered. By the time phase I was installed Multinational was under staffed and the introduction coincided with other major changes in the department. These staff shortages exacerbated the data entry problem and made it difficult to get people to attend training sessions. Moreover, the underwriters only had one PC between two of them, making it difficult to input data.

Installation in Services was complicated by the need to convert the data from the Q&A system. Services, who were already short staffed and with a backlog, spent time getting the data up to date in the Q&A system and then had a period of two weeks in mid June when they could not use any system.

 C.3.1h Installation
 Method  
The phased introduction of phase I was unfortunate because the first part of the system was of limited usefulness but it does seem to have been effective in introducing the technology to the users. It might have been better to have introduced Windows first and then had a proper launch of all aspects of phase I when they were ready.

 C.3.1i Aptitude of staff
 Individual  
There were mixed views about the importance of IT literacy or aptitude in getting a member of staff to use the system. There was a view that 'the further west you are the more IT illiterate you are' and that this explained why the UK and Holland used the system more than the USA. While there was some evidence that high IT literacy was useful, a critical factor was that staff needed to be prepared to persist and be interested in the system.

 C.3.1j Attitudes to change
 Change  
One of the reasons given for difficulties in getting the system used was that it involved change. Underwriters were 'sold the system' rather than having the implications of the change explained to them so they were unaware of the amount of data sorting out they would have to undertake. It is difficult to assess the importance of this factor but the increase in dissatisfaction in phase II caused by relatively minor changes indicates that it was important.

 C.3.1k Training
 Method  
Overall the team, perhaps encouraged by Tim, seem to have regarded prototyping and online help as substitutes for the more traditional approaches of training and user manuals. One of the features of phase III was a greater emphasis on training and user documentation. The relaunch in America was planned with an extensive training programme and training was included in the rollout to Europe. This was in response to a recognition that there had been inadequate training in phase I particularly in America. The timing of training for phase I in the UK was inappropriate because of the phased implementation but there seems to have been a general lack of interest amongst underwriters or their managers for training in the UK.

There was no user manual produced for phase I because there was a belief, attributed to Tim, that none was required because of the development approach. Unfortunately there were also some problems with the online help system in phase I and America particularly missed the manual. The role of a user manual was recognised by the team but it can not have been regarded as very important as it took over six months to produce and I was never shown a copy.

 C.3.1l Communication
 Communication  
The development team made significant efforts to keep the users informed about developments. Despite this there were some negative comments about communication and one of the reasons cited for the failure of the electronic transfer system was a lack of understanding about how it worked.

Top


C.3.2 Why was the quality of the data poor?

The responsibility for data quality seems to have resided with the users rather than the development team and in consequence the problems were not really highlighted by the Post Implementation Reviews. Moreover, some of the senior management of X seemed to have a relaxed attitude to data quality and this affected the underwriters' attitude.

There were difficulties in loading the data because of the way in which X had developed from a number of distinct groups working with different procedures and classification systems. More particularly some key information was only stored in underwriters' heads. Some of the difficulties in defining the data were inherent to the business of multinational insurance.

The data converted from the old Q&A system was already of poor quality and was out of date. In addition data problems were introduced at conversion by a lack of understanding of the insurance business by development staff. This coupled with the decision to check the data at renewal meant that poor data quality in the UK persisted for over a year. Initially it was not possible for the users to correct some of the fields but when the validation rules were relaxed there was a further impact on data quality.

 C.3.2a Responsibility for data quality
 Organisation  
In the Strategy Study in 1991 it was recognised that the new MIS system would depend on the quality of the data which was currently incomplete and that time would need to be spent sorting the data out. Despite this, issues of data quality do not appear to have been explicitly addressed in the early analysis.

Despite commitment to computerisation, senior management did not emphasize the importance of accurate data in the Q&A system perhaps because they did not have the same view of accuracy ('we don't deal in precise numbers, it's millions of pounds'). The loss of management commitment to the Multinational system when David left as phase I went live was regarded as one of the reasons the data quality was poor (although Jenny was temperamentally much better suited to getting the data right than David because of her Services background).

The problems of data quality were not really highlighted in the Post Implementation Review because it focussed on the development of the system. Even in early 1995 there was still confusion between the data quality problems and actually getting people to use the system. This perpective is reflected in the different ways the Post Implementation Review and Audit reports were handled. The Review was widely publicised but the Data Audit was kept under much tighter wraps.

In reporting the problems of data quality Colin attributed this to leadership problems but did not clearly identify who should have responsibility for it. The data audit was managed by the users so it would appear that they had responsibility for data quality at this stage. However, although the data audit found 'a lack of accuracy and attention to detail' and 'Mr Tim hit the desk over and over again', it took three months for X management to address the underwriters about the findings of the data audit by which time the impetus was lost. It also took three months between Neville being given responsibility for sorting the data out and it being announced.

Throughout the development the team seems to have regarded data quality as the users' responsibility although they scheduled time to assist the users. As I stopped observing, the X Director responsible for IT, gave the responsibility for data quality to the development team. Whether in other projects the team would assume this responsibility is questionable but from their experience with X, they would certainly treat data quality problems much more seriously from the start.

While setting deadlines seemed to motivate the developers to delivering on time, deadlines set for underwriters to get their data input correctly do not seem to have the same effect. Was this because there was more of a project mentality amongst developers? I am not sure even the threat of unemployment will ensure Neville succeeds; in phase I giving Tim the ability to look at any underwriter's business may have 'frightened the life' out of the underwriters, but it did not provide any impetus to them getting their cases in and up to date.

 C.3.2b Attitude of underwriters
 Organisation  
The lack of interest shown by senior management in controlling the business meant that they did not emphasize the importance of accurate data in the Q&A system. Many of the underwriters had a similar attitude so the data to be converted from the Q&A system was already poor quality. There was some evidence that this atttitude of the underwriters meant that 'any old thing was sometimes dumped in'.

 C.3.2c Organisation of data input
 Organisation  
One of the problems of loading the data into the system was that the sources of information were not well organised. Some data was only stored in the underwriter's head, so Services who were used to keeping data up to date and regarded it as their job could not input it. There was a continued problem getting information from the overseas offices that was unrelated to IT, more 'hearts and minds stuff'.

The original plan was that the underwriters would input the full details of new cases because this would help them understand the system, they knew all the details of the case and it would give them ownership of data on the system. Unfortunately many of the underwriters in the UK and most of the underwriters in the US did not respond. There was a move to underwriters only entering programme level information for new cases. However, the files were kept in London and the new Services staff in Essex had little experience of the complexity of Multinational cases so this had not proved successful by the time I stopped observing.

The formation of X and its widespread nature meant that before the Multinational system was implemented there were not agreed common procedures. In addition, the decision to continue to take IT services (and other business services) from local X-Group operations meant that the MIS must take feeds from a great variety of uncontrolled sources who were 'organisationally and culturally distinct'. Tim and David saw the introduction of the new system as an opportunity to establish common procedures in X particularly for accounting. The difficulties of establishing common procedures were illustrated by the underwriter files; although they talked about standard forms for underwriter files in London in March 1994, these did not become established until August 1994 and one of the problems faced by the data audit team was the proliferation and variation of underwriters' paper files.

It was also recognised that world-wide aggregation of X data would require a common classification system to be established as part of the new system, but when data was loaded different interpretations were put on fields so they were used inconsistently. Field interpretation also depended upon business policy and so could change if policy changed.

 C.3.2d Difficulties in the data
 Domain  
Getting the cases in was clearly not just an exercise in keyboard skills; even Ned who reckoned he was PC literate and thought he was up to date found a few errors when he was audited. The information was not all that precise, so for example the exact cover for a particular country might not be known without going back to the broker. There could also be a significant time lapse before the information was agreed. This complexity made it difficult to employ temporary staff to enter the data.

 C.3.2e Data conversion
 Method  
Conversion of data from Q&A was responsible for many of the initial data quality problems. The data in Q&A was not of very high quality and was already out of date by the time the full version of phase I was installed. There were also a number of misunderstandings about how records should be converted resulting in problems with some programmes that were not apparent until the programme was individually checked. The problems of this conversion were so severe that errors were still emerging over a year later.

There was a view from developers and users that the data should not have been automatically converted from the Q&A system; at the very least they should have trialled the conversion to have identified the problems earlier. In phase II the data was converted again and further errors were introduced; some of these were caused by the poor structure of the phase I database.

 C.3.2f Development staff expertise
 IT profession  
A lack of understanding of the insurance business by development staff contributed to the problems in data conversion.

 C.3.2g Installation
 Method  
The decison in the UK to convert the data from Q&A and then check it when it came up for renewal had a major impact on data quality. The consequence of this decision was that the rather fuzzy position on data quality was able to persist for well over a year. Moreover, Services had the burden of running two systems in parallel throughout that time.

It was definitely a bad decision to convert the data from Q&A with all its faults only to find that potentially it was not going to be checked for up to a year. Although the users still thought this approach was appropriate when viewed retrospectively, I feel that it would have been better to time the installation so that users would have been able to load the data immediately. This would have made it harder for data quality issues to persist. Holland took this approach (it was easier because they were smaller) and although their data still suffered some quality problems they seem to have used the system much more effectively.

 C.3.2h Validation
 Design  
The amount of validation on fields and the impact on data quality was an issue that continued to be debated. The amount of validation in phase I meant that data loading and correction was more time-consuming than it needed to be as some fields could not be amended by underwriters. In phase II some of the controls were relaxed allowing underwriters to save in the middle of a transaction but the data audit revealed that this led to more errors and a decision was taken to tighten up on validation.

Top


INDEX | ORGANISATION | TIME LINE | PEOPLE | STORY | RESEARCH METHOD | COMMENTARY | FINDINGS | HELP | TOP

© Clare Tagg 2000