Understanding Business Impacts of Web System Prototypes

David Lowe, david.lowe@uts.edu.au
Norazlin Yusop, norazlin@it.uts.edu.au
Didar Zowghi, didar@it.uts.edu.au
University of Technology, Sydney [HREF1]



Commercial Web developers typically make extensive use of prototypes early in the development life cycle. Rather than supporting requirements validation or elicitation, one of the key objectives of this utilisation of prototypes is to support clients in understanding how the system under development may impact upon their business processes or even business models. Despite this there has been little research into understanding the extent and nature of clients interpretation of early prototypes and how they inform them about potential business impacts. In this paper we report on a pilot study aimed at identifying the key issues in this prototype exploration process, and in particular what aspects of the relationship between prototypes and business models should be considered further. Key outcomes include a number of unexpected indicators, such as pointers towards potential impacts that are not adequately addressed by current prototype exploration, and differences in the capabilities of clients versus developers in understanding system impacts. We also describe key open issues that need further investigation if we are to adequately understand this aspect of the Web development process.


Web systems have a substantially increased degree of uncertainty - in terms of client understanding of their needs and developers understanding of the changing business domain. To a large extent this is a consequence of uncertainty over the impact of the system being developed on business processes and ultimately on business models. This can be described as solutions and problems being mutually constituted - a concept well understood in the area of social informatics (Sawyer, 2000) and requirements engineering (Carroll, 2002).

Given that the design of the system being developed can affect the nature of the problem being addressed, commercial Web development has often adopted a process that incorporates early utilisation of design prototypes (Lowe, 2001). Whilst this is not unusual in IT systems development, the intended purpose is somewhat different. Rather than being used as a requirements validation or elicitation tool, the prototypes are used to support joint exploration by clients and developers of the impacts of the system on business processes and models. In effect the exploration of prototypes is used to support the reformulation of the problem domain prior to final identification of requirements. This process is more crucial in Web development, where the extent of the interdependence of system design and business issues makes it crucial that this is understood as early in the Web system development cycle as possible.

This then raises the issue of just how the impact of different system designs on the business model and processes can be understood by project clients. If the problem is to be accurately reformulated jointly with the development of system designs, then clients and developers need to be able to reliably understand the ways in which business processes and models may be affected by a given design. It is this issue that underpins the work described in this paper.

Little research has focused on understanding the extent and nature of clients understanding of system impacts on their business, and specifically how different prototypes (and the way in which the prototypes are utilised) may influence this understanding. In this paper we describe the outcomes of a pilot study which aimed to identify the key issues that arise when clients explore early prototypes. Whilst these issues will then inform ongoing research, they are also useful in guiding developers and clients about aspects of software development that may need particular attention.

We begin in the following section by considering the background to this work and investigating the limited research into understanding the impacts of system designs on business models. We then describe the pilot study we have carried out to explore the factors that influence a clients' ability to understand these impacts. Key outcomes of the pilot study are the presented, including both key development issues and research lessons learnt. We finish with some pointers to ongoing research.


Traditionally, software systems were relatively discrete and static whereas businesses tend to be dynamic (Warboys, 1999) Despite this dynamism there are aspects of the business that remain invariant, in that they do not alter (Kilov, 2002). Whilst this dynamism often leads to required changes in the supporting software systems, the rate and extent of change usually implies that it can be accommodated within incremental or iterative development processes. There is, however, a class of software systems where the system itself triggers changes in the business operations. Web systems tend to be the most common exemplar of this class of applications - often as a result of their impact on the interactions that occur between the business and its external stakeholders (Chin, 2001). That is, what was considered the indicative state of the problem domain (Jackson, 1995) now might no longer be the case because the web system will change how the business works (but not the nature of the business).

In the cases where the introduction or modification of the system changes the business operations, it is important that the system design take into account these changes (lest the system address a problem which has subsequently changed!) This therefore indicates that developers and clients should be able to understand how the system will impact on, and hence have the potential to change, the business processes and models.

Whilst there has been some work done on understanding the way that technology is likely to impact an organisation (such as (Southcott, 1993)) this work has tended to not emphasise how these impacts might be addressed during the development process. Rather they have either typically focussed on an organisations response to these changes, or on ways of reorganising to accommodate the changes once they are identified. The prime example of this latter aspect is work on Business Process Re-engineering (Sharp, 2001 and Chin, 2001).

It is also useful to consider work on comparison of different prototyping approaches. The early use of prototypes in systems development has triggered discussion concerning the distinction between "throw-away" prototypes and those that eventually evolve into the final system. This was because in this domain it was possible for the prototype to evolve, unlike the older concept of prototypes of physical objects such as toys or cars. Later it was realised that this distinction probably had little value and the focus turned more to the use of the prototype - to explore feasibility, as proof of technical concept, as a simulation of behaviours, or as a limited function basis for evolution. These different goals then give rise to different solutions for the prototype implementation. Typical Web prototypes include screen dumps, skeleton sites, story boards, site maps, and information architecture and designs.

Despite this work, there has been an absence of research that relates prototypes to the way in which clients or developers develop an understanding of system impacts on business operations. Given this lack of work, we aim to investigate the extent and nature of clients' interpretation of the impacts of designs (as captured by early prototypes).

Thus our RESEARCH QUESTION is: What factors influence the ability of a client to develop a consistent understanding of the impacts of a proposed system on the existing business operations, when presented with a series of early web design artefacts?

Pilot Study

As the initial step in undertaking a rigorous analysis of key issues, we wished to identify those aspects that were likely to have a significant influence on the development of a clients' understanding of the system impact.  With this goal in mind, we carried out a small-scale experimental pilot study at the University of Technology, Sydney.  The intent of the pilot study was not to obtain statistically significant results, but rather to provide pointers to the key issues that deserve more attention from researchers and practitioners alike and can be explored in detail in a later full-scale experiment. Of course, the pilot also allows us to evaluate the experimental materials and the experimental design, but these issues are not the main focus of this paper.

Aims of the Pilot

The aims of the pilot are:

  1. To determine the relative significance of the different design prototypes, in terms of their influence on a clients' understanding of system impacts on business operations?
  2. To provide pointers to those factors that may have a significant influence on the ability of a client to develop an understanding of the impacts?
  3. To assess whether the structure and contents of the pilot can be reused in a more rigorous evaluation of those factors identified in the above point.

Note that factors identified in point 2 above are likely to be useful pointers for ongoing research, but also in informing practitioners about those aspects that warrant careful attention during commercial Web development projects.

Overview of the Pilot

We constructed a Web system development scenario, where a client has an existing (non-Web) system and it has been proposed to develop an on-line presence to support the business. A set of simple documentation was used to define the existing business, outlining the existing business model and processes, as well as existing systems. Similarly, a set of proposed design prototypes was constructed. This description of the existing systems, as well as the design prototypes, were presented back to the participants in a structured way and were coupled with several sets of questions aimed at evaluating their understanding of the existing business model, Web technologies, and the impacts of the prototypes. The pilot study scenario is outlined in Appendix 1.


A relatively small sample was use for the pilot study. There were 25 participants - all postgraduate students. There were 19 web engineering Masters students and 6 e-business Masters students. Though this is a heterogeneous sample of uneven balance, the aim of the pilot was to identify potential factors warranting further investigation rather than aiming to ensure statistically significant results.

The background of the participants only becomes an issue if there is a marked difference in the responses to the questions posed. The participants also have a range of professional experience. The Business participants on average have more than 7 years of full time employment and had past jobs titles such as computer network manager, data communications manager, store manager, business analyst and sales manager The engineering participants on average have between 1 to 3 years of full time employment and had past job titles such as a technical manager, software engineer, network engineer, customer support, analyst and database administrator. The majority of participants do not have industry experience in a project that involves the development of a business plan. The majority of Business participants have not conducted a project that involves the development of a web-based system, unlike the engineering participants.

Course of the Pilot

The major time and tasks of the pilot are listed in table 1. As this table shows, the participants were asked prior to the pilot about their subject area knowledge and their relevant work experiences. They were introduced to the pilot and given an overview of the intended research. The experimenters were careful not to reveal any specific details about the task at hand. The participants were then presented with information pertaining to the existing business model and business processes. They had 15 minutes to read the information and complete a questionnaire on that model to evaluate their understanding of the existing business. Participants were given an opportunity to ask questions from experimenters present to clarify any points. The participants were given a break of 15 minutes between parts B and C. Participants were not permitted to take any pilot materials out of the pilot room. They were then presented with a series of design prototypes and then asked to answer questions about the prototypes and the expected impact on the business model. They had 30 minutes to complete part C.

Table 1:  Tasks and time allowed




A. Skills/Knowledge Survey  


B. Read business model artifacts and answer questions on existing system  


C. Read design artifacts and answer questions on impact of design artifacts on business model  


Threats to study validity

The limited number of participants in this pilot experiment and the relatively complex nature of the research question makes it difficult to carry out a complete analysis which may be considered as a threat to the study validity. But since this was an exploratory study aimed at identifying key issues for further investigation, we were primarily interested in considering all responses carefully for the purpose of generating hypotheses for future experimentation.

Results and Analysis

An analysis of the results of the pilot study leads to a number of interesting observations. Whilst some of these were rather intuitive, a number were also somewhat surprising. These observations are important in terms of identifying the key issues to pursue in further research. They are however also important in terms of the potential messages to Web systems developers and clients about how Web projects should be managed, particularly early in a project when the system is first being defined. The observations can be broken into three key areas - paralleling the three aims of the pilot outlined above: understanding the differing roles of the assorted prototypes that are available; factors that appear to affect (or not affect in some cases) a client's understanding of the business impacts of a proposed system design; and lessons learnt about the pilot study process itself (and what this may also tell us about commercial practice). We will look at each these areas in turn.

Relative significance of different design prototypes

A factor analysis test and reliability test were conducted to measure the correlation between the level of participants understanding of the existing system and the level of understanding of the proposed online system. The results reported a positive correlation of factor score 0.863. This indicates that the participants’ understanding of the existing system may be positively related to their understanding of the design prototypes for the online system.

In particular there is a high positive correlation (0.804) and significance (p-value of 0.00, which is less than 0.05 significance level) between storyboards, example Web screen shots and business process models. Whilst requiring further investigation, this may indicate that these prototypes serve similar purposes in supporting client understanding - though this would need further investigation. These are also the design prototypes for which participant responses indicated a high level of correctness in terms of understanding. The design prototypes that have a weak correlation (0.043) and no significance (p-value of 0.427, which is more than the 0.05 significance level) are between WebML Model, Example Site Navigation Map and Screenshot of a database, Venue Information Form and Data Flow Diagram. The participants generally have difficulty answering questions and understanding processes and design elements pertaining to these design prototypes.

This maybe due to the lack of full familiarity of the participants with the notations used in these particular design artefacts such as WebML model or Data Flow Diagrams. On the other hand, some of these design artefacts may not provide the participants with any additional information that they could not have obtained from other designs inspected before.

Factors that influence client understanding

Prototype coverage of potential impacts: In order to investigate the extent to which clients may have been able to identify all potential impacts, we considered the standard deviation of responses to the questions evaluating the participants understanding from the impacts of the web system introduction on the business. As shown in Table 2 below, we found that there was a higher standard deviation (i.e. less agreement between different participants) for those questions that asked about elements not explicitly addressed in the prototypes. For example, the new system obviates the need for printers, and so the printer naturally does not appear in any of the prototypes. The result appears to be that there was a greater uncertainty about the impact on the printer. This may be also due to confusion on the part of the participants as to the intended meaning of the word "printer", i.e. printer as a peripheral device or printer as an organisation that produces printed material for its customers.

What this seems to indicate is that where the prototypes do not explicitly address a particular aspect of the business (either because those aspects are not expected to change or because they became redundant or irrelevant in the new system), clients are likely to make assumptions about those aspects. In terms of practical development, this may indicate that the prototypes themselves are invariably insufficient and they need to be supplemented by an explicit critique (by the clients) of the existing system and which elements of the business models and processes map to aspects of the new system etc.

Table 2:  Average responses to questions about the extent to which aspects of the existing
business are affected by the introduction of the new system (1=No Change, 5=Totally Changed)

Aspect impacted upon

Average Score

Standard Deviation

Role of the printer



Interaction with the printers in getting the flyers printed



Interaction with printer



The nature of value exchange between DDR and printer



Level of value perceived as being gained by the printer



All aspects



As shown in Table 3 the standard deviation of responses related to dynamic aspects (such as timing etc.) was also higher than other aspects. i.e. the prototypes which we were using provided a better understanding (or rather, a more consistent understanding) of static and informational impacts on the business, but not functional aspects of business operations. In particular, even though some of the prototypes included functional elements (such as the storyboards) these results indicate that this may not necessarily flow through into a consistent understanding of how functional elements of business processes may be affected. This is a key area requiring further investigation.

Table 3:  Average responses to questions about the extent to which dynamic aspects of the existing
business are affected by the introduction of the new system (1=No Change, 5=Totally Changed)

Aspect impacted upon

Average Score

Standard Deviation

Timing for the delivery of event information by venues



Timing for the delivery of advertisements



Timing for the delivery of performer’s bio



Timing for updating new information on DDR medium



Importance of prior knowledge of the business: Somewhat surprising to us, there was only a weak correlation between answers to part B of survey (understanding of the existing business) and answers to part C of the survey (i.e. understanding of impacts of the new system). This has important implications in terms of who is better positioned to understand the impacts of the technology - the developers (who are likely to have a stronger technological understanding) or the clients (who are likely to have a stronger business understanding).

Interestingly, however, there was also only a weak correlation between answers to part A of survey (understanding of modelling approaches etc.) and part C. This result (and the point above) may indicate that the understanding of implications is more reliant on understanding of the actual design prototypes (something which we did not explicitly evaluate) than the technology, modeling approaches, and existing system. This is a key point requiring further investigation. If this is indeed the case (i.e. that the key factor affecting understanding of business impacts is the actual depth of understanding of the prototypes) then this would indicate that the way in which the prototypes are presented and explored becomes crucial. This also relates to the following point.

Comparative ability of client and developers to understand the business impacts: Whilst the sample size is small, Table 4 shows that there is little indication of a significant difference in ability to both understand the business model (Part B) and to understand the business impacts (Part C) between the web developer students and the business students. The two groups scored comparably on both parts of the questionnaire. This may indicate that, provided developers are given appropriate opportunities to understand the client's business models and processes, they will be capable of understanding the business impacts at least as well as the clients themselves.

Table 4. Scores obtained by business students versus Engineering students
(as percentage of questions answered correctly)






66 %  

71 %  

70 %


53 %  

48 %  

49 %

Summary and Future Work

In this paper, we have discussed our initial work on business impacts of Web system prototypes. We have described the outcomes of a pilot study that explored the factors that may affect a clients and developers understanding of these impacts.

There are a number of issues raised from this pilot specifically in response to the core research question. Key challenges for system developers are as follows:

The third aim of the pilot study was to evaluate this study to prepare for a larger experiment. The format for a larger experiment (with increased sample size) will alter as we plan to compare diverse client types. Tight time constraints might have been a factor in the pilot so we plan to reduce the amount of information provided and to reduce the number of tasks subjects will have to complete. Furthermore, we intend to provide more prescriptive navigational information to the participants such that they will be guided by the types of questions asked as to what design artefacts they should examine.

There are two specific areas for further work that are currently being investigated. Firstly, we wish to explore the effect on client understanding (and hence system development outcomes) of a development process that explicitly engages the client in a review of the design prototypes as they relate to the business model.

The second area is to investigate more directly the clients' understanding of the design prototypes. We wish to explore how this correlates to understanding of the implications for the business. A strong correlation would indicate the importance of actually educating the client in the prototypes, and not just providing them directly to the client.

In that this study is one of a number of longitudinal investigations of the impact of design prototypes on stakeholders understanding of the influences that the proposed web system will have on the business, as part of a long-term investigation of this phenomenon, it helps to set the scene for what is planned to follow. The next stages of the research involves developing a clearer understanding of the design elements that can influence business processes, and the elements of business processes that can be influenced.


The authors wish to acknowledge the insights and suggestions of Ross Jeffery, Aybuke Aurum and Karl Cox, as well as the support provided by the Australian Research Council under grant no. DP0211461. We also thank UTS business and engineering students who participated in the experiment.

Appendix 1: Pilot Study Scenario

This section provides a brief outline of the pilot study treatment. This scenario describes an existing company (called DDR Promotions) that undertakes event promotion (at no cost to the event managers) through distribution of hard-copy posters and pamphlets in public spaces. Income is generated through the inclusion into this material of paid advertisements. The company collects information on events (from venues) and information on performers (from public sources or from the performers themselves). This information is then reformatted into posters and flyers, and advertising material is then included (and invoiced for - providing the source of income). The posters and flyers are widely distributed through manual delivery. The company has decided that the Web provides an ideal vehicle for enhancing the distribution of information and enriching their operations. As such they have contracted a Web development company (Web Designs P/L) to undertake the development. The developers have undertaken an initial exploration which incorporates the identification of initial acceptance criteria and then some rudimentary prototype designs. These designs are then discussed with DDR Promotions to determine their responses and feedback.


Carroll, J., (2002) "Scenarios and Design Cognition". in RE'02, (Essen, Germany), pp. 3-5.

Chin, J. (2001), "e-commerce and innovation business process reengineering". PICMET - Portland State Univ, 2001, p. 163-4 vol.1, Portland, OR, USA

Jackson, M. (1995) Software Requirements and Specifications. Addison-Wesley, Harlow

Kilov, H. (2002) Business Models: A Guide for Business and IT. Prentice Hall, NJ

Lowe, D., & Eklund, J. (2001, 22-23 November 2001). Commercial issues in specification of Web systems. Paper presented at the AWRE'2001: 6th Australian Workshop on Requirements Engineering, Sydney, Australia.

Sawyer, S., & Rosenbaum, H. (2000). Social Informatics in the Information Sciences: Current (2000). Informing Science, 3(2), 89-96.

Sharp, A., McDermott, P. (2001) Workflow Modeling: Tools for Process Improvement and Application Development, Artech House, ISBN: 1580530214

Southcott, J. (1993) "The business impact of technology change". Computing Canada, v19, n11, p26(1)

Tan, B., Lin, C., Chang, S., Chang, I. and Shih, K. (2002), "Impact of organizational characteristics and information technology maturity on Internet business application: A Taiwan survey" Journal of Asia - Pacific Business; Binghamton

Warboys, B., Kawalek, P., Robertson, I., Greenwood, M. (1999) Business Information Systems: A Process Approach. McGraw-Hill, Maidenhead

Hypertext References



David Lowe, Norazlin Yusop, Didar Zowghi, © 2003. The authors assign to Southern Cross University and other educational and non-profit institutions a non-exclusive licence to use this document for personal use and in courses of instruction provided that the article is used in full and this copyright statement is reproduced. The authors also grant a non-exclusive licence to Southern Cross University to publish this document in full on the World Wide Web and on CD-ROM and in printed form with the conference papers and for the document to be published on mirrors on the World Wide Web.