Kent Fitch, AustLit Project [HREF1], Academy Library [HREF9], UNSW@ADFA [HREF10], Australian Defence Force Academy, Canberra, ACT, 2600. k.fitch@adfa.edu.au
As application user interfaces migrate to the web, developers need to provide the same functionality and response times in HTML based UIs as users have come to expect from desktop based standalone or client-server applications.
Simple "fill-in-the-form-and-press-SUBMIT" interfaces are not enough. However, the process of developing more sophisticated and interactive web based applications often degenerates into a tangle of client-side Javascript and chaotic form parameters.
This paper outlines some basic techniques which may be used to rapidly develop flexible web based user interfaces by:
These techniques have been used successfully in two quite different distributed web applications - the document authoring system for the CSIRO web site ([HREF3], [HREF4], [HREF26]) and complex data maintenance functionality of Australian Literature Gateway (AustLit) ([HREF1], [HREF2]).
The usability of computer applications interfaces has an enormous impact on the productivity of those forced to use them. Although the delivered functionality and implementation budget of a system are the "headlines" read by IT management, the ongoing operational cost of the system as measured in the productivity of those that use it often dwarf the development costs.
A few seconds response time here, an interface induced error there, plus suboptimal use of the system and extra training costs multiplied by dozens of users over several years of system life time can create unexpected and huge costs to the system's owners.
With the advent of desktop-based standalone and client-server user interfaces in the 1990s the "art of the GUI" rightly received great attention from "Human Computer Interaction" (HCI) specialists. The goals of better UI design were to increase productivity by reducing errors and improving ease of use, and a large cast of UI "widgets" were developed with mixed results ([HREF23], [HREF24]).
Around the same time, the use of the WWW as an information dissemination tool was taking off. With the development of the Common Gateway Interface (CGI) [HREF11] framework for delivering dynamically generated content and the HTML FORM tag set for allowing users to enter and select data, the web started to become widely used as a platform for data maintenance; an alternative to the desktop based "native" UIs.
The strength and weakness of early HTML FORM based applications was their simplicity; with few of the complex UI widgets to confound users, developers were forced to use the few available UI elements.
In the beginning, there were static HTML forms. These consisted of one or more input fields of possibily different types (single and multi-line text entry, checkbox, radio button, select list) and a "SUBMIT" button. All processing of the form's data was performed on the server when the user pressed "SUBMIT", the server responding with a new web page, possibly with error messages or another form.
The main problems with this approach are response time and page flicker. Because all processing must be performed on the server regardless of the network bandwidth between the user and the server, the system will seem clunky as many simple user actions (such as wanting to add another line item to a purchase order beyond the default of, say 5) require data to be sent to the server and a new page to be parsed and laid-out by the browser. Context such as vertical scroll and cursor position will typically be lost. Some common user interface components such as popup select lists can't readily be implemented in any reasonable way.
Whilst fine for simple systems such as advanced search specification and limited data entry, simple HTML FORMs struggled to provide the sophisticated data entry and data validation users had come to expect. Moreover, the lack of client-side logic required round-trips to the server and subsequent screen redrawing whenever any logic needed to be executed.
This was a familiar scenario for mainframe developers; in the 1980s the most common applications system interface infrastructure was a IBM 3270 datastream to SNA-networked dumb terminals [HREF12]. The advent of client-server systems in the late 1980s and early 1990s moved some presentation and editing logic from the server to the (typically PC running MSDOS or Windows) client.
Netscape introduced a client-side scripting language "Javascript" (originally named "Livescript") [HREF13] and a simple API to allow Javascript to reference some of the elements of the FORM to perform tasks such as simple validation, calculation of totals, etc. Whilst the early versions were notoriously buggy and incompatible between vendor and cross-vendor (Netscape and Microsoft) releases, Javascript rapidly evolved and the FORM-addressing API developed into the "Document Object Model" (DOM) family of "standards" [HREF14]. Incompatabilities between browsers even from the same vendor remain, frustrating those who seek to develop client-side code which will run on more than one version of one particular browser.
Nevertheless, the attractiveness of web-based applications is so great that many new systems are implemented within the browser framework. The "browser wars" are for the moment, over; Microsoft's Internet Explorer version 5 and above have almost 90% market share as at April 2002 [HREF15]. This paper describes some techniques which can be used with IE 5 and above to successfully implement advanced client-based user interfaces for "mainstream" data maintenance.
The approach described here uses the HTML/DOM/Javascript technology combination which is traditionally referred to as "DHTML", but there are many other approaches in deployment and in development, including:
These other approaches have their strengths and weaknesses, but the focus of this paper is on generating forms based on instructions bound to a schema definition, and doing so within the standard most common browser environment.
There are many issues involved with developing advanced web-based applications which this paper does not seek to discuss, including:
Instead, I'm assuming you only want to implement a system which will work with IE5 and above and that you (and your users!) already know what your "dream" user interface is and security issues have been defined and handled.
I'm also assuming that although you may recognize the "market reality" of 90% market-share for IE, you may also not want to forever lock yourself into IE, so you'll be hoping to deploy a solution which vaguely adheres to some Javascript and DOM standards and avoids as far as possible obvious "Microsoft only" approaches such as scripting-MS-Word, or ActiveX controls. A version of your application which used the Netscape 6 implementation of the DOM for example should be at least "plausible".
Also, maybe like me you have been burned by the disappointments of large scale UI development using Java applets within browsers and it would take a lot to convince you to go down that route again. Concise outlines of the promises and failures of the applet approach has been provided by Anil Hemrajani's Applets: Still essential to Java [HREF21] and Tal Liron's Launching into Java [HREF22]
The following techniques can assist the implementation of advanced web-based user interfaces:
The HTML form delivers name-value pairs to the server. Sophisticated user interfaces allow the user to construct "rich" and often nested data structures with varying numbers of elements. For example, consider purchase order entry where it may be most convenient for a user to enter a purchase order with many "order lines", each with their own internal structure including, possibly delivery addresses.
The approach discussed here structures the data in an XML document. Rather than being "flattened" into name-value pairs with various numbering schemes to disambiguate multiple values of the same "name", (such as POLine4Qty2) the data is transported to and from the client as an XML document with an internal structure which is most appropriate to the application.
IE supports the concept of an "XML Data Island" ([HREF5], [HREF6]) which allows an XML document to be either sent within the normal contents of an HTML document:
<XML ID="someId">
<agent tid='A123'>
<attributes>
<name tid='N123' type='N1'>Lawson, Henry</name>
<altName tid='N124' type='N2'>Swallow, Joe</altName>
<altName tid='N125' type='N2'>An Australian Exile</altName>
<altName tid='N126' type='N2'>Tally</altName>
<altName tid='N127' type='N2'>The Exile</altName>
<altName tid='N128' type='N2' uncertain='y'>Smoko</altName>
<gender tid='G1'>male</gender>
<biography id='T123'>Henry Lawson was born at Grenfell, New South Wales, in 1867.
He was educated briefly at several schools, but was sometimes
kept home by his father to help with his carpentry....
</biography>
</attributes>
<events>
<birth tid='Z123'>
<attributes>
<birthOf tid='A123'/>
<place id='P123' type='P'>Grenfall, New South Wales</place>
<date id='D45' uncertain='n' startYear='1867' month='06' day='17'>
17 Jun 1867</date>
</attributes>
</birth>
<death id='Z124'>
<attributes>
<deathOf tid='A123'/>
<place id='P234' type='P'>Abbotsford, New South Wales</place>
<date id='D45' uncertain='n' startYear='1922' month='09' day='02'>
2 Sep 1922</date>
</attributes>
</death>
</events>
</agent>
</XML>
or referenced by a hyperlink:
<XML ID="someId" SRC="/getAgent?id=A123"></XML>
Client-side code can access the XML data island using Microsoft's XML Object Model API, which includes XPath capabilities. For example, on the above sample document:
The XML Object Model API supports updating. Hence, ignoring for the moment the process whereby the client-side script builds the user interface and solicits the updates, it is a simple process for our client-side script to either update the original data island or create a new XML Document object representing the results of the user updates. Whether to create a new complete XML document representing the new data or to create a "delta" (or "diff") document describing the changes the user made depends on the type of information you are maintaining and what structures are held on the database.
The AustLit application finds it convenient to deal with a document which shows only updated nodes and their parents and creates a "delta" document which may look like this:
<agent id='A123' updated='y'>
<attributes updated='y'>
<name id='N123' updated='y'>
<oldValue>Lawson, Henry</oldValue>
<newValue>Lawson, Henri</newValue>
</name>
<altName id='N124' deleted='y'/>
<altName type='N2' added='y' uncertain='y'>
<newValue>The Lost Exile</newValue>
</altName>
<altName id='N126' updated='y' uncertain='y'/>
</attributes>
<events updated='y'>
<birth id='Z123' updated='y'>
<attributes updated='y'>
<place oldId='P123' type='P' updated='y' newId='P124'>
<oldValue>Grenfall, New South Wales</oldValue>
<newValue>Upper Grenfall, New South Wales</newValue>
</place>
</attributes>
</birth>
</events>
</agent>
The following changes have been recorded:
The new XML document can now be "POST'ed" to the server using code such as the following:
var response = simplePostToServerXML("UpdateTopic", newTopicDocument) ;
...
function simplePostToServerXML(urlSuffix, body) {
var targetURL = SERVER_PREFIX_URL + urlSuffix ;
var xmlHttp = new ActiveXObject("Microsoft.XMLHTTP");
try {
xmlHttp.open("POST", targetURL, false);
xmlHttp.send(body);
}
catch (ex) {
alert("Post to server failed with error:" + ex) ;
return null ;
}
// error checking and response checking code omitted for simplicity
var response = ... ;
return response ;
}
The AustLit server "UpdateTopic" servlet expects to receive a POST body containing the "delta" document which it reads (from the servlet request.getInputStream() object), validates, applies and logs.
Rather than hard-coding forms, the approach discussed here dynamically generates the HTML form driven by:
A second XML data island contains the schema of the first "data" document. This schema describes the document well enough to allow an HTML FORM to be generated using the HTML DOM API. The important part here is not the schema language used (any schema language could be chosen as a base) but the presentation elements added to the schema definition which drive the form generation and validation process. A skeletal example which would work on the above data document should give you the basic idea. This example is not intended as a recommended schema format, but just to illustrate some possibilities:
<agent prompt="Agent" promptIdPath="attributes/name[@type='N1'"> <attributes> <attribute tag="attributes/name" prompt="Common Name" helpId="HumanCommonName" type="string" focus="y" /> <attribute tag="attributes/gender" prompt="Gender" helpId="HumanGender" type="enumeration" min="0" autoShowOnNew="y" enumerationId="gender" /> <attribute tag="attributes/altName" min="0" max="*" prompt="AKA" helpId="HumanAKA" type="string" /> <attribute tag="attributes/birthName" min="0" max="1" prompt="Birth Name" type="string" helpId="HumanBirthName" suppressAdd="n" /> <attribute tag="attributes/nationality" min="0" max="*" prompt="Nationality" type="shortSelect" helpId="HumanNationality" selectId="nationality" defaultText="unknown" /> <attribute tag="attributes/isExpat" helpId="HumanIsExpatriate" min="0" max="1" prompt="Is Expat" type="assertion" /> <attribute tag="attributes/biography" min="0" max="1" prompt="Biography" helpId="HumanBiography" type="bigText" /> <attribute tag="attributes/note" min="0" max="*" prompt="Note" type="text" helpId="HumanNote"/> <attribute tag="attributes/influencedByAgent" min="0" max="*" prompt="Influence-agent" helpId="HumanInfluenceAgent" selectId="agent" type="agent" /> <attribute tag="attributes/noteMaintenance" min="0" max="*" prompt="Maintenance Note" type="text" helpId="HumanMaintenanceNote" showPrivate="n" lairy="y"/> <attribute tag="attributes/lawConversion" suppressAdd='y' readOnly='y' min="0" max="1" prompt="LAW Conv notes" type="text" fieldHeight="3" /> <attribute tag="attributes/awardDetails" min="0" max="*" prompt="Award" helpId="humanAward" type="award" instrRef="awardDetails" subTopic="awardDetails" hideable="n" hidden="n" /> </attributes> <awardDetails id="awardDetails" addSelectPrompt="Award"> <attributes hideAddList="n" > <attribute tag="attributes/isAwardOf" prompt="Award Name" min="1" max="1"helpId="HumanAward" selectId="awardName" type="awardName" fieldWidth="25" suppressUncertain="y" autoShowOnNew="y" selectRestriction="v" /> <attribute tag="attributes/hasDate" prompt="Year" helpId="awardDate" type="date" yearOnly="y" autoShowOnNew="y" min="0" max="1"/> <attribute tag="attributes/section" min="0" max="1" prompt="Section" type="string" helpId="awardSection" fieldWidth="22" autoShowOnNew="n" allowEmpty="y" suppressUncertain="y" /> <attribute tag="attributes/note" min="0" max="1" prompt="Note" type="text" showPrivate="n" helpId="awardNote" fieldWidth="25" fieldHeight="2" autoShowOnNew="n" allowEmpty="y" suppressUncertain="y"/> </attributes> </awardDetails> <events> <event tag="events/birth" min="0" prompt="Birth" helpId="HumanBirth" type="group"> <attributes> <attribute tag="attributes/hasDate" prompt="Date" min="0" helpId="HumanBirthDate" type="date" autoShowOnNew="y"/> <attribute tag="attributes/hasPlace" prompt="Place" min="0" max="1" helpId="HumanBirthPlace" selectId="hierTopic" type="hierTopic" fieldWidth="25" autoShowOnNew="y" selectRestriction="P" /> <attribute tag="attributes/birthOf" type="implied" impliedSource="agent" /> </attributes> </event> <event tag="events/death" min="0" prompt="Death" helpId="HumanDeath" type="group"> <attributes> <attribute tag="attributes/hasDate" prompt="Date" min="0" helpId="HumanDeathDate" type="date" autoShowOnNew="y"/> type="hierarchicalSelect" selectId="place" autoShowOnNew="y"/> <attribute tag="attributes/hasPlace" prompt="Place" min="0" max="1" helpId="HumanDeathPlace" selectId="hierTopic" type="hierTopic" fieldWidth="25" autoShowOnNew="y" selectRestriction="P" /> <attribute tag="attributes/deathOf" type="implied" impliedSource="agent" /> </attributes> </event> </events> </agent>Points to note:
The client-side Javascript uses the schema and presentation information to construct a form. Obviously, the schema can only specify data types which the script knows about (such as string, text, date, shortSelect, agent in this example). Once the rules to construct a set of data types has been coded in the client, any schemas can reference them.
The client-side Javacript must recognize the "min" and "max" attributes to build user interface elements allowing the user to add or delete occurrances of elements.
Although the current implementation uses the Microsoft's extended DOM API to generate HTML, the same schema and data could be interpreted to produce, say XUL, or even a Java applet based UI.
Here's a partial screen-snapshot from the live AustLit user maintenance interface which demonstrates the HTML FORM generated for editing an agent record:
Users often want to add or delete occurrences of fields or field groups, such as purchase order lines. Rather than requiring a round-trip to the server and complex form regeneration and context reestablishment with consequent response time and page-update issues the approach discussed here dynamically updates the form with client-side logic.
As discussed above, the presentation schema includes information about the allowed number of occurrences ("min" and "max" attributes). It also describes whether occurrences can be added or deleted in this view. The client-side Javascript uses this information to generate the "+" and "-" buttons shown in the above screen snapshot which allow occurrences to be added and deleted. As well as these buttons, all addable elements are shown in an "Add" drop down list.
When an element containing sub-elements is added, a new "group" of elements are drawn. For example, the following image shows a new "Award" group added to the agent form:
A good rule of thumb to use when developing data maintenance suites is to encourage the user to select from a valid option rather than to enter or re-enter the data using the keyboard. For example, it is likely that most purchase order delivery addresses are maintained by the system, so rather than asking the user to enter the address it is usually a better design to ask them to select an address, perhaps constrained or defaulted by other partially entered form data such as the user's department. The approach discussed here invokes a popup selection dialog and copies the selected data to the base form. By judicious caching of data in selection lists, a round trip to the server to show the selection dialog is usually avoided.
A simple example in the AustLit system is an agent's nationality described by this definition in the schema:
<attribute tag="attributes/nationality" min="0" max="*" prompt="Nationality" type="shortSelect" helpId="HumanNationality" selectId="nationality" defaultText="unknown" />
The client-side Javascript renders elements of this type as read-only text followed by a "Choose..." button. The button invokes the "shortSelect" functionality which uses the value of the "selectId" attribute to populate the selection list of a popup modal window. The server is asked to build this modal window via a standard HTTP request. The value of the "selectId" attribute is one of the parameters passed to the modal window script. This script requests an XML document from the server returning the unique identifiers and names for all occurrences of the selectId value ("nationality" in this case), and builds a selection list. Subsequent requests for the modal window will be satisfied from the browser's cache, as will requests for the XML document containing the requested identifiers and names, depending on the expiry date set on this information by the server (volatile lists expire immediately, less volatile lists are cached for minutes or hours).
A more complex example is the "place" selector specified as follows:
<attribute tag="attributes/hasPlace" prompt="Place" min="0" max="1" helpId="HumanBirthPlace" selectId="hierTopic" type="hierTopic" fieldWidth="25" autoShowOnNew="y" selectRestriction="P" />
Here, the selectId of "hierTopic" invokes a quite different selection function which allows a selection from AustLit's hierarchical thesaurus. The "selectRestriction" attribute is passed to the popup script to constrain the selection process (in this case to "P" which is understood by this system as "Place Names"):
As a final example, the "agent" selector is used in the element which describes agents which influenced the agent being edited:
<attribute tag="attributes/influencedByAgent" min="0" max="*" prompt="Influence-agent" helpId="HumanInfluenceAgent" selectId="agent" type="agent" />
Once again, the "selectId" attribute drives selection of a modal popup window which this time is tailored to the search for and selection of an agent:
An input form will populate selection lists dynamically, possibly constrained by the form's current context. Often the values in these selection lists can only reasonably be constructed by "heavy duty" business logic on the server - the client just does not have the raw data. The approach discussed here describes how the client code can initiate synchronous but "background" connections back to the server to get this data. They are "background" in that the user doesn't see the page flicker or the "world turn". All they notice is an optional status line message and the cursor turning to an hour-glass whilst the operation completes.
For example, in the above agent searching example, upon building the form a background HTTP connection is sent to the server to retrieve the most recent agents selected by this particular user (the rationale behind this is discussed in a separate point below). These are then shown to the user in the search-results body.
If the user does not want to select any of these, they enter an agent search expression and click "Search". Once again a background HTTP connection is opened to send the search expression and receive the results (as XML) which are then displayed in the select list body.
The technique for opening a background HTTP connection is very simple, shown here in 'bare bones' form:
var resultXMLDocument =
getFromServerXML("GetFromCache&topicType=" + selectId) ;
...
function getFromServerXML(urlSuffix) {
var targetURL = SERVER_PREFIX_URL + urlSuffix ;
var xmlhttp = new ActiveXObject("Microsoft.XMLHTTP");
xmlhttp.open("GET", targetURL, false);
xmlhttp.send("");
response = xmlhttp.responseXML ;
if (response.parseError.errorCode != 0) ... // fatal error
return response ;
}
The result in variable resultXMLDocument can then be accessed using the XML Object API and used to populate an HTML SELECT list using the DOM API.
Users tend to select the same data from selection lists. The approach discussed here suggests making the most recently and frequently selected values very easy for the user to select. Remembering and retrieving these values requires server-side communication and storage.
Once again, a background HTTP session is created each time an object is chosen from a selection list and the server remembers the selection against the type of selection list and the user. Hence, previous selections "follow" a user between sessions (across PCs, across time).
Web based application systems can often benefit from the entry of formatted text by the user. Whether for emphasis or layout, basic text formatting such as dot and numbered lists, bold, italics and alignment can greatly enhance the traditional "notes" field. The approach discussed here describes use of the rich text mini-editor built into IE.
In the above schema text fields of more than one line are by default editable using the IE rich text editor. The client-side Javascript form generator inserts a 'pen' icon next to the text box, and when clicked, places the text from the box into an element with the IE attribute "contenteditable" [HREF7] on a modal popup window.
A variety of editing controls are available, as shown in our example:
An issue we've had with the editable text window is that the results are not returned as well marked up XML, which we require. So some regular expression based search and replace on the returned string is required before we place it back in our normal text window.
A major goal for the web application developer is to deliver a user interface as usable as the "state of the art" native desktop application. The API exposed by modern browsers such as IE5+ and Netscape 6 now allow sufficent control of the user interface to enable high quality user interfaces.
However, the code to implement many of the desired qualities is not trivial and the challenge is to enable it to be reused across forms and applications. One approach to doing this is to develop a suite of user interface manipulation routines which can be driven by a relatively simple definition of the data to be manipulated - a "presentation schema".
Results with this approach on 2 quite different projects have shown the method to be both viable in producing high quality user interfaces and adaptable. New schemas are easily generated to allow maintenance of new objects. User reactions have been extremely positive. They experience the response times and UI functionality they have come to expect from desktop applications, and benefit from the ease-of-use and automatic version updates of a web-based interface. Apart from the standard browser which comes preinstalled with their version of Windows, there is no local software to install or maintain. For a geographically dispersed project such as AustLit which has to operate with no technical user support or technical training budget, simplicity and robustness of operation are critical issues.
Although it seems likely that the XForms development will eventually deliver a superior approach to form and data description, the schema-based generation of DHTML Forms coupled with the other techniques discussed in this paper can deliver very effective user interfaces and are worth considering when you are next required to implement a sophisticated web-based user interface.