Prof Michael Brittain, University of South Australia. Email: Michael.Brittain@Unisa.edu.au
This paper gives a practical account of the refereeing process used in the trial of InfoTrain (Australia) in July to November 1994. It then describes the proposed forms-based / electronic mail refereeing process for use in 1995.
InfoTrain (Australia) [HREF 3] can be found at http://infotrain.magill.unisa.edu.au
The Teaching Electronic Journal project was initially conceived by Professor Jack Meadows in conjunction with Mr. Fytton Rowland and Dr Michael Brittain at Loughborough University of Technology. It is now an international initiative with participation from Strathclyde University, The City University, London, Chalmers University, Sweden, University of Adelaide, and the University of South Australia.
InfoTrain is a product of the Teaching Electronic Journal project. InfoTrain is intended to be a fully operational electronic journal. Students and staff may play the role of author, editor, referee, system administrator, and reader. They are able to explore these roles in a controlled environment that encourages experimentation without many of the constraints faced by existing professional electronic journals. InfoTrain is intended to be serious in nature and is concerned with scholarly writings in the area of Information Management.
The intention is to allow postgraduate and undergraduate students to submit the majority of source material for the journal. These students also provide content-oriented editing and refereeing functions, layout services, system administration, literature reviews, and locate material from other sources for inclusion.
InfoTrain provides staff and students with opportunities to conduct research into the operation of an Electronic Journal. Areas such as copyright, archiving, dissemination, peer review, coordination, and those areas currently unknown, are open for investigation.
InfoTrain in Australia began in February 1994 when Professor Michael Brittain formed a committee to explore the possibility of a local implementa tion of the Teaching Electronic Journal project. This committee, made up of representatives from the University of South Australia and the University of Adelaide, initiated a trial of the "InfoTrain" journal in semester 2, 1994.
In 1995 InfoTrain is funded by an internal University of South Australia grant, which provides release time for Philip Marriott to coordinate the journal's activities. The original founding committee has evolved into the InfoTrain Steering Committee which now provides policy guidance.
Participating in the first trial of the InfoTrain journal were 36 students from the subject Information Services 3, within the School of Communication and Information Studies at the University of South Australia. Six weeks of their regular teaching was set aside and work toward the InfoTrain journal formed part of the final assessment for the subject. Outcomes for students were to be an understanding of the world wide web and a first-hand experience of electronic publishing. Outcomes for the InfoTrain committee would be a knowledge of the suitability of the InfoTrain model of publishing.
After an initial exploratory period each student was expected to submit one 2000 word `scholarly' article relating to the area of information management. This would then be published in volume one of InfoTrain. It was expected that articles would comprise work already submitted and accepted in other subjects. Thus the articles would be validated by the university system. Students marked up their work in HTML and linked suitable graphic files to support their articles. These articles were immediately placed on the InfoTrain server and made available to all students (and, by default, the rest of the world).
Each student was assigned one other student's article to referee. The student being refereed was not told the name of the refereeing student. To assist students with their role of referee a lecture on the nature of refereeing in scholarly journals was given. Students were then given a paper-based referee form [HREF 4] to be completed within 7 days. Students accessed the articles of their peers through the InfoTrain server using web clients in student computer pools.
The intention was for each referee form to be returned to the author after initial inspection by the editors. The author would then make corrections before final submission. However there was insufficient time during the six week teaching block and the referee forms were not returned to the authors until after the end of semester. This has meant that the recommendations made by the referees have not been enacted by the authors.
The `published' articles, as originally submitted, were assessed by the teaching staff on their content and presentation and counted 80% toward the final grade for the InfoTrain section of the subject. The referee forms were evaluated by the teaching staff (editors) and counted 20% toward the final grade for the InfoTrain section of the subject. The criterion for assessment was the appropriateness of referee comments, as determined by the editors, to the original articles.
During assessment the editors noticed errors, omissions, etc. in a number of articles. This information was given to the authors as part of their assessment feedback for the subject. However it was unclear as to how the articles in question could be changed, as the semester had ended. Some students were due to graduate and the motivation (assessment?) for authors no longer existed. It was decided to leave the articles on the server as originally submitted until a decision on their future could be made.
These articles form Volume 1 [HREF 5] of InfoTrain.
While successful as a learning process, in terms of the InfoTrain journal as a product, a number of criticisms were levelled by academic staff at the university. These referred to the Quality of Articles on InfoTrain and the Refereeing Process.
1. It was argued that with the world having easy access to InfoTrain, the University of South Australia in general and the School of Communication and Information Studies in particular are on public show as never before. The fear is that poor quality articles may lead to an adverse impression being formed about the standards required within the university. Of particular concern is the inclusion of articles that :
3. The process of refereeing the submitted articles was considered inadequate. The form provided is too broad and gives inadequate guidance to referees. While the idea of peer refereeing is considered to be a potentially valuable educational experience, the process should ensure that referees have sufficient knowledge of the topic to be able to judge the article's content as well as its structure and format.
In response to these criticisms:
It was clear that reliance on the University system to validate student articles was not successful in this instance. While students submitted work that had been produced as part of University work in other subjects, in the majority of cases this work was submitted for assessment during the same semester as the InfoTrain trial. Therefore it is unlikely that the assessed work was returned to the student before it was submitted to the InfoTrain journal. Thus students were unable to revise their articles, if necessary, before submission to InfoTrain. In the future a clear statement of submission standards and a suggested development path for articles may prove helpful to students. The reliance on the University system to validate (referee) student work on the basis of assessment results from various subjects and courses would seem to be sound. Perhaps work of less than credit standard should be excluded from publication. One suggestion is that students submit evidence of their article's sufficient scholarly worthiness for viewing by the outside world. This could be in the form of a grade, comment, and signature from an academic staff member.
Due to the short time allocated in the first trial the emphasis was on the process of publishing and the involvement of students rather than final product. In the second trial greater emphasis will be placed on quality. If validation by the university is successful then students will peer-review articles that already meet a minimum standard for publication. An acceptable product should be guaranteed with student involvement adding to the quality of published articles.
A major deficiency in the refereeing process was the excessive time taken between refereeing an article and feedback to authors. The intention was to use a web-based form to gather responses. However the web browser available to students did not support forms so a paper-based equivalent was used.
The results of the first trial suggest that some refinement of the refereeing process in InfoTrain is needed. Figure 1 is a new model for refereeing in InfoTrain. The important aspects of this model are as follows:
Figure 1: the new forms-based review model for InfoTrain.
The model operates as follows:
1.1 A student marks up an article in HTML and emails it to the InfoTrain journal. A system administrator (other student) places the article in the restricted access section of InfoTrain. (this activity has the potential to be automated in the future)
1.2 An Editor (other student) is notified by a System Administrator that an article has been submitted (this activity has the potential to be automated in the future).
1.3 The Editor, after preliminary scrutiny, assigns one or more Referees (other student/s) to review the article.
2. The Referee reviews the article by using two windows on his/her web client; one containing the article the other containing a review form [HREF 6]. The form is completed, in one session, and the server forwards it to the Editor.
3. The Editor adds any additional comments and recommendations and emails the review form to the Author. If the article is suitable for publication then step 5 is initiated.
4. The Author revises the articles as necessary and goes to step 1.
5. The System Administrator moves the article from the restricted to the public area.
Note: Steps 1 to 4 may be repeated a number of times to refine an article.
AusWeb95 The First Australian WorldWideWeb Conference