Helen Ashman [HREF1], School of Computer and Information Science [HREF2], University of South Australia [HREF3], S.A., 5095, helen.ashman@unisa.edu.au.
At Ausweb 2000, we reported on the use of a "war games" exercise to teach Web security in a hands-on environment, after the first year of using this form of assessed exercise. This short paper updates this, reporting on nine successive years of teaching in this fashion, as well as discussing the transfer of the assessed exercise to a new university.
In the first year reported in [Ashman2000], there were 86 students on the course. Subsequent years saw a dramatic rise in the student numbers, partly due to the overall student number fluctuations. In one year there were around 250 students, making it absolutely necessary to run two separate versions of the exercise, in preference to having groups of 11 or more students. The facilities were expanded to meet the demand, with the original Network Lab with 10 machines growing to a lab of 23 machines.
The original course, which taught both cryptography and security, was broken into two separate courses, so that the security was taught on its own. This meant that students were not discouraged from attempting the course by the mathematics component, which was now removed to the other course.
Students are allocated into groups by the lecturer, ensuring an even mix of final-year undergraduates and taught conversion masters students, as well as an even mix of degree types (straight computer science or joint with mathematics or business).
The exercise runs for around 8 weeks, with the first four weeks given over to setting up the group's host with services and tightening security. The second four weeks of the exercise are the "war games" where each group endeavours to hack any or all of the other groups, while defending their own host and services as best they can. The services they had to set up were basic command line terminal (telnet or ssh), file transfer (ftp or sftp or similar) and Web services including forms where arbitrary users could input data as well as plain Web pages and password-protected pages.
As well as attempting to hack other groups' hosts and services, groups are expected to generate significant quantities of non-hostile traffic, which has to be satisfactorily processed by the host services. The mechanism for ensuring a reasonable quantity of this "normal" traffic was to insist that each group regularly (ideally daily) test the normal services of every other group and log their findings. This not only generated plenty of "normal" traffic, but had the benefit of giving excellent feedback on the success and operational status of other groups to the course convenor, and this group-level peer review influences the mark for those other groups.
The assessable deliverables comprised a group report, worth 20% where the group described their activities and included logs of their testing of other groups. The activities included both attempts at hacking as well as defending their own host and services, and were expected to include details such as known facts about what was attempted as well as their own analyses such as how well they believed they handled attacks or how the other party had dealt with their activities.
Each student additionally had a second deliverable, their individual report, where they describe their own contribution to the group's work. This was marked to give 10% of the final result for the course.
The individual report also expected students to peer review each other member in their group, giving a justified mark out of ten. These marks were averaged to give the final 10% of the assessed exercise.
At the University of South Australia the exercise is worth 60% of the course result, which greatly increases the scope for peer review as well as a group-based post-mortem.
The group-based peer review has informally been part of the assessment procedure at the University of Nottingham with the marker taking into account the robustness of each group's host and services, as described by other groups. In the University of South Australia, this component has now been formalised by assigning 10% of the final course result to the group-level peer review.
This mutual assessment at group level is extremely useful for many reasons. Firstly and most obviously, it assists the course marker with assigning a mark to each group for their performance, which would otherwise have had to be done periodically by the marker at the cost of a great deal of time and effort. Since students need to generate "normal" traffic anyway to help disguise the hostile traffic, it is useful to exploit the information they gather in the assessment procedures. Note however that it does not supplant the marker's assessment of the group (which is contained in the 20% group report) but rather supplements it.
A second benefit of the group peer review is that students are involved in the ranking of other groups, and thus gain a real understanding of how well their own group has performed in the exercise. It's no secret that students frequently have no idea how well they perform in exercises and exams (the Dunning-Kruger effect [DK99]) and reviewing the efforts of others goes some way toward dispersing false self-analyses. While the individual peer review is one critical feature in this, the group peer review assesses a different set of characteristics, not the individual but rather the group's success at presenting a robust face to the world, to managing its time and task and resource allocation successfully. The group is a different entity to the individuals making it up, and the success of the group's infrastructure and interactions are assessed in the group peer review.
For the ninth year, at the University of South Australia, the final 10% of the 60% exercise assessment was given over to a seminar series given by the groups, marked by the lecturer, where students present to the entire class a summary of their activities during the exercise, including their analyses of what they believed was happening. This has often been missing from the post-mortems in previous years, or very scanty, and while all the relevant information is contained in the group reports, groups do not necessarily see the reports of other groups and may never know what really happened during the exercise.
The seminar series addressed this weakness, so that students gained first-hand information on the activities of other groups, in particular as it applied to their own group but also more generally. It was also found to generate a good quantity of discussion amongst the students as they compared notes on various events during the exercise.
As a perceived lack of feedback is generally a weakness frequently criticised by students, this post-mortem seminar series is an excellent mechanism for delivering quite specific feedback in a very timely fashion.
In the University of South Australia, the exercise was for the first time run on a virtual network. The supporting software was VMWare which was adequate for the purpose.
However a major drawback of the use of VMWare is that most of the students have had no prior experience with it. This implies an additional overhead of learning as students come to grips with how to use VMWare itself, which is not a core or even required part of the course. Students also need to learn how to install services on a host which in itself is not specific to learning about Web security, although it is critical that they learn how the selection and setting up of services can compromise security. So there is a high overhead in the setting-up stage of the exercise, which creates a high workload for technical support staff and the lecturer as well. So ideally the exercise should run on real machines in a real network, as most students already have knowledge of this environment. For 2008, the exercise is scheduled to be run on real machines in an isolated laboratory to avoid the problems of the virtual environment.
However the scheduling of the course at the University of Nottingham was not ideal for the taught Masters students, as the course was delivered in the first semester of the academic year. This meant that the majority of Masters students had very little, if any, exposure to computer science concepts at all at this early stage of their conversion degree.
This disparity in background was investigated at the end of the 2006-07 academic year in the University of Nottingham. It was found that while the taught Masters students did suffer an overall lower average mark than the undergraduates, the coursework was not responsible for this disparity, and in fact the Masters and undergraduate students had an almost identical overall exercise mark average. Hence it seems that the disparity in results was not in any way due to the exercise. On the contrary, the exercise provided an opportunity for students with lesser technical skills to still make valuable contributions to their group and to some extent overcame the liability of a lack of background in computer science.
In the University of South Australia, the course runs in the second semester, by which stage the Masters students have some exposure to computer science principles, if not the same background as their undergraduate colleagues. There were also around 14 "external" students who were generally unable to come to laboratory sessions. However, the use of discussion groups as well as the need to assign different roles within groups meant that these students integrated well into the exercise.
The popularity of the exercise and course is also reflected in the student uptake, as it was taken by 70% of the 125 eligible undergraduates in the 2005-06 academic year, with the course being compulsory for only 26 of these. This represented the highest selection rate among non-compulsory courses. So the course popularity is reflected both prior to course registration and after the course is completed.
Possible reasons contributing to the course popularity include the obvious relevance of the content, not just to computer scientists, but the community at large. With desktop computers in an increasing number of homes, security of computers must be understood to some extent by all owners, and perhaps it could even be argued that computer security is as basic as home security.
Another motivation for taking the course is that students (and just as importantly, their parents) perceive security to be a potentially highly-employable area. As computers take over more and more aspects of industrial and domestic life, security requirements increase to match. Students feel there is benefit in being able to tell employers that they have hands-on skills in Web and network security.
The adversarial nature of the exercise may also contribute to its popularity, for much the same reason that sporting teams competition is attractive. However it is worth noting that the exercise is indeed adversarial, not just competitive, and that this might discourage some students, for example female students. An area for future research is to determine whether this adversarial exercise is repellent to female students.
One frequent observation by students and others is that students tend to spend well over the nominal time expected of them for the quantity of marks available. In fact, it has been necessary for the lecturer to on occasion remind the students of their other commitments,
The students take the exercise very seriously, perhaps as a result of its adversarial nature. There have been numerous occasions when students urgently seek technical advice, trying to recover their hacked host, whereas urgency tends to usually be reserved for deadlines.
The adversarial form of teaching security is being extended to further courses, with a honeypot exercise planned for the first semester of 2008. However in this course, students will be i) defending their host and services from unknown external threats, and ii) analysing traces of usage for detection purposes.
G5CSEC Computer Security, [HREF5].
Kruger, J. and Dunning, D. (1999) "Unskilled and Unaware of It: How Difficulties in Recognizing One's Own Incompetence Lead to Inflated Self-Assessments", Journal of Personality and Social Psychology 77(6), pp1121-34.