Characteristics of .au Websites: An Analysis of Large-Scale Web Crawl Data from 2005

Robert Ackland, Centre for Social Research, Research School of Social Science, The Australian National University, robert.ackland@anu.edu.au

Amanda Spink, Faculty of Information Technology, Queensland University of Technology, ah.spink@qut.edu.au

Peter Bailey, Information Engineering laboratory, ICT Centre, Commonwealth Scientific and Industrial Research Organisation, peter.bailey@csiro.au

Keywords

Websites, Australia, CSIRO, commercial, social.

Abstract

This paper presents a preliminary analysis of Websites contained in a large-scale crawl of the .au domain that was collected by the Commonwealth Scientific and Industrial Research Organisation (CSIRO) in late 2005. Our analysis is based on a 10 percent random sample of the CSIRO crawl dataset, which contains around 10 million Web pages from approximately 200,000 Websites. This paper represents the first step in a larger planned project, which we title the “.au Census”, the aim of which is to use large-scale Web crawls to conduct research into commercial and social aspects of the Australian public Web. The primary aim of the present paper is to analyse the data’s properties. We find that the dataset predominantly consists of commercial Websites (78 percent of sites are “.com.au”) while .edu.au, .org.au and .net.au sites each account for around 5-6 percent of all sites (and .gov.au sites are approximately 2 percent).