Figure 1 – G2:P2P System Overview
Richard Mason [HREF1], Centre for Information Technology Innovation, [HREF2] , Queensland University of Technology [HREF3], r.mason@qut.edu.au
Dr Wayne Kelly [HREF4], Centre for Information Technology Innovation, Queensland University of Technology , w.kelly@qut.edu.au
(This work was supported by Microsoft Research)
Peer-to-peer (P2P) computing has significantly broadened the range of applications used on the web. By breaking the client/server model imposed by web browsers, P2P systems have allowed ordinary web users to easily supply their own resources to the internet community without relying on third party servers. However, current P2P applications have primarily concentrated on sharing files (Gnutella, Freenet). We present a fully decentralised P2P network that allows users to share computing cycles. Furthermore, we allow applications to be developed for this network using the Microsoft .NET Remoting infrastructure. This provides a familiar programming model for developers and offers the potential for porting existing .NET Remoting and Java RMI applications relatively simply.
Most modern software applications provide some form of Internet support. While this may be as simple as providing an online update service, increasingly network communication is becoming an integral part of software development. Since programming for the Internet is a reasonably complex task, a variety of technologies have been created to simplify this, including: Web Services, Java Remote Method Invocation (RMI) and Microsoft .NET Remoting. All of these approaches allow programmers to trigger the execution of some code on a remote machine and receive the results of that execution. However, these approaches are all based on a client/server architecture
Our approach automatically distributes remote objects and work requests(remote method calls) amongst the members of a peer-to-peer network. This allows users to share computing cycles similar to how Gnutella allows users to share files. The system, called G2:P2P1, is presented as an extension to the .NET Remoting framework, allowing a simple upgrade path for current users of both .NET Remoting and Java RMI, and providing a familiar interface for all object-oriented programmers.
This paper will outline the design of the system, its integration with the .NET Remoting infrastructure and examine how applications can be written on the system. Section 2 will give you an overview of the system’s design. Section 3 will then provide some background information on P2P networks and .NET Remoting. Section 4 will examine the details of the system before giving an example of its use in Section 5. We then provide a review of other work in this field (Section 6) before presenting the future direction of this work (Section 7) and some closing comments (Section 8).
Figure 1 – G2:P2P System Overview
Although the system consists of a network of peers, machines generally take on one of two personas, volunteers or clients(see Figure 1). Volunteers are machines who wish to share computing power with the network. Sharing is performed by running a generic volunteer process (see Figure 2) which joins a self-organising, fully decentralised P2P network.
Client machines host applications that wish to make use of the computing power of the network. Rather than creating remote objects on a specific server machine, as would happen with regular .NET remoting or Java RMI, our system transparently creates remote objects on one of the volunteer machines within the network. Clients (and volunteers) submit "jobs" and receive results by remotely invoking methods on these objects. By using our custom .NET Remoting channel, the application programmer is freed from the details of organising the volunteer machines and deciding how the objects are distributed on these volunteers. In fact, the majority of the application’s source code will look identical to a normal object-oriented program.
Peers are free to come and go from the network as they please and the system will handle reallocation of objects and ensure the communication channels are maintained. The actual communication on both volunteers and clients is handled entirely by the G2:P2P channel, which does not distinguish between the two personas.
One of the most exciting developments in web programming during the last few years has been peer-to-peer (P2P) computing. There are currently two conflicting definitions of what constitutes a P2P network. For our purposes we define a P2P network as a network which does not assume any central control mechanism. This is sometimes referred to as Pure P2P or fully decentralised. While there are a large number of applications which term themselves as P2P, once our definition is applied a relative few remain.
While P2P adherents often list a large range of applications that could benefit from the P2P paradigm, most current applications seem to be restricted to the file sharing domain. A number of cycle sharing applications, such as SETI@Home[HREF5] and GIMPS[HREF6], are referred to as P2P; however, their actual design is entirely client/server. A P2P design could offer significant benefits to cycle sharing projects.
The most noteworthy benefit of the P2P paradigm is scalability. With suitable design, P2P networks can scale to include thousands of members whereas to provide similar scalability in a client/server application usually requires a costly server or servers (web farm) with significant bandwidth requirements. P2P design allows these costs to be shared amongst all of the network members.
It should be noted that not all P2P designs scale well. In particular, early decentralised systems like Gnutella[7] produced excessive network traffic. One approach to combating this is to use a hybrid between client/server and pure P2P architectures. While this approach has benefits, we are not considering it in our definition of P2P as its greatest benefits are only revealed during distributed searching.
Improved communication is another benefit of a P2P approach to distributed computing. Unlike client/server architectures, all participants in the network are capable of communicating with each other. While this may be simulated in client/server systems by passing communication via the server, this once again forces all communication through a single bottleneck.
Designing P2P systems, however, is not a simple task. Unlike client/server models, P2P systems have no controlling entity. Tasks such as distributing content on the network must be performed using a decentralised algorithm. Each member of the network must use its own local knowledge to create a global effect.
The Microsoft .NET platform contains substantial support for web based programming. This is most notable in two areas, web services and .NET Remoting[HREF7]. .NET Remoting, the successor to DCOM, allows programmers to create distributed applications where objects are hosted on remote server machines. This approach, which is analogous to Java RMI, allows network communication to be hidden beneath method calls, significantly simplifying design and implementation for the programmer.
In contrast to RMI, .NET Remoting provides a highly extensible framework which allows various channels and formatters to be plugged-in as required. Channels are responsible for performing the actual network communication according to a specific network protocol. The .NET framework comes with two standard channels for the TCP and HTTP protocols. Formatters are responsible for serialising objects prior to communication across a channel. The .NET framework provides a binary formatter, as well as a SOAP formatter. .NET Remoting also allows for the inclusion of arbitrary message sinks that can be used to filter/transform both incoming and outgoing messages. These can be used to implement high-level features such as transaction management or single threading somethingorother similar to the method interception performed by containers in Enterprise JavaBeans. Figure 3 shows how channels, formatters and message sinks collaborate to perform a remote request.
Figure 3 – .NET Remoting - Channels and Formatters
Development of .NET Remoting applications is reasonably similar to standard object-oriented development. The problem is decomposed into a set of objects and their interactions are specified. The one additional step required is to specify which objects should be hosted remotely. Remote objects must be of a class derived from the MarshalByRefObject type, as shown2:
The actual Remoting system is configured through an XML file. This file is loaded using the RemotingConfiguration class:
The remoting configuration must be performed before any remote objects are created. Details of which types are remote and which formatters, channels, and messages sinks are to be used are contained in the actual configuration file. The following example marks the MyRemoteObject type as a remote object which should be contacted on the "g2.fit.qut.edu.au" server using the TCP protocol. Since no formatter is specified, the default TCP formatter (binary) will be used.
The most important sections are the channels element and the client element. The channels element contains all of the channels that should be registered for the application, while the client element lists the types that should be remotely activated.
The configuration file for the server is similar to this, but has a service element instead of a client element. There are also a multitude of additional options that may be specified through remoting configuration files.
Remoting configuration can also be set programmatically via the RemotingServices class; however, using configuration files allows the configuration to be adjusted without recompilation.
From the programmer's perspective, remote types are created and used identically once the remoting configuration is loaded:
The .NET framework will automatically create a transparent proxy class during the call to new. This is significantly different to Java RMI, which requires programmers to use a tool to generate proxy classes for acessing remote objects. .NET Remoting provides this effect without any additional effort from the programmer.
The most difficult design problem for P2P systems is the communication protocol. The first popular pure P2P system, Gnutella, was infamous for producing excessive network bandwidth; bringing networks to a crawl and essentially excluding dial-up users from the network. A number of research projects have since addressed this issue and created more efficient protocols.
The Pastry[11] project provides a distributed hash table (DHT) feature on a fully decentralised P2P network. The system takes advantage of network proximity, that is the physical relationship between nodes on the network, and allows objects on the network to be discovered using O(logN) routing hops (where N is the number of nodes on the network).
Pastry routing is based on the concept of a logical node space. Each node in the network is assigned a NodeID which represents a location within the circular node space (Figure 4). NodeIDs consists of a sequence of L, 2b-bit digits. The nodes also maintain O(logN) routing information which is used to direct messages to their targets. Scalability of the routing information is essential to minimise memory requirements on each node and to limit the amount of communication as nodes join and leave the network.
Figure 4 – Pastry Node Space – L = 4, b = 2
Example Routing - 2011 -> 3223
The Pastry routing information consists of three sections: a routing table, a leaf set and a neighbourhood set. The neighbourhood set holds the nodes which are physically closest to the node. The Pastry protocol does not specify how physical locality is tested, but possible options include the number of IP routing hops or the latency (measured empirically). The neighbourhood set is primarily involved in choosing which nodes to store in the routing table to ensure good proximity properties.
The routing table is responsible for the majority of the Pastry routing process. When a node receives a message it is responsible for passing it onto a node whose NodeID has at least one extra digit in common with the message’s target. For example, in Figure 4, if node 3301 receives a message sent to 3223, it must pass that message on to either node 3222 or 3210 as they are the only nodes with a prefix of "32". The routing table maintains a set of Nodes with varying digits in common with the containing node. This table is used to quickly discover a suitable node for passing the message on to.
The leaf set is used to maintain the set of nodes which are logically closest, that is whose NodeIDs are closest to the containing node’s. This is used for the final hop when routing a message to ensure that the message arrives at the absolute logically closest node.
The G2:P2P system is presented to application programmers as a custom channel for .NET Remoting. This channel implements the Pastry protocol to distribute objects amongst the members of a P2P network. The channel is useful for transparently distributing computation amongst a P2P network and forms the basis of a network for easily sharing computing cycles.
Because the G2:P2P system is entirely contained within a custom channel, most existing .NET Remoting programs could take advantage of the system by simply changing their configuration file. Additionally Java RMI applications use a similar abstraction for network communication allowing their design to be quickly ported to .NET Remoting and similarly reap the G2:P2P system’s benefits. Unfortunately, Java RMI does not have the extensible structure required to provide cycle sharing in the same manner.
While most custom remoting channels are implemented using a single message sink for the client and another message sink for the server, our implementation consists of a number of message sinks on each side. This is necessary since this channel has a number of properties that differ from the standard remoting architecture. These differences will be highlighted in this section.
The major feature of the G2:P2P system is its support for transparently dispersing objects amongst a network of machines. Since .NET Remoting traditionally uses a client/server architecture, our channel must be capable of assigning objects to machines in the P2P network, along with handling the actual communication to these objects.
All objects in .NET Remoting are externally referred to via a unique Universal Resource Identifier(URI). A typical URI when using the TCP channel may be tcp://servername/{Object GUID}. This URI consists of three parts:
A similar URI must be created for objects when using the G2:P2P channel. The protocol description can simply be adjusted to reflect the channel (g2p2p://) and the object identifier may stay identical; however, the server address does not have a simple equivalent, in fact this must be dynamically assigned when the object is created.
Since the channel uses Pastry routing for its actual communication, the assignment of object location should reflect the addressing scheme of Pastry. Pastry uses NodeIDs to route messages through the network. By assigning a NodeID to each object as it is being created, we can make use of Pastry routing to decide the actual machine that will host the object. To ensure a reasonable balance of objects amongst volunteers the NodeID should be evenly distributed about the node space. This is achieved by randomly assigning the NodeIDs to objects. Once a NodeID is associated with an object we refer to them as ObjectIDs. Incorporating the ObjectID into the URI gives the following format: g2p2p://ObjectID. Note that we have dropped the {Object GUID} as it is unnecessary once the ObjectID has been created. Since this URI is developed dynamically, a message sink must be used to associate it with the object's proxy.
To allow multiple independent G2:P2P networks to co-exist, we also extend the URI to include a network name. These network names are the closest analogy to the servername section of TCP or HTTP URIs. Each G2:P2P network can be thought of as a single logical server which is capable of supporting multiple remote objects in parallel. By using different network names we allow for completely separate logical servers to be referenced through a common G2:P2P channel. This gives us the following as the final format for our URI: g2p2p://network/ObjectID.
A noteworthy benefit of using an ObjectID to refer to objects, is that it is independent of any particular volunteer. If objects were referenced via their volunteer's physical address (ie. their IP number) then object URIs would become invalid when the volunteer left the system. Instead, our system allows volunteers to migrate their objects to a neighbouring node before leaving. Since objects are referred to via an ObjectID, the Pastry routing will automatically adjust to ensure future method calls reach the object unimpeded.
Another significant difference with the G2:P2P channel is that while standard .NET Remoting specifies remote types on both the client and the server side, our system uses a generic application on each volunteer machine. This allows a variety of applications to be run on the system without requiring the volunteer to about which types will be remoted. To handle this, the channel must be able to dynamically specify which types are being remotely served.
This feature is supplied using the CustomActivatorSink message sink. The CustomActivatorSink is inserted on the receiving side of the channel and allows the channel to provide a specialised activator class. An activator class is used in .NET to handle the construction of objects. In .NET Remoting a unique remoting activator is used to handle any activation messages sent over the remoting framework. This activator uses the remoting configuration to decide whether a particular type should be allowed to be activated on the server. Our custom activator allows objects with types that aren’t registered with the remoting configuration to be created to allow arbitrary applications to use the same generic volunteer.
The activator also separates objects into their own AppDomain. AppDomains are like a mini OS process and provide a boundary for security and component loading. By separating objects into their own AppDomains, we automatically ensure that objects from different applications do not conflict with each other. Additionally we can restrict the security permissions for an object. .NET provides a fine-grained security framework called Code Access Security. By restricting an AppDomain’s permission set we can ensure that volunteer machines are protected from dangerous activities such as accessing the local file system or executing unsafe code.
A significant benefit of the G2:P2P system is its ability to execute methods on multiple objects in parallel. This allows applications with heavy computation to improve their performance by distributing the computation around a network of volunteers. While standard .NET Remoting also allows this style of work dissemination, the G2:P2P channel greatly simplifies it by transparently handling the allocation of objects to volunteers. It can also automatically adjust which volunteer objects are allocated to as the size of the network changes.
Figure 5 – Mandelbrot Generator Client
To demonstrate this ability, a sample application has been created which calculates and displays the Mandelbrot fractal. We use a relatively simplistic method of parallelising the application. The image is split into N regions and a remote object, of type MandelbrotSection, is created to handle each of these regions. The image is created by calling a CalculateMandelbrot method on each of these objects and collating the results. Figure 6 gives an overview of the Mandelbrot application’s design.
Figure 6 – Mandelbrot Generator Design
To allow the Mandelbrot to be calculated concurrently, some means of starting multiple method calls in parallel is required. The .NET framework provides built-in support for this using asynchronous method invocation. To use asynchronous method invocation a method must be invoked through a delegate. A delegate is similar to a function pointer in C; however, they are strongly typed and may include an object reference, allowing instance methods on objects to be accessed.
All delegates supply a BeginInvoke and EndInvoke method for asynchronous invocation. BeginInvoke starts execution of the method and immediately returns, allowing the method to execute concurrently in another thread, process or machine. EndInvoke can then be used once the method has finished executing to retrieve its results. There are two methods of detecting when the method has finished. During the call to BeginInvoke an AsyncCallback delegate can be provided. This will cause a specified method to be called when the results are ready. Alternatively the caller may use the IAsyncResult object returned by BeginInvoke. This object includes a boolean property, IsCompleted, which indicates if the method is finished. It also includes a WaitHandle which can be used to block until the method finishes. The following code shows how the Mandelbrot is calculated and displayed:
The system also allows remote objects to provide feedback to volunteers. This is simply provided for interest to the volunteer and is entirely optional. Feedback is supplied by implementing the IVolunteerFeedback interface:
The InvalidateFeedback event is used by the remote object to indicate when it has new data to display. When this is triggered the volunteer process will retrieve the data via the Feedback property. Figure 2 shows the G2:P2P Volunteer with the feedback from a MandelbrotSection object.
Finally, it should be noted that this application could be run on a local machine or using standard .NET Remoting by simply changing the remoting configuration file. Currently the file is set to use the G2:P2P channel but if the MandelbrotSection type was not specified in this file, the application would be completely decoupled from the G2:P2P system.
Recently, there has been a substantial amount of research into P2P networks. While much of this work has concentrated on specific applications, four projects in particular have developed general purpose P2P routing schemes: Pastry[12], Tapestry[13] , CAN[10] and Chord[12]. While these projects all have slightly different properties, they all provide similar semantics for efficiently routing messages within a P2P network. As such, any of these systems could be used as the basis for the G2:P2P project; however, unlike CAN and Chord, Pastry and Tapestry incorporate locality information in their routing algorithms.
There is also significant interest in application development on these networks. PAST[6] and OceanStore[9] are file storage applications built on Pastry and Tapestry respectively. Files are referenced using FileIDs which correspond to IDs in the underlying routing layer and are analogous to the ObjectIDs in our own work. This method of addressing content on P2P networks has also been used on a variety of other projects such as large-scale event notification[4] and content distribution[3]
While there is currently little work into cycle sharing on pure P2P networks, there is considerable work in this field using client/server or hybrid approaches.
The initial web based approaches, such as SETI@Home[HREF5] and GIMPS[HREF6] were restricted to a single application. Subsequent projects, such as Charlotte[1], Javelin[5] and their subsequent descendents, provided a generic platform for application writers. These systems used the Java platform to provide security and cross-platform support; however, application development requires knowledge of system specific APIs which can be quite complex. These systems initially used a client/server approach to distribute jobs; however, more recent approaches have expanded to a more hybrid approach where networks of servers are used.
The Knitting Factory[2] extension to Charlotte provides direct communication between volunteers using Java RMI; however, their approach relies on a central server passing Java RMI references to the volunteers. This means that references will become invalid when a volunteer leaves the system. Our system avoids this problem by using Pastry routing to automatically adjust communication as machines come and go from the network.
The G2[8] project uses a web service metaphor to simplify application development. Applications are written in the standard .NET web service style and called through a custom proxy class which submits the calls as jobs to the system. This significantly lowers the barriers to developing cycle-stealing applications.
This system supplies a solid foundation for the development of a complete cycle-stealing framework. The current channel supplies the basic means of creating remote objects and executing method calls on these objects. Furthermore, since the system is implemented within .NET Remoting framework, additional features can be added piecewise using message sinks.
The most important aspect not addressed by this system is fault tolerance. Since both P2P and cycle-stealing systems assume that participating machines are volatile, a method for compensating for the frequent coming and going of volunteers is required. Current cycle-stealing systems generally rely on a server to persist job data but the fully decentralised design of our system precludes that option. Other P2P systems generally rely on substantial replication to persist data; however, this may produce too much overhead to be effective in this setting.
The current object dispersal method relies entirely on random assignment of ObjectIDs to distribute objects. Further work needs to be performed to ensure that we make efficient use of the available volunteer computers. Currently it is possible that multiple objects will be assigned to the same volunteer, while one of its neighbours remains empty. A method of ensuring the load is more balanced will be developed. Additionally performance could be increased through methods like eager scheduling as developed by the Charlotte project. However, current eager scheduling techniques are incompatible with the distributed object programming model employed by G2:P2P.
While the system provides a generic volunteer, it does not describe a method for distributing application specific code to these volunteers. Currently the application assemblies must be manually transferred to volunteers. A variety of options are available to distribute this data from basic "download from the client" approaches through to striped approaches that download multiple sections in parallel from other volunteers.
We have presented a simple method for running distributed computing applications on a P2P network. By integrating the system into the .NET Remoting framework we have provided a powerful programming model that should be reasonably familiar to .NET programmers, or indeed programmers of any modern object-oriented platform. The system allows programmers to write parallel, object based systems without having to worry about the details of how many machines are available to perform work or how to distribute the objects on these machines. Additionally the system allows multiple applications to simultaneously use a common P2P network without conflict.
1. Baratloo, A., et al, Charlotte: Metacomputing on the Web, in 9th International Conference on Parallel and Distributed Computing Systems (PDCS-96), 1996.
2. Baratloo, A., et al, An Infrastructure for Network Computing with Java Applets, Concurrency: Practice and Experience, 1998, 10(11-13), p. 1029-1041.
3. Castro, M., et al, SplitStream: High-bandwidth content distribution in a cooperative environment, in IPTPS'03, 2003, Berkeley, CA.
4. Castro, M., et al, SCRIBE: A Large-Scale and Decentralised Application-Level Multicast Infrastructure, IEEE Journal on Selected Areas in Communications (JSAC), 2002(Special issue on Network Support for Multicast Communications).
5. Christiansen, B.O., et al, Javelin: Internet-Based Parallel Computing Using Java, Sixth ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming, 1997.
6. Druschel, P. and Rowstron, A., PAST: A Large-Scale, Persistent Peer-to-Peer Storage Utility, in HotOS VIII, 2001, Schoss Elmau, Germany.
7. Kan, G., Gnutella, in Peer-to-Peer: Harnessing the Power of Disruptive Technologies, A. Oram, Editor. 2001, O'Reilly & Associates, Inc.: Sebastopol, p. 94-122.
8. Kelly, W., P. Roe, and J. Sumitomo. G2: A Grid Middleware for Cycle Donation using .NET, in The 2002 International Conference on Parallel and Distributed Processing Techniques and Applications, 2002, Las Vegas.
9. Kubiatowicz, J., et al. OceanStore: An Architecture for Global-Scale Persistent Storage, in Proceedings of the Ninth International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS 2000), 2000.
10. Ratnasamy, S., et al, A Scalable Content Addressable Network, in ACM SIGCOMM 2001, 2000.
11. Rowstron, A. and Druschel, P., Pastry: Scalable, decentralized object location and routing for large-scale peer-to-peer systems, in 18th IFIP/ACM International Conference on Distributed Systems Platforms (Middleware 2001), 2001, Heidelberg, Germany.
12. Stoica, I., et al, Chord: A Scalable Peer-to-Peer Lookup Service for Internet Applications. in ACM SIGCOMM'01. 2001. San Diego, CA.
13. Zhao, B., K. Kubiatowicz, and A. Joseph, Tapestry: An infrastructure for fault-resilient widearea location and routing, University of California Technical Report: UCB//CSD-01-1141, April, 2000.
[1] This project forms part of a larger G2 project which examines varying programming models and architecture choices for cycle stealing systems. Details of a task-based client/server system called G2:Classic are found in [8]. return to text
[2] All examples in this paper will be given in C#; however, the G2:P2P system is usable from any .NET language. return to text
