Exploring Heterogeneity of Unreliable Machines for P2P Backup Piotr - - PowerPoint PPT Presentation

Exploring Heterogeneity of Unreliable Machines for P2P Backup

Piotr Skowron1, Krzysztof Rzadca2

1p.skowron@mimuw.edu.pl 2krzadca@mimuw.edu.pl University of Warsaw

May 31, 2013

Piotr Skowron, Krzysztof Rzadca Exploring Heterogeneity for P2P Backup

Backup solutions are expensive. The market for backup is vast.

Why p2p backup system? The other alternatives for backup are expensive:

Piotr Skowron, Krzysztof Rzadca Exploring Heterogeneity for P2P Backup

Backup solutions are expensive. The market for backup is vast.

Why p2p backup system? The other alternatives for backup are expensive:

• approx. $1000 for renting 1TB of cloud storage per year

(Amazon, Google, Rackspace or Dropbox),

Piotr Skowron, Krzysztof Rzadca Exploring Heterogeneity for P2P Backup

Backup solutions are expensive. The market for backup is vast.

Why p2p backup system? The other alternatives for backup are expensive:

• approx. $1000 for renting 1TB of cloud storage per year

(Amazon, Google, Rackspace or Dropbox), more than $12,000 for a single backup server with raw capacity

• f 14TB,

Piotr Skowron, Krzysztof Rzadca Exploring Heterogeneity for P2P Backup

Backup solutions are expensive. The market for backup is vast.

Why p2p backup system? The other alternatives for backup are expensive:

• approx. $1000 for renting 1TB of cloud storage per year

(Amazon, Google, Rackspace or Dropbox), more than $12,000 for a single backup server with raw capacity

• f 14TB,

more than $7,000 for a tape-based backup system for 14TB,

Piotr Skowron, Krzysztof Rzadca Exploring Heterogeneity for P2P Backup

Backup solutions are expensive. The market for backup is vast.

Why p2p backup system? The other alternatives for backup are expensive:

• approx. $1000 for renting 1TB of cloud storage per year

(Amazon, Google, Rackspace or Dropbox), more than $12,000 for a single backup server with raw capacity

• f 14TB,

more than $7,000 for a tape-based backup system for 14TB, + cost of maintenence,

Piotr Skowron, Krzysztof Rzadca Exploring Heterogeneity for P2P Backup

Backup solutions are expensive. The market for backup is vast.

Why p2p backup system? The other alternatives for backup are expensive:

• approx. $1000 for renting 1TB of cloud storage per year

(Amazon, Google, Rackspace or Dropbox), more than $12,000 for a single backup server with raw capacity

• f 14TB,

more than $7,000 for a tape-based backup system for 14TB, + cost of maintenence, the scalable solutions are even more expensive.

The market for the backup solutions is vast. E.g. DataDomain in 2009:

Piotr Skowron, Krzysztof Rzadca Exploring Heterogeneity for P2P Backup

Backup solutions are expensive. The market for backup is vast.

Why p2p backup system? The other alternatives for backup are expensive:

• approx. $1000 for renting 1TB of cloud storage per year

(Amazon, Google, Rackspace or Dropbox), more than $12,000 for a single backup server with raw capacity

• f 14TB,

more than $7,000 for a tape-based backup system for 14TB, + cost of maintenence, the scalable solutions are even more expensive.

The market for the backup solutions is vast. E.g. DataDomain in 2009:

• ver 3,000 customers,

Piotr Skowron, Krzysztof Rzadca Exploring Heterogeneity for P2P Backup

Backup solutions are expensive. The market for backup is vast.

Why p2p backup system? The other alternatives for backup are expensive:

• approx. $1000 for renting 1TB of cloud storage per year

(Amazon, Google, Rackspace or Dropbox), more than $12,000 for a single backup server with raw capacity

• f 14TB,

more than $7,000 for a tape-based backup system for 14TB, + cost of maintenence, the scalable solutions are even more expensive.

The market for the backup solutions is vast. E.g. DataDomain in 2009:

• ver 3,000 customers,
• ver 8,000 systems deployed,

Piotr Skowron, Krzysztof Rzadca Exploring Heterogeneity for P2P Backup

Backup solutions are expensive. The market for backup is vast.

Why p2p backup system? The other alternatives for backup are expensive:

• approx. $1000 for renting 1TB of cloud storage per year

(Amazon, Google, Rackspace or Dropbox), more than $12,000 for a single backup server with raw capacity

• f 14TB,

more than $7,000 for a tape-based backup system for 14TB, + cost of maintenence, the scalable solutions are even more expensive.

The market for the backup solutions is vast. E.g. DataDomain in 2009:

• ver 3,000 customers,
• ver 8,000 systems deployed,

was bought by EMC for $2,4 billion.

Piotr Skowron, Krzysztof Rzadca Exploring Heterogeneity for P2P Backup

P2P architecture gives additional possibilities for backup

Why p2p backup system? The p2p backup is cheaper:

Piotr Skowron, Krzysztof Rzadca Exploring Heterogeneity for P2P Backup

P2P architecture gives additional possibilities for backup

Why p2p backup system? The p2p backup is cheaper:

common PCs are cheaper than reliable servers,

Piotr Skowron, Krzysztof Rzadca Exploring Heterogeneity for P2P Backup

P2P architecture gives additional possibilities for backup

Why p2p backup system? The p2p backup is cheaper:

common PCs are cheaper than reliable servers, the unused disk space on the desktop workstations.

The p2p architecture can be explored:

Piotr Skowron, Krzysztof Rzadca Exploring Heterogeneity for P2P Backup

P2P architecture gives additional possibilities for backup

Why p2p backup system? The p2p backup is cheaper:

common PCs are cheaper than reliable servers, the unused disk space on the desktop workstations.

The p2p architecture can be explored:

the bandwidth of many nodes scales better than of a single server,

Piotr Skowron, Krzysztof Rzadca Exploring Heterogeneity for P2P Backup

P2P architecture gives additional possibilities for backup

Why p2p backup system? The p2p backup is cheaper:

common PCs are cheaper than reliable servers, the unused disk space on the desktop workstations.

The p2p architecture can be explored:

the bandwidth of many nodes scales better than of a single server, the load on the network is more evenly distributed causing less bottlenecks,

Piotr Skowron, Krzysztof Rzadca Exploring Heterogeneity for P2P Backup

P2P architecture gives additional possibilities for backup

Why p2p backup system? The p2p backup is cheaper:

common PCs are cheaper than reliable servers, the unused disk space on the desktop workstations.

The p2p architecture can be explored:

the bandwidth of many nodes scales better than of a single server, the load on the network is more evenly distributed causing less bottlenecks, better protection in case of natural disasters (geographical dispersion).

Piotr Skowron, Krzysztof Rzadca Exploring Heterogeneity for P2P Backup

P2P architecture gives additional possibilities for backup

Why p2p backup system? The p2p backup is cheaper:

common PCs are cheaper than reliable servers, the unused disk space on the desktop workstations.

The p2p architecture can be explored:

the bandwidth of many nodes scales better than of a single server, the load on the network is more evenly distributed causing less bottlenecks, better protection in case of natural disasters (geographical dispersion).

How to take advantage from the p2p architecture?

Piotr Skowron, Krzysztof Rzadca Exploring Heterogeneity for P2P Backup

We can take advantage from p2p architecture by smart replica placement

Flexibility in replica placement is important If system design allows that any replica can be placed anywhere, we can choose the best replica placement.

Piotr Skowron, Krzysztof Rzadca Exploring Heterogeneity for P2P Backup

We can take advantage from p2p architecture by smart replica placement

Flexibility in replica placement is important If system design allows that any replica can be placed anywhere, we can choose the best replica placement. Flexibility in replica placement is important The current p2p storage systems do not give this flexibility (rather they use the structured overlays, e.g. DHT).

Piotr Skowron, Krzysztof Rzadca Exploring Heterogeneity for P2P Backup

We design a contract-based system

We build a prototype implementation that is based on mutual contracts. Contract A contract for storing a data chunk of the owner i on the replicator j is a promise made by j to keep is data chunk for a certain amount of time. Contracts can be exploited in two ways:

Piotr Skowron, Krzysztof Rzadca Exploring Heterogeneity for P2P Backup

We design a contract-based system

We build a prototype implementation that is based on mutual contracts. Contract A contract for storing a data chunk of the owner i on the replicator j is a promise made by j to keep is data chunk for a certain amount of time. Contracts can be exploited in two ways: contracts form an unstructured, decentralized architecture that enables to optimize replica placement,

Piotr Skowron, Krzysztof Rzadca Exploring Heterogeneity for P2P Backup

We design a contract-based system

We build a prototype implementation that is based on mutual contracts. Contract A contract for storing a data chunk of the owner i on the replicator j is a promise made by j to keep is data chunk for a certain amount of time. Contracts can be exploited in two ways: contracts form an unstructured, decentralized architecture that enables to optimize replica placement, contracts allow strategies for replica placement that are incentive-compatible.

Piotr Skowron, Krzysztof Rzadca Exploring Heterogeneity for P2P Backup

The architecture of a contract-based system

The system uses 3 storage overlays:

Piotr Skowron, Krzysztof Rzadca Exploring Heterogeneity for P2P Backup

The architecture of a contract-based system

The system uses 3 storage overlays: The control information allows peers to locate and to connect to each other – it is kept in DHT.

Piotr Skowron, Krzysztof Rzadca Exploring Heterogeneity for P2P Backup

The architecture of a contract-based system

The system uses 3 storage overlays: The control information allows peers to locate and to connect to each other – it is kept in DHT. The data is kept in the unstructured overlay – contracts.

Piotr Skowron, Krzysztof Rzadca Exploring Heterogeneity for P2P Backup

The architecture of a contract-based system

The system uses 3 storage overlays: The control information allows peers to locate and to connect to each other – it is kept in DHT. The data is kept in the unstructured overlay – contracts. The meta-data describing replication contracts is kept by both the data owner and the replicator.

Piotr Skowron, Krzysztof Rzadca Exploring Heterogeneity for P2P Backup

Replication contracts

Keeping information about contracts:

Piotr Skowron, Krzysztof Rzadca Exploring Heterogeneity for P2P Backup

Replication contracts

Keeping information about contracts: Each peer keeps contracts in persistent index structure called DataCatalog.

Piotr Skowron, Krzysztof Rzadca Exploring Heterogeneity for P2P Backup

Replication contracts

Keeping information about contracts: Each peer keeps contracts in persistent index structure called DataCatalog. Each peer also keeps contracts that it is responsible for.

Piotr Skowron, Krzysztof Rzadca Exploring Heterogeneity for P2P Backup

Replication contracts

Keeping information about contracts: Each peer keeps contracts in persistent index structure called DataCatalog. Each peer also keeps contracts that it is responsible for. The information about each contract is kept in two places –

• wner and replicator:

Piotr Skowron, Krzysztof Rzadca Exploring Heterogeneity for P2P Backup

Replication contracts

Keeping information about contracts: Each peer keeps contracts in persistent index structure called DataCatalog. Each peer also keeps contracts that it is responsible for. The information about each contract is kept in two places –

• wner and replicator:

periodic removal inconsistencies,

Piotr Skowron, Krzysztof Rzadca Exploring Heterogeneity for P2P Backup

Replication contracts

Keeping information about contracts: Each peer keeps contracts in persistent index structure called DataCatalog. Each peer also keeps contracts that it is responsible for. The information about each contract is kept in two places –

• wner and replicator:

periodic removal inconsistencies, fixing failures.

Piotr Skowron, Krzysztof Rzadca Exploring Heterogeneity for P2P Backup

Contracts are continuously renegotiated

Piotr Skowron, Krzysztof Rzadca Exploring Heterogeneity for P2P Backup

Contracts are continuously renegotiated

Each replica has its utility, quantifying the quality of the replica.

Piotr Skowron, Krzysztof Rzadca Exploring Heterogeneity for P2P Backup

Contracts are continuously renegotiated

Each replica has its utility, quantifying the quality of the replica. The goal of the system architecture is to support any utility function. In our experiments we took multi-criteria utility function; utility is higher if:

Piotr Skowron, Krzysztof Rzadca Exploring Heterogeneity for P2P Backup

Contracts are continuously renegotiated

Each replica has its utility, quantifying the quality of the replica. The goal of the system architecture is to support any utility function. In our experiments we took multi-criteria utility function; utility is higher if: the replicator is less loaded,

Piotr Skowron, Krzysztof Rzadca Exploring Heterogeneity for P2P Backup

Contracts are continuously renegotiated

Each replica has its utility, quantifying the quality of the replica. The goal of the system architecture is to support any utility function. In our experiments we took multi-criteria utility function; utility is higher if: the replicator is less loaded, at least one replica is far from the owner,

Piotr Skowron, Krzysztof Rzadca Exploring Heterogeneity for P2P Backup

Contracts are continuously renegotiated

Each replica has its utility, quantifying the quality of the replica. The goal of the system architecture is to support any utility function. In our experiments we took multi-criteria utility function; utility is higher if: the replicator is less loaded, at least one replica is far from the owner, all other replicas are close (do not overload the network),

Piotr Skowron, Krzysztof Rzadca Exploring Heterogeneity for P2P Backup

Contracts are continuously renegotiated

Each replica has its utility, quantifying the quality of the replica. The goal of the system architecture is to support any utility function. In our experiments we took multi-criteria utility function; utility is higher if: the replicator is less loaded, at least one replica is far from the owner, all other replicas are close (do not overload the network), the replicator has higher availability. The contracts for the replicas with the lowest utilities are

• ptimized – continuous optimization.

Piotr Skowron, Krzysztof Rzadca Exploring Heterogeneity for P2P Backup

The worst replicas are found in distributed priority queue

Each peer keeps a list of d worst known replicas (with version numbers). The lists of different peers are merged through gossiping protocol.

Piotr Skowron, Krzysztof Rzadca Exploring Heterogeneity for P2P Backup

The worst replicas are found in distributed priority queue

Each peer keeps a list of d worst known replicas (with version numbers). The lists of different peers are merged through gossiping protocol.

P1

• Repl. Util.

R1 R4 R5 1 5 8 P1

• Repl. Util.

R3 R2 R6 3 8 9 P1

• Repl. Util.

R9 2 P1

• Repl. Util.

R7 R8 2 4 Piotr Skowron, Krzysztof Rzadca Exploring Heterogeneity for P2P Backup

The worst replicas are found in distributed priority queue

Each peer keeps a list of d worst known replicas (with version numbers). The lists of different peers are merged through gossiping protocol.

P1

• Repl. Util.

R1 R9 R3 1 2 3 P1

• Repl. Util.

R1 R3 R4 1 3 5 P1

• Repl. Util.

R1 R9 R4 1 2 5 P1

• Repl. Util.

R7 R8 2 4 Piotr Skowron, Krzysztof Rzadca Exploring Heterogeneity for P2P Backup

The worst replicas are found in distributed priority queue

Each peer keeps a list of d worst known replicas (with version numbers). The lists of different peers are merged through gossiping protocol.

P1

• Repl. Util.

R1 R9 R3 1 2 3 P1

• Repl. Util.

R1 R3 R4 1 3 5 P1

• Repl. Util.

R1 R9 R4 1 2 5 P1

• Repl. Util.

R7 R8 2 4 Piotr Skowron, Krzysztof Rzadca Exploring Heterogeneity for P2P Backup

The worst replicas are found in distributed priority queue

Each peer keeps a list of d worst known replicas (with version numbers). The lists of different peers are merged through gossiping protocol.

P1

• Repl. Util.

R1 R9 R3 1 2 3 P1

• Repl. Util.

R1 R7 R3 1 2 3 P1

• Repl. Util.

R1 R9 R7 1 2 2 P1

• Repl. Util.

R1 R7 R9 1 2 2 Piotr Skowron, Krzysztof Rzadca Exploring Heterogeneity for P2P Backup

The asynchronous messaging

Each pear has a group of synchro-peers

Sender message Sender sends a mesage to the receiver Synchro-peer Receiver Synchro-peer Synchro-peer Synchro-peers of the receiver Piotr Skowron, Krzysztof Rzadca Exploring Heterogeneity for P2P Backup

The asynchronous messaging

Each pear has a group of synchro-peers

Sender message If the receiver is down it sends the message to any available synchro-peer Synchro-peer Receiver Synchro-peer Synchro-peer Synchro-peers of the receiver Piotr Skowron, Krzysztof Rzadca Exploring Heterogeneity for P2P Backup

The asynchronous messaging

Each pear has a group of synchro-peers

Sender The synchro-peers synchronize the messages Synchro-peer Receiver Synchro-peer Synchro-peer Synchro-peers of the receiver Piotr Skowron, Krzysztof Rzadca Exploring Heterogeneity for P2P Backup

Experiments – settings

We run the experiments of the prototype implementation in two different settings: Students Lab (150 machines):

the amount of local data sampled from the distribution of storage space used by the students on their home directories, the local storage space depended on machines local hard:

10GB (50% machines), 20GB (10% machines), 40GB (40% machines).

PlanetLab (50 machines); every machine:

1GB of local data, 10GB of storage space.

Piotr Skowron, Krzysztof Rzadca Exploring Heterogeneity for P2P Backup

Students Lab is a pesimistic case

0.2 0.4 0.6 0.8 1 probability 0.2 0.4 0.6 0.8 1 availability availability of computers

Figure: The cumulative distribution function of the availability of the computers in students lab.

0.2 0.4 0.6 0.8 1 probability 3 6 9 12 duration [in hours] session duration time between consecutive sessions

Figure: The cumulative distribution function of the session durations and the times between consecutive sessions for the computers in students lab (the nights are filtered out).

Piotr Skowron, Krzysztof Rzadca Exploring Heterogeneity for P2P Backup

Quality of asynchronous messaging

5 10 15 20 25 30 message delivery time [hours] 5 10 15 20 25 30 number of synchro-peers

Figure: The dependency between the number of synchro-peers and the delivery time of the asynchronous message. Delivery time measured from the first time of availability of the receiver after the message was sent.

0.2 0.4 0.6 0.8 1 probability 5 10 15 20 25 30 number of synchro-peers

Figure: The dependency between the number of synchro-peers and the probability of delivery a message to any synchro-peer.

Piotr Skowron, Krzysztof Rzadca Exploring Heterogeneity for P2P Backup

Update time of the replicas

High cost of unavailability

0.2 0.4 0.6 0.8 1 probability 3 6 9 12 15 18 21 24 update time [hours] First replica Second replica Third replica

Figure: The cumulative distribution function for the time

• f creating a replica of data
• chunk. The time is relative to

the data owner. Lab; data collected over 4 weeks of running time.

0.2 0.4 0.6 0.8 1 probability 1 2 3 4 5 update time [hours] First replica Second replica Third replica

Figure: The cumulative distribution function for the time

• f creating a replica. The time is

relative to the data owner. Planet-Lab; data collected over 3 days.

Piotr Skowron, Krzysztof Rzadca Exploring Heterogeneity for P2P Backup

Quality of the optimization protocol

First, in Students Lab, we set the utility so that the size of data should be proportional to the availability of the machine.

Table: The ratio: total size of replicated data (in MB) to the availability for the first 3 days of experiments in the lab environment.

Utility (weighted replicated data) [MB/avail.] day average standard deviation 1 34487 6086 2 60489 8141 3 69658 5496

Piotr Skowron, Krzysztof Rzadca Exploring Heterogeneity for P2P Backup

Quality of the optimization protocol

Second, on PlanetLab, we used the strategy that requires:

• ne all replicas to be as close to the owner as possible (TTL

distance) and the expected update time of all replicas is bounded by 4500s.

Piotr Skowron, Krzysztof Rzadca Exploring Heterogeneity for P2P Backup

Quality of the optimization protocol

Second, on PlanetLab, we used the strategy that requires:

• ne all replicas to be as close to the owner as possible (TTL

distance) and the expected update time of all replicas is bounded by 4500s. We got the following results: The average TTL distance between the replica and the owner was equal to 11.6 (standard deviation equal 3.7). Only 2 machines exceeded their backup window (by < 108 s).

Piotr Skowron, Krzysztof Rzadca Exploring Heterogeneity for P2P Backup

Quality of the optimization protocol

Second, on PlanetLab, we used the strategy that requires:

• ne all replicas to be as close to the owner as possible (TTL

distance) and the expected update time of all replicas is bounded by 4500s. We got the following results: The average TTL distance between the replica and the owner was equal to 11.6 (standard deviation equal 3.7). Only 2 machines exceeded their backup window (by < 108 s). After we changed the weight so that being close is more important to backup window:

Piotr Skowron, Krzysztof Rzadca Exploring Heterogeneity for P2P Backup

Quality of the optimization protocol

Second, on PlanetLab, we used the strategy that requires:

• ne all replicas to be as close to the owner as possible (TTL

distance) and the expected update time of all replicas is bounded by 4500s. We got the following results: The average TTL distance between the replica and the owner was equal to 11.6 (standard deviation equal 3.7). Only 2 machines exceeded their backup window (by < 108 s). After we changed the weight so that being close is more important to backup window: The average TTL distance was equal to 8.1 (standard deviation equal 3.8). Now 13 machines exceeded their backup window:

The average excess of the backup window was equal to 222s, The maximal excess was equal to 415s.

Piotr Skowron, Krzysztof Rzadca Exploring Heterogeneity for P2P Backup

Conclusions

During implementation and initial tests we encountered numerous issues we did not expect: e.g.:

Piotr Skowron, Krzysztof Rzadca Exploring Heterogeneity for P2P Backup

Conclusions

During implementation and initial tests we encountered numerous issues we did not expect: e.g.: updating data catalog remotely whenever any contract is changed is highly inefficient,

Piotr Skowron, Krzysztof Rzadca Exploring Heterogeneity for P2P Backup

Conclusions

During implementation and initial tests we encountered numerous issues we did not expect: e.g.: updating data catalog remotely whenever any contract is changed is highly inefficient, revoking the contracts cannot be done asynchronously changing contracts too often is inefficient,

Piotr Skowron, Krzysztof Rzadca Exploring Heterogeneity for P2P Backup

Conclusions

During implementation and initial tests we encountered numerous issues we did not expect: e.g.: updating data catalog remotely whenever any contract is changed is highly inefficient, revoking the contracts cannot be done asynchronously changing contracts too often is inefficient, each contract must be kept by both the data owner and the replicator.

Piotr Skowron, Krzysztof Rzadca Exploring Heterogeneity for P2P Backup

Conclusions

During implementation and initial tests we encountered numerous issues we did not expect: e.g.: updating data catalog remotely whenever any contract is changed is highly inefficient, revoking the contracts cannot be done asynchronously changing contracts too often is inefficient, each contract must be kept by both the data owner and the replicator. Simulations are not enough We should verify the ideas, in addition to simulations, by constructing prototype implementations.

Piotr Skowron, Krzysztof Rzadca Exploring Heterogeneity for P2P Backup

Conclusions

Cost of unavailability The irregular environments negatively influence the durations of data transfers. Choosing machines with better availability strongly reduces this effect.

Piotr Skowron, Krzysztof Rzadca Exploring Heterogeneity for P2P Backup

Conclusions

Cost of unavailability The irregular environments negatively influence the durations of data transfers. Choosing machines with better availability strongly reduces this effect. We can explore p2p architecture for backup It is possible to build an efficient and reliable backup system, even using weakly-available machines with irregular session times.

Piotr Skowron, Krzysztof Rzadca Exploring Heterogeneity for P2P Backup

Questions? Also, feel free to send any questions to: p.skowron@mimuw.edu.pl.

Piotr Skowron, Krzysztof Rzadca Exploring Heterogeneity for P2P Backup