Scorp1us

Why not just use RSync?

Recommended Posts

So RSync is good at transferring files, and the differences between them. Meaning added, deleted or changed blocks. And given that you create/modify at one location at a time. Rsync seems a better fit.

Bittorent is engineered for getting 1 file from many sources. You're not going to have many sources in your private cloud.

So why should we use bit torrent. I can't think of any reasons to, but many not to:

- ISP throttling/blocking.

- RSync is more efficient.

Share this post


Link to post
Share on other sites

If RSync is better geared for your use case, use it.

BitTorrent Sync is intended as a replacement/supplement to Dropbox and other cloud storage systems.

Additionally, BitTorrent Sync is many to many file transfers, where rsync is one to one. When dealing with large numbers of systems to keep synchronized, BitTorrent Sync scales better once the origin system's upstream is saturated.

Share this post


Link to post
Share on other sites

You're not going to have many sources in your private cloud.

I beg to differ! I currently sync between 7 devices (including two always-on WHS's)

So why should we use bit torrent. I can't think of any reasons to, but many not to:

- ISP throttling/blocking.

1) You can change the ports that BitTorrent Sync uses, as well as choose TCP over UPD - it would be difficult for an ISP to completely block BitTorrent Sync

2) You can separately limit the upload/download rates of BitTorrent Sync - so you can easily tailor BitTorrent Sync to your connection speed/available bandwidth

So why should we use bit torrent. I can't think of any reasons to, but many not to:

- RSync is more efficient.

I may be wrong, but doesn't RSync only transfer a single file at a time? BitTorrent Sync can transfer multiple files concurrently!

Give BitTorrent Sync a go! You'll then be able to more accurately determine its capabilities/performance when compared with RSync! :)

Share this post


Link to post
Share on other sites

True, but it would be interesting to see in some tests who performs better.

Given that most changes are small incremental ones, rsycn should be competitive for a large number of test cases.

I thing you'd need an organization to generate enough load that rsync starts to falter. Which is fine, but when I hear "dropbox" I think individual.

Share this post


Link to post
Share on other sites

Dropbox is rsync made for people. It is nice and perfect. Sync is about moving huge data, try to move 300Mb file via Dropbox and Sync and you fill the difference.

Sync never planned to be a replacement for Dropbox/rsync, while it has some features that looks the same. If you need to send parents 500Mb of your kid's photo/video - you don't need a rsync, right?

We already have some diff based transfer, however not rsync like. Sync can't detect shifted blocks of data. If block wasn't shifted, it will not be send again.

Share this post


Link to post
Share on other sites

Forgot your original question :) I am not aware of any rsync based solution that is serverless and distributed. Sync is truly server-less and distributed, this is why it is not rsync based.

Share this post


Link to post
Share on other sites

As an aside, I just find it amusing that it is 2013, 50 years after the first filesystems and 20 years since the took off web we're still inventing ways to move files between computers.

Share this post


Link to post
Share on other sites

Forgot your original question :) I am not aware of any rsync based solution that is serverless and distributed. Sync is truly server-less and distributed, this is why it is not rsync based.

It depends on how you define "server" RSync can operate on it's own by comparing two files. If one of the files is remote, it is heavily benefited by being able to calculate the checksums locally, then comparing them. So yes, there may be a "server" on the remote system that reads the file and transfers the checksums. Then once that is done, the server accepts and applies the incoming deltas.

Share this post


Link to post
Share on other sites

Yes, this was a surprise to us too, that Sync is the only effective way to send 1Gb between 3 computers.

This is an interesting claim. No matter what protocol, you have to send 2gb to have the same file on all computers.But I am interested in what happens next. If you modify the file, what has to be transmitted? Perhaps it is just publish/download. But once distro I used only provided Rsync of ISOs so you'd only get the 20mb or so of changes. How does bittorrent respond?

To me one of the advantages of Bittorrent is when you have lots of seeders. But if you have a microcloud of 3, I don't see BT doing nearly as well.

Share this post


Link to post
Share on other sites

This is an interesting claim. No matter what protocol, you have to send 2gb to have the same file on all computers.But I am interested in what happens next. If you modify the file, what has to be transmitted? Perhaps it is just publish/download. But once distro I used only provided Rsync of ISOs so you'd only get the 20mb or so of changes. How does bittorrent respond?

We made a first step in this direction. We understand importance of rsync, but it will be next step.

To me one of the advantages of Bittorrent is when you have lots of seeders. But if you have a microcloud of 3, I don't see BT doing nearly as well.

You are right, this is one of the advantages. Another one is uTP and direct connection between peers. To give you real example, try to pass 500 Mb to your friend. If you don't mind, compare the speed of Dropbox and Sync. This will surprise you.

Share this post


Link to post
Share on other sites

  • Get around the problem of firewalling
  • Cross platform
  • I bet Android and iPhone get their own apps over time
  • It'll be so simple sh33pl3 can use it

So RSync is good at transferring files, and the differences between them. Meaning added, deleted or changed blocks. And given that you create/modify at one location at a time. Rsync seems a better fit.

Bittorent is engineered for getting 1 file from many sources. You're not going to have many sources in your private cloud.

So why should we use bit torrent. I can't think of any reasons to, but many not to:

- ISP throttling/blocking.

- RSync is more efficient.

Share this post


Link to post
Share on other sites

Why bicker about it? rsync is still there as it always has been. Use the appropriate tool for the task at hand.

Very much so. But I suggest you take it further. Use the appropriate protocol for the task at hand. If BT would be inferior, switch to Rsync. Then you have one tool that can work better in more situations. But the important distinction is that the computer knows, so the computer picks. The choice can be hidden from the user and they just see lots of win.

Share this post


Link to post
Share on other sites

Good questions the one Scorpius asks. As far as I know, if BTSync uses the bittorrent protocol, an approach to detect file variations would be something like a systems that "recalculates" the chunks in which a file is partitioned and then compare the chunks with variatons? Is that feasible?

Share this post


Link to post
Share on other sites

Good questions the one Scorpius asks. As far as I know, if BTSync uses the bittorrent protocol, an approach to detect file variations would be something like a systems that "recalculates" the chunks in which a file is partitioned and then compare the chunks with variatons? Is that feasible?

BTSync does that. However this is not how rsync works. If you will add just one byte, all chunks will be changed. You need to detect shifted chunks.

Share this post


Link to post
Share on other sites

So, correct me if I'm wrong, but I just spent more time trying to figure out how to use rsync with two dynamic IP addresses over the internet than the time it took me to install Sync on two machines and start syncing. Keeping in mind that these two machines are windows and linux(ARM at that).

Rsync is not for everyone. I know my way around a computer and it wasn't just "simple" to do what Sync does out of the box.

Sync:

me: Download this, install it, use this key.

friend: OK

Rsync:

me: can I come over and set this remote backup thing up?

friend: This is more trouble than it's worth.

Or something along those lines.

Share this post


Link to post
Share on other sites

So, correct me if I'm wrong, but I just spent more time trying to figure out how to use rsync with two dynamic IP addresses over the internet than the time it took me to install Sync on two machines and start syncing. Keeping in mind that these two machines are windows and linux(ARM at that).

Rsync is not for everyone. I know my way around a computer and it wasn't just "simple" to do what Sync does out of the box.

Sync:

me: Download this, install it, use this key.

friend: OK

Rsync:

me: can I come over and set this remote backup thing up?

friend: This is more trouble than it's worth.

Or something along those lines.

Yes, I have to share that opinion too, BTSync offers a near-zero learning curve, at least for the basics. RSync is a very potent application no doubt about that, but it is not for my granma for instance, not even my little brother. So the user adaption on the utilty is a very important matter.

With respect to the mechanism that BTSync uses to detect and propagate file changes. The time taken to re-calculate the chunks of the changed file is dependent on the size of it right?. How much would that be for 1GByte, lets say?

Share this post


Link to post
Share on other sites

That's not a fair comparison. If someone wrote Sync with all the features it has, which uses BT, then a fair comparison is if someone wrote a comparable Sync for RSync. BT and RSync are the protocols, Sync is a app using the protocols.

Share this post


Link to post
Share on other sites

The problem I have with labeling this thing as a good thing for the "cloud" is cloud files are not the same as general torrents. Torrented files are copies of an original that never changes. With cloud files,, there is a higher expectation that this is a working file.

Share this post


Link to post
Share on other sites

BTSync does that. However this is not how rsync works. If you will add just one byte, all chunks will be changed. You need to detect shifted chunks.

Can you elaborate on what Sync is capable of detecting and how it responds?

Share this post


Link to post
Share on other sites

Sync splits file to 4Mb chunks, if one chunk was changed - it will be transferred. It works well on files that doesn't change the size, for example TrueCrypt volume.

If you add one byte in front of the file, all chunks will be shifted and Sync will not be able to detect shifted data.

Share this post


Link to post
Share on other sites

That's not a fair comparison. If someone wrote Sync with all the features it has, which uses BT, then a fair comparison is if someone wrote a comparable Sync for RSync. BT and RSync are the protocols, Sync is a app using the protocols.

So, IF someone wrote a program that uses Rsync and was as easy to set up and use as Sync is now, the only benefit I see is when syncing two machines. When you add more machines to the mix, the distributed nature of BT is a clear winner.

And this is still a big IF someone made a very simple to use Rsync program, which there might very well be.

The way I see it, linux is by far a more powerful OS than windows. Why do people use windows(when they have a choice)? Ease of use. People love stuff that just works.

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.