Castor

Entire file or parts of file?

Recommended Posts

Does SyncApp upload the entire file or only parts of file that are modified? I have a 11 MB zip file that got uploaded entirely when I made a change in one of the spreadsheet file in the zip.

BTW, the zip file i used is password protected and has the extension .zipx. It has only .xlsx file in it which was also password protected. The xlsx file has many sheets. I made the change in one of the sheet.

Share this post


Link to post
Share on other sites

Dropbox does that, for couple of reasons: it saves them money on harddrive space and network traffic. Which actually makes perfect sense. In case of Sync, it might be faster to just transfer the file, rather than calculate difference and then apply it to the file.

Having in mind that all files are binary nowadays, this is not as effective as difference for big text files.

Anyway we have this in mind, however importance of this feature is questionable, of course if leave aside marketing noise about this feature.

Share this post


Link to post
Share on other sites

Also ... if the file is encrypted, it will not matter if you have diff, because encryption often (usually?) modifies the whole file.

So, yes, rsync and Dropbox might do this by chunking for normal binary or text files, but for encrypted ones it doesn't really help.

Share this post


Link to post
Share on other sites

It's very important feature.

Sync should re-check the file if it was modified and resync only modified pieces of the file, no need to resync entire 4 GB file if only 1 byte was changed in it.

Share this post


Link to post
Share on other sites

What kind of files do you guys want to sync between your computers?

I think sharing 4 GB files is not very common. Most people share documents up to a couple of hundreds of kB, maybe some pictures with some mB.

Share this post


Link to post
Share on other sites

You really think, this is a questionable feature? oO

Delta copying (or syncing) saves bandwith & time and both things are very valuable things (at least in most countries where your upstream is rather limited). I've used rsync for my cloud backups so far (under Windows *urgs*) just because it's so efficient because of it's delta copying feature...

Share this post


Link to post
Share on other sites

Don't get me wrong, we do understand importance of the diff and its advantages. And actually Sync support delta copying if file size stays the same.

However implementing rsync like diff in distributed environment without dedicated server, is kind of complex task.

Share this post


Link to post
Share on other sites
Sync support delta copying if file size stays the same

Changed 1 byte in 200 MB file and Sync started sending the whole file. Maybe piece size is more than 4 MB or Sync couldn't determine which part of the file changed?

Share this post


Link to post
Share on other sites

Most of my files are video editing projects (500MB-10GB file size) and photos (20-50MB file size), so having differential sync would save a huge amount of time and bandwidth as it wouldn't have to send a whole 2GB file after I changes a small part of it

Share this post


Link to post
Share on other sites

I'd just like to throw in my support for this feature. I know a lot of people (including myself) that will use this for keeping scientific datasets in sync, and it would be problematic to re-transfer several gigabyte files when one line changes.

Share this post


Link to post
Share on other sites

We got it. We'll do it. This year.

This is awesome. This is the last ingredient missing from being my personal holy grail of sync tools! +1 from me.

(use case: TrueCrypt volumes)

Share this post


Link to post
Share on other sites

If you read carefully, Sync already has diff that works on block level and fits perfectly with TrueCrypt. Since it is drive mirror of the same size, it is changed by blocks. Sync will not transfer blocks that are similar between peers.

The rsync diff is used when you add/remove some data from a file. Sync doesn't handle this.

Share this post


Link to post
Share on other sites

The rsync diff is used when you add/remove some data from a file. Sync doesn't handle this.

Ah, I didn't completely understand until you said that. Now it makes sense. I was trying to sync back an entire VM and it ended up transferring the entire 19GB, which filled the disk on the other end and broke the transfer -- luckily leaving the old intact file in the trash folder. Now I'm wondering if I could get the diff functionality you mentioned simply by setting the VM to allocate the entire virtual disk. I will check back if I find success...

Share this post


Link to post
Share on other sites
Now I'm wondering if I could get the diff functionality you mentioned simply by setting the VM to allocate the entire virtual disk. I will check back if I find success...

I'd be interested in your results - I'm hoping to use Sync to synchronise all my virtual machines between two computers where each VM is around 25Gb.

Share this post


Link to post
Share on other sites

Well, it took a few days to get going. I originally transferred the vmdk to my home machine using BTSync prior to my post. I tried syncing with the dynamic disks, but it wanted to transfer the whole 20GB files. I don't even have 20GB of free space on either host disk, so that failed. That's when I saw kos13's post about the file sizes.

So, to get matching file sizes, I converted both vmdks to fixed disks with the VboxManage tool. Their sizes and file names were identical. The vmdk on my home machine had an earlier modified date than the vmdk on my work machine. Upon trying the sync, BTSync still wanted to transfer the entire files. Additionally, it wanted to send the older vmdk up to my work machine to replace the newer one. Needless to say, the whole process failed due to lack of free space. I managed to recover the vmdk too.

My next step is to ditch the home fixed vmdk and resync the work one down, then try to make changes to the home one and see if it transfers only chunks. It's probably going to be another day or so before that experiment is complete.

Share this post


Link to post
Share on other sites

Aaand... that didn't work either. This time I used BTSync to transfer down the fixed vmdk from my work to home. After using the work VM and shutting down, BTSync proceeded to sync down the entire 20GB file. So it looks like the current version of BTSync does not support this use case. I'm confident that it will be able to do so eventually.

Share this post


Link to post
Share on other sites

Delta sync is one of two features that I'd love to see added to BTSync. +1 for me. The other feature is one of a sort of global hash table where two hashes of two files that are identical wouldn't have to be transferred because that block of data is already at the destination. According to this thread, they're going to do it:

Perhaps these two features are related. I'm not a developper so I don't know.

What's the roadmap for proposed features? When can we expect a version 1.0 release?

Share this post


Link to post
Share on other sites

Is there any activity on this feature yet or can I expect it to be a long way off yet?

This is a VERY old thread! - please see the Unofficial FAQ entry "When a file changes, does BitTorrent transfer the entire file again, or just the part that's changed?"

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.