Small changes in large files are not detected properly

[SkimbleShanx]

First of all I must say BTSync is absolutely brilliant! I've been looking for a program like this for a long time and finally it is available !

 

Most of the time the program does a great job, but I'm having some trouble syncing a large Truecrypt volume of 8GB.

 

I've set Truecrypt to change the timestamp after closing the volume. When I open the encrypted volume and add a tiny textfile (or make a small change to the folder structure) and then close the volume again, BTSync picks up the change and starts indexing. After a while a new event is shown in the History tab stating 'Updated file personal.tc'.

 

So, I made a small change to my Truecrypt volume on the 'source machine' and BTSync stated that the volume was updated. But when I open the volume on the 'target' machine there is no change (MD5 checksums of both files are different). It seems like in this case BTSync is able to detect a change in the file, but isn't able to detect the blocks that are changed.

 

Is there anyone who can explain what is going wrong. What can I do to fix this issue (or have this issue fixed by the BT-team)?

 

Thanks in advance!

[GreatMarko]

Have you searched this forum for the word "TrueCrypt", as this topic has already been discussed a number of times.

 

Due to the way that a TrueCrypt volume stores data, any change to it, however small, will result in the entire volume needed to the sync'd & transferred again. It's likely therefore that it's taking Sync a little while to re-index and subsequently transfer the 8GB file each time a change is made.

 

As I say, do search the forum, as there are plenty of other discussions about syncing TrueCrypt volumes and their associated issues

[SkimbleShanx]

Thank you for your response .

 

I've read the discussions about Truecrypt on this forum, but I could not find a similar case. Maybe I've not been clear enough explaining the issue.

 

As you can read in my post, some synchronization on the encrypted container is done by BTSync. As BTSync is able to detect a change to the volume (changed timestamp), it starts syncing (a part of?) the file. The syncing doesn't take very long. After the syncing is done (no more data transfer at all!), the History tab states 'Updated file personal.tc'. This suggests that there was some form of synchronization. But when I open both volumes on the source and on the target machine, the files are still different (target doesn't contain the last changes).

 

In my opinion it doesn't matter if BTSync tries to sync the whole file after a change or just a small part of it, as long as I can be sure that files get synced by BTSync. Being able to synchronize parts of files is a very nice feature, as long as I can trust the program to synchronize all files correctly (no matter what size or type, Truecrypt or no Truecrypt).

[Raccoon]

From what i read and remember fixed size TC containers should be able to be partially synced

the one with non-fixed size were transfered all over again even you change few bytes.

Right?

[GreatMarko]

From what i read and remember fixed size TC containers should be able to be partially synced

the one with non-fixed size were transfered all over again even you change few bytes.

Right?

 

It depends on the position within the file where the data has changed that determines how much of it is re-transferred - Please see "When a file changes, does BitTorrent Sync transfer the entire file again, or just the part that's changed?" in the Unofficial FAQ

[Raccoon]

well then the question is, when you change 1 byte in your files that are in an encrypted TrueCrypt container, does it change just 1 byte in the container data too, or much more due to the encryption?

[SkimbleShanx]

Good question! I'm not sure  . . .

 

Still, if there is a change (no matter how small), it should be synchronized by BTSync.

 

 

In my case the following happens:

• After I close my Truecrypt container, the timestamp is altered (this is a setting in Truecrypt)
• BTSync picks up the change and starts indexing (on source machine)
• After indexing, the status column in the Devices tab indicates an upload of 8 GB, but no transfer is shown in the transfer tab
• There doesn't seem to be any data transmission after that (no transfers in the Transfer tab and the status in the Devices tab still states an upload of 8GB, but in fact nothing happens)

(Note: After a few days the Truecrypt file on the destination machine has not changed. Smaller files are synced without a problem. I use Windows 7 clients to synchronize.)

 

 

I wonder about two things:

• How does the software detect the changed parts of the file? The software is indexing, but nothing seems to happen afterwards.
• Why does the Devices tab indicate an upload (of 8 GB) while no transfers are shown in the Transfers tab?

[Raccoon]

hm that seems to be different problems, once the change is noted i have no problems with syncing big files. CHeck if the destination has enough free space. If it doesn't it wont start without outputting error until there is free space.

 

Now lets return to the original problem.

If on byte in a file in a encrypted container is changed how much does that change the container and does it re-transfer the whole container file or just the 4mb chunk that has the changed bytes in it?

[SkimbleShanx]

Thanks for the response, Raccoon . I still have roughly a terabyte of space left on the destination drive, so storing a couple of gigs on it shouldn't be a problem.

 

Although a very interesting question and the answer to it could solve the problem, finding out how much an encrypted container changes on the outside after a change was made on the inside wasn't the purpose of this topic. What I would like to know is why a change in the container file isn't synced by BTSync after the change was detected by means of a time-stamp change. After the time-stamp change is detected, BTSync starts the indexing process, but at the end, no data is transferred and no changes are made on the destination machine.

I wonder if the algorithm that is used to compare files is profound enough to pick up these changes. Maybe the software is capable of detecting a change, but is not capable of determining the exact bytes that are changed. 

[albertramsbottom]

I think you doing this the wrong way round.  Create a truecrypt container and store your bittorrent sync and drop box folders inn it/  Have it mount on start up and just enter your password to unlock the container.  You can use cloud fogger to encrypt your sync from dropbox etc.

 

This way you only sync the files in the truecrypt container