Bug Report: btsync Upgrade can cause removed files to reappear

[hhz]

Ok, here's what happened.

Four machines with btsync version $old. All synced.

Now upgrade three of them to btsync version $new. This has a new index file format and requires the machine to re-index the shared files.

Let $new re-index on the three upgraded machines, but let the fourth machine stay offline.

Then remove a few files, create a few other files and let the three $new machines sync each other.

Finally, boot the fourth machine, upgrade its $old version to $new. It will now re-index its content and re-transmit the old files, which had been removed on the other three machines.

Tada! The removed files are back on all four machines.

[kos13]

This is as designed. Keep in mind that upgrading due to a changes in protocol and indexing is similar to a new Sync version installation. So your case is I have Sync on three machines and deleted a file, then install Sync on a new machine where this file exists. So it will be treated as someone recovered the file.

It might be not exactly how you expect it to work, but this was a very though upgrade for us.

HTH

kos

[hhz]

But that defeats the original purpose of btsync to have it run on many machines that independently sync each other.

If an admin has to make sure that all users of a shared directory must upgrade at the same time and when all machines are in sync, it's not decentralized anymore.

Sorry, I still think that using timestamp and file existence to find out what to sync is the wrong approach. I have been using btsync for a very short time now and already stumbled over sync issues because of this several times.

[hhz]

I presume that an actual change history that can be merged from several distributed machines (a la git-annex or Sparkleshare) would be more robust against the kind of sync issues I keep having with btsync right now.

[kos13]

But that defeats the original purpose of btsync to have it run on many machines that independently sync each other.

Not really, the issue here is that 1.1.27 was a big update and wasn't backward compatible with previous versions. Due to incompatibility it wasn't really an upgrade. We stabilizing Sync and I hope there will be no more such incompatibility cases.

Sorry, I still think that using timestamp and file existence to find out what to sync is the wrong approach. I have been using btsync for a very short time now and already stumbled over sync issues because of this several times.

We know how to do it right, but first step we had to make is versioning. So stay tuned.

[ChrisH]

This is as designed.

Then your design is flawed. An upgrade (regardless of how big the changes are) should never lead to data loss.

I (kinda) get the decision to use the simplest conflict resolution method there is - last added wins -, but that only works for changed files after the first initial sync has been done. For all other cases the user should be asked what to do (e.g. when adding a new folder with an existing secret there should be a option to decide whether to replace local or remote files on conflict), and like I said, a "simple" (for the user) update should not change the status quo.

[ChrisH]

Oh, and please see this thread: http://forum.bittorrent.com/topic/19668-help-newer-files-overwritten-by-older-files/ concerning the same problem.

[kos13]

I presume that an actual change history that can be merged from several distributed machines (a la git-annex or Sparkleshare) would be more robust against the kind of sync issues I keep having with btsync right now.

I agree with you that this is right way to go.

Then your design is flawed. An upgrade (regardless of how big the changes are) should never lead to data loss.

This is definitely a pain and not something we intentionally put to the system, however you have to understand that root cause of the issue - you upgrade machines in the middle of sync. I.e. two upgraded machines perform some changes and then you bring third machine that cannot be aware of these changes due to an incompatible protocol.

[hhz]

ChrisH, please note that I did not experience data loss. I only had files restored that should have been removed.

koz, I did not upgrade in the middle of a sync. When I turned off the fourth machine, its sync had been completed. When I turned it on, the three others were synced, as well, but in a newer state.

Still, the result was unexpected.

[ChrisH]

ChrisH, please note that I did not experience data loss. I only had files restored that should have been removed.

Okay, I had changed files overwritten with their old versions.

koz, I did not upgrade in the middle of a sync. When I turned off the fourth machine, its sync had been completed. When I turned it on, the three others were synced, as well, but in a newer state.

Exactly.

[ChrisH]

however you have to understand that root cause of the issue - you upgrade machines in the middle of sync. I.e. two upgraded machines perform some changes and then you bring third machine that cannot be aware of these changes due to an incompatible protocol.

I don't think that's the root cause, but okay. Like someone else already said in this thread, if you have to centrally synchronise and manage updates on all clients then a decentralized sync solution kinda loses its value.