Bitwise

Bug Report: Bittorrent Sync Syncing Old Files Over New Ones

Recommended Posts

I just performed a sync using BitTorrent Sync v1.4.110.

 

I am 100% certain BitTorrent Sync overwrote newer files with older files.

 

I don't have any idea why it would do this, except for a serious bug.  The file dates and file contents clearly show that new files were overwitten by old versions.

 

I had performed a backup of some of the data before enabling the sync, and the backup confirms the problem.

 

A little more detail: I had previously used BitTorrent Sync on these folders using 1.4.9x.  Only 2 computers are using BTSync and they run the Windows 64-bit version. They are on the same wireless network.  Relay and tracker servers are turned off for each sync folder.  I hadn't run BTSync in over a month.  I upgraded BTSync to v1.4.110 on both computers.  As part of the BTSync upgrade on System #1 (which had many new files), I selected the option to run BTSync after the program version upgrade.  Once it was upgraded and loaded, I ran BTSync on System #2 (which I had already upgraded to the latest version).  It than began to sync.  Everything looked fine and appeared to run smoothly.  But when I checked the files, in many cases, old versions overwote the new versions.  I now have to sift through thousands of files to determine the damage BTSync caused and try to rectify as much of the damage as possible.

 

UPDATE: I just had a full-time computer programmer verify all the above facts, and he concluded there is a serious problem with BitTorrent Sync, and that it definitely overwrote NEWER files with OLDER ones in multiple cases.

 

This seems to be a huge problem.

Share this post


Link to post
Share on other sites

I can confirm this problem.

My setup: PC1 has newest file is read+write, PC2 is a linux server to act as cloud and readonly, PC3 has old file and is read+write.

 

PC1 syncs to PC2 with the new file, PC3 is offline.

PC3 goes online and syncs with PC2, now uploading the old file and overwriting the new file, PC1 is offline.

PC1 goes online and syncs with PC2, now receiving the old file, PC3 is offline.

The new file is now removed unless you have backup of it.

 

The reason for PC1 and PC3 to both have write is because my laptop and desktop should both be able to add/update/delete files, but the newest file should ALWAYS overwrite the oldest one.

 

Using 1.4.110 Beta on Windows 8.1,10 and Linux.

Edited by Digto

Share this post


Link to post
Share on other sites

I am now running version 2.0.93 running on Windows 8.1, Windows 10 and Ubuntu.

 

And this is still a problem, I fear that having more than one pc able to write in the folders is dangerous, so I'm going to try having only 1 pc that can write, and make the rest read only.

Share this post


Link to post
Share on other sites

Same problem. 

I have been using btsync since December (documents and music) and lately a to sync folder with source code.

Big mistake.

Btsync has never really seemed to sync properly, and stops syncing frequently. 

Today, I needed to look at some source code on a fedora box and noticed that it hadn't synced for more than two weeks. So I restarted btsync, without it helping. BTsync still claimed everything was in sync, when it clearly wasn't. I disconnected and reconnected the share leaving the files in place. Big mistake. Every old file was copied from the stale folder to the remote fresh one overwriting two weeks of work. Luckily most of it was commited to a git repo, but still loads of work in progress was lost.

 

BTW, you keep referring to version numbers, but neither the download nor the output when starting it mentions it. Anyway, I'm sure it is the latest, since the web UI didn't report an update was available.

Also, do you really expect me to "turn on debugging" and recreate the problem? I mad enough as it is.

Share this post


Link to post
Share on other sites

Same problem. 

I have been using btsync since December (documents and music) and lately a to sync folder with source code.

Big mistake.

Btsync has never really seemed to sync properly, and stops syncing frequently. 

Today, I needed to look at some source code on a fedora box and noticed that it hadn't synced for more than two weeks. So I restarted btsync, without it helping. BTsync still claimed everything was in sync, when it clearly wasn't. I disconnected and reconnected the share leaving the files in place. Big mistake. Every old file was copied from the stale folder to the remote fresh one overwriting two weeks of work. Luckily most of it was committed to a git repo, but still loads of work in progress was lost.

 

BTW, you keep referring to version numbers, but neither the download nor the output when starting it mentions it. Anyway, I'm sure it is the latest, since the web UI didn't report an update was available.

Also, do you really expect me to "turn on debugging" and recreate the problem? I mad enough as it is.

Update. I was able to restore everything overwritten from the .sync/Archive folder, so there is a good thing.

What I didn't mention i my initial bug report was that these were old 1.4 shares. I disconnected all clients, and recreated the shares as 2.0(?) shares, lets see how that works out. 

Also, the setup is 3 peers, Opensuse 13.1 and 13.2 on a LAN, and a Fedora 21 peer outside that (firewall/NAT). The fedora 21 peer was the one falling out of sync all the time.

Share this post


Link to post
Share on other sites

Can you check if all of your machines have synchronized system clocks?

It happened to me a couple of times that new versions were replaced with old files. I found that at least one reason for that is desynchronized clock on one of the machines. It's easy and always reproducible. Since then I enabled automatic clock syncing from NTP on all of my systems.

It's a pain that BTSync relies on system clock so heavily instead of doing some sort of own clock synchronization. Each instance could synchronize its own clock with the rest of BTSyncs, then calculate the difference between its clock and system clock and use such "corrected" time to see what files are really the new ones.

Share this post


Link to post
Share on other sites

Can you check if all of your machines have synchronized system clocks?

It happened to me a couple of times that new versions were replaced with old files. I found that at least one reason for that is desynchronized clock on one of the machines. It's easy and always reproducible. Since then I enabled automatic clock syncing from NTP on all of my systems.

It's a pain that BTSync relies on system clock so heavily instead of doing some sort of own clock synchronization. Each instance could synchronize its own clock with the rest of BTSyncs, then calculate the difference between its clock and system clock and use such "corrected" time to see what files are really the new ones.

 

All machines are running NTP. It would have to be a serious clock drift problem if BTSync suddenly thinks that two week old files suddenly are never than files timestamped minutes ago. Just to make sure, I rechecked all my clocks:

 

ssh host1 'date +"%F %T.%N"'; ssh host2 'date +"%F %T.%N"'; ssh host3 'date +"%F %T.%N"'

2015-04-30 13:15:37.345802964
2015-04-30 13:15:37.486356324
2015-04-30 13:15:37.489242791
 
Edited by Stardust

Share this post


Link to post
Share on other sites

Any news on this?

I've just discovered this massive bug and it's easily reproducable.

It's a catastrophic scenario.

New peers added with older versions of files simply overwrite all newer instances of those files on all peers!

Wow.

It's got nothing to do with clock syncing.

Our tests have shown 10 year, 1 year, 1 month, 1 day, 3 hour old files overwriting files that are less than 10 min old.

There's no hope of using BT sync in any kind of professional environment with this sort of issue.

 

Can we get a formal response about this issue from support?

Share this post


Link to post
Share on other sites

I've just discovered this massive bug and it's easily reproducable.

Firstly, are you using the most recent version of Sync (which at time of writing is 2.2.5)? If not, please update to 2.2.5 to see if this solves your issue (for instance, 2.2.0 contained a fix for "Newer file replaced by old one from offline peer")

If updating all your peers to 2.2.5 does not resolve your issue, please share your "easily reproducible" steps

Share this post


Link to post
Share on other sites

@great marko: The windows peers are running 2.2.5, but is there a simple method for updating the version running on the synlogy NAS?  I haven't been able to work out how to update that "package" so far from 2.0.93.

@romanz:  standard folders AFAIK

Share this post


Link to post
Share on other sites

@Brad@IO

Could you please check it? It's important as issue you describe was fixed for Standard folders in 2.2.5 (while still exists for Advanced ones - its a bit more complex to fix it for advanced folders).

 

Just open your folder share dialog, if it shows you "key" tab - it is standard one, otherwise it is Advanced.

Share this post


Link to post
Share on other sites

@romanZ: Interesting.  We've always picked STANDARD when we add a share, but it seems they've all been set up as ADVANCED folders.  Maybe because the share window next presents a permissions option before you share the link.  Am I right in thinking that if we see this permission option then we're setting up an ADVANCED folder?

 

@great marko: found the manual instructions for updating BTsync on the Synology NAS.  Done. 

 

Happy to test again, but will that only be useful if the folers are converted-to or re-shared as STANDARD?

Share this post


Link to post
Share on other sites

I'm facing the same problem. I had for several weeks only two computers synced together (Win10 and OSX). No trouble. Then I added a Synology NAS to the peers (running latest 2.2.5-1 version). That was the beginning of the troubles. By looking at the logs and file creation times, it seems that the creation time is wrong on Synology. It's set to the date the file was copied to. So when my computer changes a file, keeping its original creation time, the NAS thinks its version is newer and overwrite the changes. Why is file creation time used to check which file is newer? Most of the time, creation date is not changed by OSes. Is this problem limited to Synology?

Share this post


Link to post
Share on other sites

@jacoch

Sync doesn't use the file creation time, but it is using file modification time instead. Also, when Sync delivers file, it adjusts its mtime so it will be the same as on other peers.

 

I suspect that your Syno does not allow to alter mtime (technically, any FS on Linux can be mounted in the way it does not allow to change mtime).

 

There is a simple test you can do to check it. SSH to your NAS as admin, go to some folder (which stays on the same FS as your synced folder) and try to "touch" files. See if it updates the mtime of a file.

Share this post


Link to post
Share on other sites

Modifications times are fine. That's why I thought about the creation time. It's the only difference. So I looked at the logs more closely and noticed an error when setting attributes. That could be the cause of the troubles. Here are some interesting entries from my logs (it's a file uploaded on January 13th reverting to a version from January 6th). Maybe you can explain more clearly why the old file finishes to be sent to other peers.

 

[20160113 14:56:25.726] MC[C17F] [7E7A]: will request nodes for /My/Path/To/File.pdf
[20160113 14:56:27.187] MC[C17F] [7AF7]: will request nodes for /My/Path/To/File.pdf
[20160113 14:56:27.308] MC[C17F] [7AF7]: processing nodes message for /My/Path/To/File.pdf
[20160113 14:56:27.308] MC[C17F] [7AF7]: will request files for /My/Path/To/File.pdf
[20160113 14:56:27.429] MC[C17F] [7AF7]: Local file My/Path/To/File.pdf is older than remote t:1452691269/1452693379 ot:28110/2210841 o:10CA626E2DA8A6F6AF702A6EE757641ABC7FEE40/10C0FE7D53478F7664D731BDBAEE10705B527E7A
            Wed, 13 Jan 2016 13:21:09 GMT / Wed, 13 Jan 2016 13:56:19 GMT
[20160113 14:56:27.429] JOURNAL[C17F]: got file from remote: "My/Path/To/File.pdf" state: 1 type: file total:17 have:17 size:540230 t:1452693379 mt:1452693379 ot:2210841 o:10C0FE7D53478F7664D731BDBAEE10705B527E7A h:813725282900FCC03403E890737182DAB70E0A2D
            Wed, 13 Jan 2016 13:56:19 GMT
[20160113 14:56:27.453] Update have pieces for file "/volume1/Plurial/Internet 08/My/Path/To/File.pdf", was: 17, now: 17
[20160113 14:56:27.453] SyncFileEntry: writing file attributes to file "/volume1/Plurial/Internet 08/My/Path/To/File.pdf", mt:1452693379
            Wed, 13 Jan 2016 13:56:19 GMT
[20160113 14:56:27.453] SyncFileEntry: Failed to write attributes for file /volume1/Plurial/Internet 08/My/Path/To/File.pdf - 1
[20160113 14:57:33.850] Update have pieces for file "/volume1/Plurial/Internet 08/My/Path/To/File.pdf", was: 17, now: 0
[20160113 14:57:33.875] Update have pieces for file "/volume1/Plurial/Internet 08/My/Path/To/File.pdf", was: 0, now: 17
[20160113 14:57:33.875] JOURNAL[C17F]: entry "/volume1/Plurial/Internet 08/My/Path/To/File.pdf" state: size:540230 mt:4294967295 t:1452693453 h:813725282900FCC03403E890737182DAB70E0A2D (new torrent for file)
            Wed, 13 Jan 2016 13:57:33 GMT
[20160113 14:57:33.876] JOURNAL[C17F]: setting time for file "/volume1/Plurial/Internet 08/My/Path/To/File.pdf" to 1452693453
            Wed, 13 Jan 2016 13:57:33 GMT
[20160113 14:57:33.876] SyncFileEntry: Set owner time for entry "/volume1/Plurial/Internet 08/My/Path/To/File.pdf" to 28266
[20160113 14:57:33.877] JOURNAL[C17F]: entry "/volume1/Plurial/Internet 08/My/Path/To/File.pdf" state: size:540230 mt:1452084885 t:1452693453 h:813725282900FCC03403E890737182DAB70E0A2D (file updated)
            Wed, 06 Jan 2016 12:54:45 GMT / Wed, 13 Jan 2016 13:57:33 GMT
[20160113 14:57:41.888] MC[C17F] [7AF7]: will request nodes for /My/Path/To/File.pdf
[20160113 14:57:42.008] MC[C17F] [7AF7]: processing nodes message for /My/Path/To/File.pdf
[20160113 14:57:42.008] MC[C17F] [7AF7]: will request files for /My/Path/To/File.pdf
[20160113 14:57:42.097] MC[C17F] [7E7A]: will request nodes for /My/Path/To/File.pdf
[20160113 14:57:42.130] MC[C17F] [7AF7]: Local file My/Path/To/File.pdf is newer than remote t:1452693453/1452693379 ot:28266/2210841 o:10CA626E2DA8A6F6AF702A6EE757641ABC7FEE40/10C0FE7D53478F7664D731BDBAEE10705B527E7A
            Wed, 13 Jan 2016 13:57:33 GMT / Wed, 13 Jan 2016 13:56:19 GMT
[20160113 14:57:42.130] MC[C17F] [7AF7]: will send file /My/Path/To/File.pdf
 

Edited by jacoch

Share this post


Link to post
Share on other sites

@jacoch

Still it looks like the file system mounted does not allow to set modification time. Although, it usually happens when one mounts FS manually. As I understand, you did not perform any manual operations on your FS, it was partitioned and mounted by Syno DSM, right? If yes - could you please let me know which DSM version you use and which FS did you choose?

 

Thanks.

Share this post


Link to post
Share on other sites

Well, I would not say that it doesn't allow to set file modification time. It works properly on the shares. But maybe BTSync client doesn't support my configuration yet. I'm using DSM 6 beta and btrfs as FS.

Share this post


Link to post
Share on other sites

Hi Guys,

bumping this old thread to see if there's new info.

When I had problem with older files overwriting newer ones back in Oct 2015, RomanZ, confirmed that the issue

<snip>...was fixed for Standard folders in 2.2.5 (while still exists for Advanced ones - its a bit more complex to fix it for advanced folders).

As a result we've never used Advanced folders, but we would like to be able to take advantage of them.

I've scanned the change logs several times since, but been unable to find any mention of a specific fix.

Is this issue fixed for Advanced folders now? 

cheers

Share this post


Link to post
Share on other sites

That response doesn't clarify where we are at regarding older files overwriting newer files in Advanced folders.

There's an increasing number of companies we deal with where discussion of Resilio sync comes up as an option and we always mention the problem with older files overwriting newer ones.  It would be great to be able to know that issue has been 100% fixed, with no caveats.

Are you saying:

  1. the issue with standard folders discussed in the thread is fixed but the specific known issue with Advanced Folders referred to by RomanZ still exists
    or
  2. the specific known issue with Advanced Folders referred to by RomanZ is fixed but overwriting of new files with old still occurs with Advanced Folders due to other known issues
  3. there are NO known, reported or outstanding issues regarding overwriting of new files with old in Advanced folders.

Is the answer 1, 2 or 3?

Or, if it's easier:

As previously stated, we just want to know:

Can we use Advanced Folders without the risk of older files overwriting newer files. 

Is the answer Yes or No?

I can start a new thread if it will lead to a clear YES or NO answer.

Share this post


Link to post
Share on other sites

it's 3: 
the issue reported and discussed here, referred to by Roman is fixed. 

There are a lot of other use cases when older file may get synced - for example, it's being edited at the same time on different computers by different people. It's out of Sync's control and whoever saves and closes the file last, uploads its copy to others, even though those others may have their "latest" copy saved some 10 seconds before.  
and alike. These cases do not depend on Standard/Advanced

So I cannot tell you for sure "Yes or No", I don't know all your use cases and I cannot predict them. 
When it comes to this particular issue reported, discussed and referred to here by Roman- Yes, you can.  

Share this post


Link to post
Share on other sites

I can confirm that this STILL happens as of version 2.5.12 (1191)

I had a previously synced machine offline for a month. Fired it up, once it completed the sync. All of the old outdated files overwrote the newer files on the  machine that's always online. 

This event has convinced me to uninstall this unreliable sync service.

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.