maxexcloo

Slow Change Detection & File/Folder Deletions Not Syncing

Recommended Posts

Hello,
First of all, let me thank you for BitTorrent Sync, it is an excellent, clean application that (aside from my recent issues) does everything perfectly. Seriously, I've been waiting years for an application like this!
 
My issues are as follows:
  • Recently I've been noticing that btsync seems to be slow at detecting filesystem changes. Adding a file usually requires that the filesystem scan runs through before the UI will reflect the change and start syncing.
  • When deleting files or folders the change is reflected locally but does not propagate across to my other sync client. This means that my files often become messy as I use my desktop only once a week.

Steps taken to reproduce the problem:

  • Creating a new file fairly deep in the directory tree ("D:\Backups\Devices\Desire HD\ClockworkMOD\Download\Google Applications 4.3 (BaNks Face Unlock).zip").
  • Waiting for sync to occur (this took 5 minutes as my disk scan time is set to 300).
  • Once sync has occurred, deleting the new file off the other PC.
  • The file does not get deleted off the original origin PC, in fact, the UI on both PCs don't even register the deletion, even after time has passed or the clients are restarted.

My setup is as follows:

  • 1 Desktop running Windows 8 64 bit.
  • 1 Laptop running Windows 8 64 bit.
  • Both PCs have a virtually identical setup and run the same applications.
  • I have a single sync folder on both PCs at "D:\Sync\BitTorrent Sync" that contains symlinks to all the folders I want to sync. Overall the folder contains ~450 GB of data and maybe 200,000 files.
  • The settings are at default values with the relay server unticked.

Steps I've taken:

  • Tested on LAN and Wireless.
  • Completely uninstalled and reinstalled BitTorrent Sync.
  • Completely removed and readded the sync folder (which took around 14 hours to finish!).
  • Changed directory scan interval and SyncArchive settings, both to no effect (I would like to turn SyncArchive off but it isn't clear if this disables deletions).
  • Ensured that the sync keys are properly entered and not read only ;)

Honestly at this point I'm at a loss, I can live with this if the issue is unsolvable but I would really love it if there was some form of solution.

Thank you so much for your time, Max.

 

EDIT: Just a note: This is a copy of the email I've sent (which included a log file), I posted it here so that others can potentially help and above all, once a solution is (hopefully!) found, I can post it :)

Share this post


Link to post
Share on other sites
My issues are as follows:
  • Recently I've been noticing that btsync seems to be slow at detecting filesystem changes. Adding a file usually requires that the filesystem scan runs through before the UI will reflect the change and start syncing.
  • When deleting files or folders the change is reflected locally but does not propagate across to my other sync client. This means that my files often become messy as I use my desktop only once a week.

 

Not all operating systems report changes in files to Sync in realtime (although Windows does) - but your issue is likely due to your Symlinks (see next paragraph). As a fall back, the advanced "folder_rescan_interval" setting is available to force a rescan of all monitored folders at a recurring interval. By default, this is set to 600 seconds (10 minutes). You can decrease this value to force Sync to rescan your folders for changes more frequently

 

However, in relation to changes not correctly propagating across devices, avoid using SymLinks! Whilst products like Dropbox etc only allow you to Sync one central folder, thus necessitating the need for SymLinks, BitTorrent Sync allows you to sync any folder - making SymLinks somewhat redundant!

 

If you insist on using SymLinks, try creating them the other way around instead!

Share this post


Link to post
Share on other sites

Hi!

I have the same problem since 1.2.73 on three Macs with several folders including a zillion files (no SymLinks included). Up to version 1.2.68 syncing happened almost in an instant. Now it takes 30 min up to several hours to detect changes and sync them to the other machines. (Also to show them in the history takes very long and it even says "Machine X synced at xx:yy" showing a time AFTER an unrecognized change happened that was not shown in the history and only really happens hours later.) Tested it in all directions with small files, also re-added folders to re-index them. Did not help.

My guess: it has to do with the changes you did to indexing (to stop infinite indexing) OR it did not like the "running upgrade" from 1.2.68 to 1.2.73 without a clean re-indexing. Have not changed anything else in my set-up that worked well for many weeks.

 

Thanks!

Share this post


Link to post
Share on other sites

Not all operating systems report changes in files to Sync in realtime (although Windows does) - but your issue is likely due to your Symlinks (see next paragraph). As a fall back, the advanced "folder_rescan_interval" setting is available to force a rescan of all monitored folders at a recurring interval. By default, this is set to 600 seconds (10 minutes). You can decrease this value to force Sync to rescan your folders for changes more frequently

 

However, in relation to changes not correctly propagating across devices, avoid using SymLinks! Whilst products like Dropbox etc only allow you to Sync one central folder, thus necessitating the need for SymLinks, BitTorrent Sync allows you to sync any folder - making SymLinks somewhat redundant!

 

If you insist on using SymLinks, try creating them the other way around instead!

 

I have tested with files outside of the symlinked folders and the issue persists there too unfortunately.

Also on your tip about using multiple folders: While I could do this I believe the symlinks issue is a real one and as I have upwards of 10 folders symlinked it would be quite a task to switch my entire filesystem layout on multiple devices just to satisfy one program. Also, I use symlinks for many other purposes and there isn't really a good replacement for some of my uses.

Share this post


Link to post
Share on other sites

The issues are still occurring on 1.2.82. I have yet to recieve a reply on my email ticket and am really stumped at this issue, are there any good alternatives I can use?

 

Update to my post of Yesterday, 11:41 PM :

Downgraded to 1.2.68 and it immediately synced everything again....

Tried this, the issues still occurred for me unfortunately.

 

Trying 1.1.69 next...

Share this post


Link to post
Share on other sites

I have the same issue. BTsync should update immediately as do all its competitor solutions. Having to change a scan interval is not an option for me.

Honestly the bigger issue for me is the lack of file delection detection and syncing, it means my directory trees quickly become out of sync as old files stay around...

Share this post


Link to post
Share on other sites

Hi!
Update: I am now again on version 1.2.82 and it works fine.
 
Important issues are still:
 
- Renaming often not recognized as such, resulting in new unnecessary uploads
- Aliases not resolved
- the tab "devices" in some folders never shows a complete initial sync but it does sync correctly.
- No new version for 3 months :-(
 
Best!

Share this post


Link to post
Share on other sites

Having the same problem on btsync 1.2.82. I have btsync running on Windows PC, and syncing it to btsync server running on RHEL6 (both running version 1.2.82). It takes about 4 - 10 minutes for my files to be synchronized.

 

My btsync client on RHEL6 is behind a firewall (except the port which is starts listening on), so I have to use those relay / tracker servers (not exactly sure how these things works, but it's enabled on both Windows and RHEL6 btsync clients).

 

Is there any chance that these issues could be fixed soon?

Share this post


Link to post
Share on other sites

I'm going to have to chime in on this one as well, as I'm seeing the same thing as maxexcloo.  Today, I've deleted the same files I've previoiusly deleted three other times because instead of deleting them from the other machine, BTSync seems h****-bent on recreating them on the machine I've just deleted them off of.  I am sync'ing between a Linux and Windows 7 machine.  The files were orginally downloaded into the Linux folder, sync'ed acrossed to the Windows machine, but then either moved or deleted from the Linux folder.  Even the moved ones reappear, which means I now have two copies of these files in both locations.  I am running 1.2.82 on both machines as well.

 

I guess I could stand the delays as long as it did the right thing, but alas it doesn't even do that!

Share this post


Link to post
Share on other sites

Me too.   I have 3 Linux machines with one shared folder.  The master folder is on machine "A", while "B" and "C" are set up with the read-only secret from "A".  So I expect any changes made on "A" to replicate to "B" and "C".  All the machines are on the same LAN, and tracker use is turned off.

 

I have the control web pages up from all three machines.  "B" and "C" quickly found each other but did not exchange anything.  This is expected since they both only have the read-only secret.  But "A" never found "B" or "C" and nothing is being transferred.

 

Exactly where is this "folder_rescan_interval" specified?   But it has been well over 10 minutes anyway so I doubt if that is the problem.

 

Version 1.2.82-1.  All three machines are running Arch Linux, though "A" is x86_64, "B" is x86, and "C" is ARM.

 

UPDATE: Nodes "B" and "C" did finally sync, but it was after an hour had elapsed.

Edited by TwoNotes

Share this post


Link to post
Share on other sites

HI,

I am also having this problem, I am using a Mac Pro as the central server that holds everything.
it is also backing up using time machine, and crashplan..

3 Macbooks sync to the Mac Pro.

I noticed that when I sync from the Mac Pro to the Macbooks, it takes hours, sometimes days. I've tried to remove the sync folders, then re-adding the folders, etc and for most part it helps, but seems to still not be any better. I have about 2TB of data that BT is monitoring.

But when I sync from the Macbooks to the Mac Pro, it takes minutes.
same goes with Macbooks to Macbooks. It seems to be better.
The MacBooks cpu's are not really any newer or faster.


I'm thinking maybe my Mac Pro is too overloaded (2008) syncing and backing up other processes.

Before I spend the money for upgrade to Mac Pro, suggestings? will this help me?

All macs running the latest version of OS X, also the Macpro is runing as a server too , but only really using VPN and services for more part.
 

Share this post


Link to post
Share on other sites

@mikelee,

What kind of data do you sync? Heavy files or big amount of tiny ones? When you sync Mac Pro -> Macbook, do you just see slow connection speed or some other symptoms (like, saw-shaped traffic)? Which version of BTSync do you use?

Share this post


Link to post
Share on other sites

Hi,

I am syncing two home folders except the library folder. Idea is that I want to be able to pick up any Mac and have it all with me.

The largest files are pictures and movies. iphoto being most active. Movies are not that aften updated.

I noticed that if I reboot the computer it helps. But seems after awhile it gets 'hosted' up again.

I decided to get a faster computer to see if that helps.

I don't think its a slow connection because when it does work it works fine. It seems like its slow in detecting the change. Not slow in transferring.

I am using the latest beta release

Share this post


Link to post
Share on other sites

@mikelee,

 

Try one small test before upgrading your hardware: change the advanced setting "disk_low_priority" to false and see if it helps.

Share this post


Link to post
Share on other sites

hi,

 

ok, I've upgraded to a much faster computer and I think it's still not helping...

 

I added a file to the Mac pro, and within 5 mins later two MacBooks, show that "Mac Pro added file....", but it's been 17 mins now and still no file transferred.

 

Any clues?

 

Thanks.

Share this post


Link to post
Share on other sites

Same with me on 1.3.94 with 1 iMac, 2 MacBooks Pro (all Mavericks) and one old MiniMac. Initial syncing of 1 TB (divided into about 10 different sync-Folders) takes many days although all folders are identical and nothing needs to be copied. After a while all folders are shown as indexed completely but the arrow-symbols indicate that the larger folders are not complete (like 62 GB left to upload and 12 GB left to download). Those values almost never change or only a few MBs per day. Nevertheless smaller sync-Folders are synced right away, the bigger ones are synced, too. But sometimes half an hour later or even a day later. Don't see a pattern. 

 

As of version 1.3 there seems to be another slowing down issue with extended attributes (which are very welcome!) There are some old files (15 years old) and some newer ones (usually from outside customers, maybe of windows origin) where the attributes cannot be synced. (The files exist on all machines already!) BTS tries to sync them frantically again and again, heating up all the macs and making the fan spin like crazy for hours. CPU is on 100% and more. BTS then stops syncing other files (like ones I am working on right now), gets stuck (leaving many !sync-files) and needs to be restarted. After restarts it quickly resyncs my new files and falls back into the old routine of heat, fan, CPU etc... after a few hours.

 

In the last versions before 1.3 this was not a problem any more after about 2 days of initial indexing.....

Share this post


Link to post
Share on other sites

Yes, me too since looking at the logs, I am getting a LOT of "Failed to write attributes for file...."

 

I have made sure that I have permissions to the files.

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.