KawateTadako

Frequent and long indexing stressing HDD & CPU

Recommended Posts

I use BTSYNC Pro on a windows PC, but the shared 70GB directory is on a NAS. Because of that I can't use window's OS notification system for indexing file changes. The work around is to set scanning intervals using  rescan interval in Sync settings -> Advanced -> Power user. I've set it to 36000 which is fine for me. This has been working fine on version 2.2.5

Recently I've updated to the latest 2.3.3 and my NAS was giving heat warnings. Finally, found out BYSYNC was indexing very frequently and for prolong periods of time causing my HDD to spinup. 

Temporarily I've downgraded back to 2.2.5 and indexing is back to normal, but I still want to keep updated.

Is there a workaround fix or am I doing something wrong? 

 

cpu_chart.png

Share this post


Link to post
Share on other sites
Quote

Recently I've updated to the latest 2.3.3 and my NAS was giving heat warnings. Finally, found out BYSYNC was indexing very frequently and for prolong periods of time causing my HDD to spinup. 

I'm having trouble with BtSync indexing frequently consuming more  than 70% of CPU after upgrading to 2.3.3 (296). I'm running on Mac OS X 10.11.3.

Can I provide more information to help you track this issue? Is this already on your radar? Or it's a better idea to downgrade until a proper fix?

Share this post


Link to post
Share on other sites

So I contacted support and their suggestion is to set  rescan interval to "0" (the default is 600) 

This stops the constant indexing with 2.3.3 ,but also mean new files won't be indexed and synced.  

Share this post


Link to post
Share on other sites

That is funny. If that is their proposed solution, I have a better one: quit the application. Why keep it running if it will not index and sync files?

For now I'm trying to use older versions, looking for a more stable one. But I think it's time for another round of searches for BtSync alternatives.

Share this post


Link to post
Share on other sites

My BTSync is on a windows PC that has access to a shared folder on a NAS. If someone else on the network make changes to that shared folder, how will BTSync know to update if it doesn't scan and index? 

 

Share this post


Link to post
Share on other sites

@KawateTadako If you change files over SMB Sync still can receive notification from your NAS OS, though notification subsystem is less reliable than rescan. Although, if your files are updated via SMB it is in general not very good to change them with some other tools like Sync or anything else. Please see this article about SMB.

Share this post


Link to post
Share on other sites

I'm using Freenas and it doesn't support file change notification. I need to use rescan interval; I am very aware of this. I've been doing it with Version 2.2.5 for a while now fine. Indexing works and minimal load on the freenas.

The problem comes when upgrading to 2.3.3. Indexing takes a very long time about half a day as indicated by the graph. Downgrading back to 2.2.5 everything is normal again. 

Share this post


Link to post
Share on other sites

I am looking at the 500MB+ of sync.log.xxx.zip (that is a lot of compressed logs )

Appears BTSYNC 2.3.3 is indexing my files over and over again for hours and hours, almost all day. Something is wrong.

Uninstalled 2.3.3, wiped userdata and installed 2.2.7. Added my 70GB folder to index and it finished in 8 minutes. 

 

Share this post


Link to post
Share on other sites

Can anyone point to a safe link to download 2.2.7? BitTorrent Sync official site has AFAIK only 2.3.3 and - frankly - should remove it soon or at least leave older versions available. I thought this problem was due to recent upgrade to El Capitan but I see is a bug everywhere.

Share this post


Link to post
Share on other sites
2 hours ago, peogbert said:

Can anyone point to a safe link to download 2.2.7? BitTorrent Sync official site has AFAIK only 2.3.3 and - frankly - should remove it soon or at least leave older versions available. I thought this problem was due to recent upgrade to El Capitan but I see is a bug everywhere.

forum.getsync.com/topic/41708-latest-desktop-build-227

Share this post


Link to post
Share on other sites

I have had nothing but problems with the 2.3.x release branch. Every version has had the "indexing forever" bug, with neverending high CPU usage. If I forget to quit BitTorrent Sync when I'm on battery it's drained in no time. The changes made from 2.2 to 2.3 introduced some critical bugs to core functionality, and it makes me second guess my purchase of a pro license.

Share this post


Link to post
Share on other sites

Same Problem here. Since I upgraded, btsync is scanning on my Disc around the clock. And "Setting rescan interval to 0" is not a satisfying answer to me, because my NAS is also an active peer on downloading ISOs and other stuff. Feeling a bit like a beta tester...

mapper_truecrypt2-week.png

Share this post


Link to post
Share on other sites
On 12 March 2016 at 6:03 PM, KawateTadako said:

forum.getsync.com/topic/41708-latest-desktop-build-227

Thank you KawateTadako!

And:

56 minutes ago, generator said:

Some problem, has downgrading to 2.2.7 worked for other people and is it safe to downgrade?

After downgrading to 2.2.7 it indexed in few minutes all what it has tried to index for hours (if not days) and it is syncing as a charm.... (OSX El Capitan 10.11.3 here)

Share this post


Link to post
Share on other sites

@KawateTadako @MaBu81 We've recently found one more possible root cause.

Starting from 2.3 we've fixed notifications over SMB (if SMB server and client support SMB 3.0, which seems to be your case, as logs contains number of notifications pending). All notified changes are pending the rescan and prolong the indexing. Could you please try disabling the "enable_file_system_notifications" power option on your NAS and see if your indexing will take the same amount of time as in 2.2.7?

@repertor @gabled-bravado-dell @generator We need logs to see what happens on your machines. In general, indexing in 2.3 got kind of lower priority than other tasks performed by Sync, therefore it may last slightly longer, but it should not take several times longer.

Share this post


Link to post
Share on other sites

@RomanZ my FreeNAS-9.3-STABLE is running SMB2, SMB 3.0 isn't supported. I upgraded to 2.3.3 again, set  "enable_file_system_notifications  to false. Indexing still taking ongoing after an half hour. On 2.2.7 it only takes minutes to complete.

Looking at sync.log I see OnNotifyFileChange, Journal: assign file_id, will scan folder, entry, erase file_id lines repeated for the same files over and over again. Sync.log overflows, get archived as *.zip until I have GB's of log stored.

Share this post


Link to post
Share on other sites

@KawateTadako Investigation of your log has shown that file notifications could be the reason of indexing slowdown. I wonder if you could send us fresh logs after you disabled notificaitons? Please also mention this forum topic so support engineer who takes care of you will be aware of it. Thank you!

Share this post


Link to post
Share on other sites
On 3/16/2016 at 0:27 AM, RomanZ said:

@KawateTadako Investigation of your log has shown that file notifications could be the reason of indexing slowdown. I wonder if you could send us fresh logs after you disabled notificaitons? Please also mention this forum topic so support engineer who takes care of you will be aware of it. Thank you!

Updated to 2.3.5, still having the same issue. Left it indexing for 40 minutes before giving up and reverted back to 2.2.7 which index completed in minutes. 

uploaded new logs to ticket request #40669

Share this post


Link to post
Share on other sites
On 15.3.2016 at 2:26 PM, RomanZ said:

@KawateTadako @MaBu81 We've recently found one more possible root cause.

Starting from 2.3 we've fixed notifications over SMB (if SMB server and client support SMB 3.0, which seems to be your case, as logs contains number of notifications pending). All notified changes are pending the rescan and prolong the indexing. Could you please try disabling the "enable_file_system_notifications" power option on your NAS and see if your indexing will take the same amount of time as in 2.2.7?

*SNIP*

Seems so be the reason for me. Removing the only Samba directory I use in common with btsync stopped the endless indexing. Trying out to set the "enable_file_system_notifications" when Im home again.

Thank you so far.

Share this post


Link to post
Share on other sites

@KawateTadako Thanks for the fresh logs. There is something very special about your filesystem. It looks like the ID of every folder changes rather often, and new Sync (starting from 2.3) is actively using it. We'll do our best to find our why it happens and address it.

Share this post


Link to post
Share on other sites

I seem to have a similar problem on some of my computers after updating from 2.2.7 to 2.3.5. The indexing on 2.3.5 (365) seems quiet frequent (on most of my devices) and taking long compared to 2.2.7. It's not stopping to index at all on my "Server".

___

- On my Workstation (Win 8.1, i7-2600k, 24 GB RAM) with a subset of all my folders (19 folders, ~360 GB), everything seems to work like before. CPU (0-0,2% usage when not indexing and closed UI) and RAM usage (~190 MB) of BTsync is normal, indexing is finished quite fast stressing one logical CPU core (12,5% CPU usage). 

- On my Notebook (Win 8.1, Thinkpad Yoga S1, i5-4200u, 8 GB RAM) with most of my folders (22 folders, ~600 GB), it seems BTsync is constantly doing something. CPU (almost constantly around 0,2-2% usage when not indexing and closed UI), RAM usage (~250 MB) indexing takes quite long and stressing again one logical CPU core (25% CPU usage). This behavior is quite annoying, since it's draining my battery quite fast!

- On my "Server" (Win7, Core2Quad Q6600, 3 GB RAM) with all of the folders (26 folders, ~890 GB) BTsync is constantly indexing the same four folders that are syncing with my sisters MacBook Pro (I have to check which version of BTsync is running on that device) even though her Notebook is offline at the moment. CPU usage (constantly 12-15% usage) and RAM usage (~830 MB) is also quite high, but no wonder if it's constantly indexing. Is there anything I can do to stop the constant indexing of those four folders?

- On my Android Phone (Samsung Galaxy S4 i9505, 5.0.1) everything is finally working fine since the last update to 2.3.5 (480), since before that it was constantly indexing there. Even syncing to the 128 GB SD Card! 

- On my sisters MacBook Pro (still have to check the BTsync version there), with a small subset of the folders (4 folders, ~260 GB), I have no idea if BTsync is running properly. 

___

I guess I should have stayed with the old version (2.2.7) as well. But before I'm going to revert to that one, I'd like to know if there is some other solution that I could try. Should I send logs? If yes, from which devices? How big should my log file size be with 890 GB of files?

Share this post


Link to post
Share on other sites

I've updated my workstation, server and notebook to 2.3.6, but I still seem to have the same problem. It's quite annoying that the server is constantly indexing the folders. I assume it's due to the folder count and size compared to the other machines, which just have a subset of those. The CPU and HDD of the server are never idle due to the indexing. Even more annoying is the reduced battery life on my notebook due to the frequent indexing of Sync. 

Feel like rolling back to 2.2.7, but before that I'll send you the log files of my server, maybe that will help solving the issues. 

I guess I'll just paste the link to this thread in the help request?

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.