mr.canada Posted August 11, 2016 Report Share Posted August 11, 2016 I'm attempting to setup a sync between two mac servers. The folder i'm trying to sync is very large - 2.3 TB (yes, TB, not GB). It contains 215,000 items. The two macs are running current OS (today it's 10.11.6), and current Mac Server. They each have full 1gbps fiber internet. Honestly, is this just too much for Sync to deal with? It's been "indexing" for 3 days. If i allow the machines to communicate, then the sync does start to work... i've left it for 24 hours and several gigs has synced, but it's also Indexing at the same time. I imagine these processes compete for the CPU, so i've paused syncing in hopes of Indexing being able to complete. I wish there was a progress bar, % compete, ETA for the Indexing process as there is for the Sync process..... Quote Link to comment Share on other sites More sharing options...
Helen Posted August 12, 2016 Report Share Posted August 12, 2016 Open Size column in Sync UI and see if the initial indexing has gone through - how big the Size is*. It's very likely that you just see periodic rescan in UI. Increase it in power user settings , option "folder_rescan_interval" . it's in seconds. Put there a few hours, or a day (in seconds!) . *To compare the size: better not check folder size in File manager, but do that in console command 'du -hs" in the folder. Compare that to what Sync has to see if Sync was able to index everything. Quote Link to comment Share on other sites More sharing options...
mr.canada Posted August 24, 2016 Author Report Share Posted August 24, 2016 Thank you Helen - this was helpful. Revealing the Size column in the UI allows me to see the difference between what is already sync'ed, and what needs to be sync'ed. It does not reveal the progress of the indexing process, but i suppose that's a chicken-egg dilemma - no way for Sync to know how much data is has not yet indexed until it has indexed it. I increased the index interval to 1 day. It no longer indexes endlessly. But i don't really understand what this process does.... now that indexing only happens once a day, how does Sync know what files to transmit during the day? does it rely on filesystem change notification from the OS to be informed each time a file is added or modified? If so, why index every day - why not just do it once and be done with it? indexing 2.3TB of data takes a long time - maybe an hour or so. why repeat this process daily? Quote Link to comment Share on other sites More sharing options...
mr.canada Posted August 25, 2016 Author Report Share Posted August 25, 2016 I've found some more detailed explanations here... https://help.getsync.com/hc/en-us/articles/205458185-Setting-how-often-Sync-should-check-for-file-changes- This article points out that OS filesystem notifications are not always perfect... ------- Different operating systems have limitations for folder-monitoring applications. For example, if a file path is located very deep in a Windows file structure, it will not send change notifications for these deeper levels. To cover such cases, Sync periodically re-scans all folders for changes. ------- Can you please tell me how reliable Mac OSX 10.11.6 is in this case? I have a LOT of files, and VERY deep folder paths. I would love to be able to increase the rescan interval to a very long time - perhaps 1 week. But I also need Sync to keep up with new files in near realtime. Quote Link to comment Share on other sites More sharing options...
Helen Posted August 25, 2016 Report Share Posted August 25, 2016 Yes, exactly. Sync has two ways to know about file update - system notifications or folder rescan. While notifications may fail (e.g. they do not work on some file system, on network shares, etc), Sync will learn about the files itself when doing rescan job periodically Sync starts scanning the folder, takes some time doing it (the bigger the folder is, the more nested folders and files are there), the longer it takes, stops scanning. Then waits the rescan_period, and starts new scan. If you increase folder_rescan_interval it will put off the new scan. Folder rescan also starts when Sync is launched. Unfortunately, I cannot tell how the notifications will work in your particular case, there is no strict dependancy. On Macs they are more or less reliable. I'd rather just keep an eye on it, and if syncing doesn't start within a reasonable wait-time, decrease rescan. Quote Link to comment Share on other sites More sharing options...
coewar Posted August 31, 2016 Report Share Posted August 31, 2016 OK.. so I see there are some challenges knowing if things changed.. however in my case the folder is a READ-ONLY one. So that means that Sync has no need to scan anything at all. It should just act when it receives updates from other instances. Surely that change can be worked on separately. Quote Link to comment Share on other sites More sharing options...
Helen Posted September 1, 2016 Report Share Posted September 1, 2016 @coewar, Sync must know if files change regardless of share being RO or RW. Folder rescan is done on every share in Sync. Quote Link to comment Share on other sites More sharing options...
coewar Posted September 5, 2016 Report Share Posted September 5, 2016 @Helen, Why would sync need to scan anything if the share is RO, *and* the option to overwrite files is on? Quote Link to comment Share on other sites More sharing options...
Helen Posted September 6, 2016 Report Share Posted September 6, 2016 @coewar, Sync needs to know if files changed in every share, even RO, even with Overwrite enabled. Otherwise, If it doesn't get a notification, and doesn't rescan the RO folder, how would it know that it needs to re-download the file from RW peer? Quote Link to comment Share on other sites More sharing options...
coewar Posted September 6, 2016 Report Share Posted September 6, 2016 Well, it would know simply because it receives notification from the remote instance of Sync on another computer. If you are syncing with the RO option (and especially with Overwrite enabled) then the only thing Sync needs to watch for is changes reported by the source which would be the other computer. When that other computer's Sync tells this Sync that something change, that's when it writes and downloads. So when it receives notification from the remote Sync that files X, Y, and Z have had changes, then it can locally scan just those files, without having to routinely scan them. Hence, it would not need to re-scan the entire directory, and it would only need to scan anything when it gets updates from another Sync instance. Am I missing something still? Quote Link to comment Share on other sites More sharing options...
Helen Posted September 7, 2016 Report Share Posted September 7, 2016 And what if nothing changes on RW peer, but does change on RO? Sync will never know that and on RO peer you will have a mismatching different file? And you don't know that, and when it's time to restore from the bakcup RO folder, you will have there not what you expect. So, Sync does and need to rescan the shares on all peers. All shares it has. If you don't want Sync to rescan them, either increase rescan interval to a day, or disable it at all - put 0 (zero) to folder_rescan_interval parameter. Quote Link to comment Share on other sites More sharing options...
coewar Posted September 7, 2016 Report Share Posted September 7, 2016 I'm not trying to argue but trying to understand better. As far as my situation, increasing the time is good enough for now. There still seems to be some gap I'm not making here though. If nothing changes on the RW peer, but does change on the RO peer, currently what would happen? Does the change done on the RO peer get essentially "undone" at the next re-scan, assuming no change was done on the RW peer? Quote Link to comment Share on other sites More sharing options...
Ronery Posted September 8, 2016 Report Share Posted September 8, 2016 Question, is " folder_rescan_interval" option available for configuration file mode on Linux? Quote Link to comment Share on other sites More sharing options...
Helen Posted September 8, 2016 Report Share Posted September 8, 2016 11 hours ago, coewar said: If nothing changes on the RW peer, but does change on the RO peer, currently what would happen? the file gets invalidated and all syncing for it stops. Even if you edit this file on RW peer later, it won't be synced. To avoid that, keep option "Overwrite any changed files" enabled on RO share. With that, if a file changes on RO, and Sync notices that, it will re-download the copy of the file from RW peer. Here you can read examples of what will happen. Hope it help. And this is one of the reasons why even on RO shares folder rescan is necessary. @Ronery, Yes, integer "folder_rescan_interval":99999. Here are other advanced option that you can put to config Quote Link to comment Share on other sites More sharing options...
IvanGill Posted April 26, 2017 Report Share Posted April 26, 2017 Windows 10, Same problem with same shared folder on multiple laptops indexing constantly and eating battery. Only the largest of three shared folders affected and I assume size / number of files is the cause. Increasing folder_rescan_interval from 600 to 1200 and restarting Resilio Sync resolved the issue. Thanks to everyone above for the advice Quote Link to comment Share on other sites More sharing options...
uremember809 Posted October 10, 2017 Report Share Posted October 10, 2017 nothing works 3 computers only one of those keep indexing and never stop for even a second Quote Link to comment Share on other sites More sharing options...
Helen Posted October 10, 2017 Report Share Posted October 10, 2017 @uremember809, please contact support with debug logs. Thank you. Quote Link to comment Share on other sites More sharing options...
John O Posted December 4, 2017 Report Share Posted December 4, 2017 @Helen - you mention that you can set the folder_rescan_interval to 0 to not have them indexed. But that is set for all folders/shares, right? I only have one folder that have a very large number of files, but I have a few other folders that do change more often and should be scanned. So my question, is it possible to set the scan to 0 for just one folder? The other folders are important for more timely and accurate syncing, while the large one is more of an archive/backup. Quote Link to comment Share on other sites More sharing options...
Helen Posted December 5, 2017 Report Share Posted December 5, 2017 8 hours ago, John O said: is it possible to set the scan to 0 for just one folder? No. You either disable it for all or keep it for all. Quote Link to comment Share on other sites More sharing options...
Jimmy Gristle Posted November 9, 2018 Report Share Posted November 9, 2018 @Helen, it's 11 months later, and this bug still hasn't been resolved on OSX (can't speak for Windows users). I've cleared the cache, paused/resumed, restarted, to no avail. Maybe you could hire someone from Dropbox to help out... Quote Link to comment Share on other sites More sharing options...
yhal Posted October 21, 2019 Report Share Posted October 21, 2019 On 4/25/2017 at 8:20 PM, IvanGill said: Windows 10, Same problem with same shared folder on multiple laptops indexing constantly and eating battery. Only the largest of three shared folders affected and I assume size / number of files is the cause. Increasing folder_rescan_interval from 600 to 1200 and restarting Resilio Sync resolved the issue. Thanks to everyone above for the advice Thanks for that solution, it worked for me. That was stuck with 1.5 gb files and start after few seconds after this change, 1200 instead of 600 for folder_rescan_interval Thanks again Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.