Takes Forever To Load List Of Sync Folders

[benplumley]

When I turn on my computer, it can take up to 20 minutes for Sync to get past the 'loading list of sync folders' stage. I only have 3 synced folders with a combined total of under 20GB. If I kill and restart Sync, it takes just as long.

[Helen]

benplumley,

 

How many files do you have in these folders? Are these files big? What is your PC OS? 

 

When BitTorrent startsб it reads info about synced folder/files from sync.dat file. Is sync.dat in your case available for BitTorrent Sync? It is located in: 

Windows: %APPDATA%\BitTorrent Sync\sync.log
Mac: ~/Library/Application Support/BitTorrent Sync/sync.log
Linux: ./sync/sync.log

 

 Did you try re-adding the folders to BitTorrent Sync? 

[benplumley]

Total size 18.24GB, total number of files 15583. I'll try re-adding the folders. I have sync.dat and I'm able to open it, so I'd assume BTSync can open it too.

 

I have another problem as of yesterday that's causing all files to be unable to sync. It affects all my peers. Here are the relevant debug log lines: 

redacted, will reupload

The last line is repeated once per second. The file never gets synced, and nor do any other files I add. The debug log doesn't mention any other files, just Past Papers.

Edited June 14, 2014 by benplumley

[benplumley]

Here are the logs.

[2014-06-14 20:26:37] Fully synced with 10905A0FF843F7039A4DFCA735465265614E490A[2014-06-14 20:26:37] Merge: processing nodes message for /[2014-06-14 20:26:37] Merge: will request nodes for /Barton[2014-06-14 20:26:37] Merge: sending get_nodes message[2014-06-14 20:26:38] Sending broadcast ping for 4 shares[2014-06-14 20:26:38] Merge: processing nodes message for /Barton[2014-06-14 20:26:38] Merge: will request files for /Barton[2014-06-14 20:26:38] Merge: will request nodes for /Barton/Past Papers.xlsx[2014-06-14 20:26:38] Merge: sending get_nodes message[2014-06-14 20:26:38] Merge: processing nodes message for /Barton/Past Papers.xlsx[2014-06-14 20:26:38] Merge: will request files for /Barton/Past Papers.xlsx[2014-06-14 20:26:38] Merge: sending get_files message[2014-06-14 20:26:38] Merge: processing files message with 2 files[2014-06-14 20:26:38] Merge: Local file Barton is older (1402769804) than remote (1402773817)[2014-06-14 20:26:38] Merge: Local file Barton\Past Papers.xlsx is older (1402666636) than remote (1402773817)[2014-06-14 20:26:38] SyncFilesController: Got 2 files from remote (192.168.0.20:33088)[2014-06-14 20:26:38] SyncFilesController: Got file from remote (192.168.0.20:33088): Barton state: 1 type: folder total:0 have:0 t:1402773817 mt:1402773998 0000000000000000000000000000000000000000[2014-06-14 20:26:38] SyncFilesController: Got file from remote (192.168.0.20:33088): Barton\Past Papers.xlsx state: 1 type: file total:2 have:2 t:1402773817 mt:1402773794 6004C0E6A150A68A8457A2CDD63F3FF15C21A89C[2014-06-14 20:26:38] SyncFilesController: remote file Barton\Past Papers.xlsx is newer and different from local. Removing local torrent.[2014-06-14 20:26:38] State sync finished for folder \\?\C:\Users\Ben\Documents[2014-06-14 20:26:38] Tree hash changed while merging, cancelling sync with 10905A0FF843F7039A4DFCA735465265614E490A[2014-06-14 20:26:39] LoadTorrent: requesting to load file Barton\Past Papers.xlsx which is modified on disk (cur:1402773794 stored:1402773794 size1:59789 size2:59789 hasInfo:1) - rejecting until file info is updated[2014-06-14 20:26:40] LoadTorrent: requesting to load file Barton\Past Papers.xlsx which is modified on disk (cur:1402773794 stored:1402773794 size1:59789 size2:59789 hasInfo:1) - rejecting until file info is updated[2014-06-14 20:26:41] LoadTorrent: requesting to load file Barton\Past Papers.xlsx which is modified on disk (cur:1402773794 stored:1402773794 size1:59789 size2:59789 hasInfo:1) - rejecting until file info is updated 

I just re-added the folder, and it's completely broken it. There are ~15000 files in this folder and it indexes 32 files then stops. I'll try reinstalling it.

Update: it's indexing them now.

[RomanZ]

@benplumley

 

Okay, what about initial issue? Are files re-indexed 20 minutes or it is done in a prompt manner now?

[bradbamford]

I have the same problem. I'm running Windows 2008 R2 on both ends on a LAN.

 

2 shared folders

 

1 folder with 6.1GB in     34,330 files

1 folder with 1.9TB in 1,606,693 files

 

Both client versions: 1.9.109
LAN Encription is false
Disk low priority is false
Force LAN TCP is missing (was this setting removed?)
Debug Logs is empty with just the standard FFF

[RomanZ]

@bradbamford

 

Under "same problem" I guess you have the topic name issue - folders loading takes a lot of time. 

1) What is the amount of memory on your server? 1.6M of files is large amount and will consume lots of memory.

2) Is sync.log completely empty? There should be some data even when debug logging is off. Could you please verify it? note that debug.txt should contain "FFFF" instead of 3 Fs (just in case it was not a typo).

[bradbamford]

@bradbamford

 

Under "same problem" I guess you have the topic name issue - folders loading takes a lot of time. 

1) What is the amount of memory on your server? 1.6M of files is large amount and will consume lots of memory.

2) Is sync.log completely empty? There should be some data even when debug logging is off. Could you please verify it? note that debug.txt should contain "FFFF" instead of 3 Fs (just in case it was not a typo).

 

Happens on both ends.

Each box has 24GB ram.

On average, BTSync.exe consumes 900,000 K memory.

Total memory use is always under 35%, CPU under 25%.

Yes, sync.log works just fine, but the debug.txt never shows any errors.

To test, I blocked BTSync in Windows Firewall. That added entries to debug.txt, but the "folders loading takes a lot of time" issue never shows any errors. It's just slow.

[Webmaster33]

I have the same problem, loading Resilio Sync takes forever, or at least too much to handle the syncing.

474 080 files in 97 336 directories, total of 36 Gb.
Windows 7 Pro on both sides.

Could you put efforts to optimize loading of the files?
Or do it in background, because the shared directories are not even displayed, before Loading is not finished.

Thanks!

 

On 2014. 06. 13. at 2:57 PM, Helen said:

benplumley,

 

How many files do you have in these folders? Are these files big? What is your PC OS? 

 

When BitTorrent startsб it reads info about synced folder/files from sync.dat file. Is sync.dat in your case available for BitTorrent Sync? It is located in: 

Windows: %APPDATA%\BitTorrent Sync\sync.log
Mac: ~/Library/Application Support/BitTorrent Sync/sync.log
Linux: ./sync/sync.log

 

 Did you try re-adding the folders to BitTorrent Sync? 

 

 

 

[Helen]

are you sure this is the same problem? The tread is almost 3 years old, and since then Sync has been updated quite a few times. What Sync version you have now? 
If it takes too long to load, then this might be just loading database to RAM (it's been discussed here on forum multiple times). Check size of .db files in storage folder 

[Webmaster33]

4 hours ago, Helen said:

are you sure this is the same problem? The tread is almost 3 years old, and since then Sync has been updated quite a few times. What Sync version you have now? 
If it takes too long to load, then this might be just loading database to RAM (it's been discussed here on forum multiple times). Check size of .db files in storage folder 

Using Resilio Sync v2.5.5 on both side.
The used memory by Resilio Sync is about 1.3Gb, using 25% of 4 processors, so 1 processor is running at 100% (which is used by Resilio Sync).
Total physical memory is 8Gb, 85% is used now.

Also I get very often app has "Not responding" State in task list, for long time.

Likely it is the same problem, as I use Bittorrent Sync since around v1.2 - 1.3 version. About since 2014, 3 years :-)

I always had this problem, but in earlier versions it was not so bad.
Perhaps as the file number increases, the loading time & indexing time increased.
But now it makes Resilio Sync unusable. I have to wait several hours to become responsive & finish loading, then do something.
The 474 080 files in 97 336 directories, total of 36 Gb. should be not much for an online backup software, like Resilio Sync.

Today, after several hours I reached the state, that I was able to copy key, then paste into other side.

EDIT2:
Total size of .db files stored in c:\Users\user\AppData\Roaming\Resilio Sync\: 5,65 Gb.

EDIT3:
I will try to add to the .sync\IgnoreList of one backup folder, which contains Mozilla profile, to exclude the cache:
\Mozilla_Profile\Profiles\myprofile\thumbnails\
\Mozilla_Profile\Profiles\myprofile\cache2\

 

[Helen]

oh, RAM usage, database have been discussed here on forum a very lot times. Sync on start reads the database  and loads them to RAM. the bigger database you have, the longer it takes. 

Syncing other apps cache and their service files is indeed not a good idea - these change quite a lot, new ones are created -> database grows bigger, more RAM taken, more resources are necessary to manage this folder - the folder tree is really big.

Apart from ignoring the files, you can remove the share from Sync and add it back so that database gets cleared. Also, increate rescan interval in power user settings. 

[Webmaster33]

Thanks for your reply & ideas.

I would not like to re-add the folders, because syncing 35-40Gb would take a long time.
I will only do this, if this would clear such big database items, which would not be cleared after a time.

Yesterday I increased rescan interval from 2 hour to 6 hour (21600).
Also, yesterday I upgraded from Resilio Sync v2.5.5 to Resilio Sync v2.5.6.

Currently the sync is going, but two folders (10 Gb and 5 Gb, both having  a lot files, are progressing very slowly, currently they are at 4%).

 

[Helen]

22 hours ago, Webmaster33 said:

because syncing 35-40Gb would take a long time.

there's no need to resync all the files, you can use the folders with already synced files.

[Webmaster33]

On 2017. 07. 18. at 10:02 AM, Helen said:

oh, RAM usage, database have been discussed here on forum a very lot times. Sync on start reads the database  and loads them to RAM. the bigger database you have, the longer it takes. 

Syncing other apps cache and their service files is indeed not a good idea - these change quite a lot, new ones are created -> database grows bigger, more RAM taken, more resources are necessary to manage this folder - the folder tree is really big.

Apart from ignoring the files, you can remove the share from Sync and add it back so that database gets cleared. Also, increate rescan interval in power user settings. 

I think the 5Gb .db file is a bit too much for storing info of 500.000 files.
In average, it seems that one file needs about 10700 bytes in the database.
Seems to be a but too much.

What data is stored in the db?

• filename
• filesize
• directory name & path
• hash(es) of file

Any history data or other data are stored in the .db file?
 

[Helen]

Sync keep entries of files that are now and that have ever been in the share.  Entry - if the file and each subfolder level in its path. Count each of them around 2 Kb. 

Database 0 is just the record of entry <-> data. Data - timestamp, modified timestamp, size, total number of pieces, local number of pieces, selective sync or not, hash, file owner, synced or not, etc. 

No, history is not stored

[Webmaster33]

On 2017. 08. 07. at 7:42 AM, Helen said:

Sync keep entries of files that are now and that have ever been in the share.

So the db will grow endless...???

Isn't there need for garbage collection?
There are a lot files, which are deleted, and will never come back.
The space of those entries could be recycled after a few months.

[Webmaster33]

Currently still takes several hours to load the Resilio Sync after restarting it.
Now just says since 4-5 hours at window title: Sync Home (not responding).

[Namtrooper]

I would like to add that this same issue still occurs in 2018.

[Gane O'dwyer]

@Namtrooper as was already mention in this topic Sync on start reads the database  and loads them to RAM. the bigger database you have, the longer it takes. As a quick fix try to re-add folders in Sync and check the results otherwise please do contact support team and describe your setup in details and include information about your peer, type of OS, Sync version etc.

[Zipperhead]

I have this same issue, have read this entire thread and do not see any solution.   I am syncing to folders with many sub folders and many large files per subfolder.  But with selective sync turned on I would expect the ".rsls" place holder files to sync up quickly since the entire file is not syncing across the network.  but it seems that each .rsls placeholder file takes 15-30 seconds or so to show up so it will take many hours for all the placeholder files to show up.  Is there any settings that can be changed to increase the speed of the selective sync?   Everything seems to be working but very slow.  And it's not a bandwidth issue because I have about 180 Mbps download 10 Mbps upload speed.

Note:  some files take longer than others, like a minute or two.  And perhaps this is just how long it takes to sync those files over the internet and if so I am ok with that.  I just want to know if there are settings that can be changed to speed up the selective sync of the placeholder ".rsls" files.

Edited February 5, 2019 by Zipperhead
wanted to expound a little on the issue

[Gnathonic]

The problem is that Resilio doesn't handle its database files very well. Old entries, internal fragmentation, something is wrong with them that causes extended load times.

Fortunately these db files don't actually have anything critically important in them. It is used as a cache and for tracking deltas. If you are synced up you can safely force it to generate fresh ones by deleting the old degraded ones and launching Resilio. You can find the offending db files here "%appdata%\Resilio Sync". Close Resilio. Delete all suspiciously large db files. Then relaunch Resilio. It will begin indexing and recreate the missing db files based on the current state of the associated sync folder. These freshly created db files will load much faster than the older degraded ones they replaced.