Indexing After Re-Starting Sync Client


Recommended Posts

  • 3 weeks later...
  • 1 month later...

How does btsync scale, if it don't keep the indexed information some where in a cache. If everything is in memory, every time btsync start, does btsync need to re-read all the data from disk and re-index? Will get worst, when someone try to sync folders larger than 4TB. It is command today to have disk sizes larger than 4TB or so. Is btsync not scalable?? May be I am missing something? Can someone explain?

Link to post
Share on other sites

@boruguru

Sync caches all the information in SQLite DB. Every time it starts it does not re-read all the data in sync folders. It loads data from DB into memory and rescans sync folder for changes. While scanning, it does not actually read files content, but rather relies on file size and modification timestamp. If any of those has changed - Sync is going to update file's hash.

So, actual folder rescan depends more of files amount rather than of total size.

Link to post
Share on other sites

Thank you and that clarifies it. Why all information is loaded into memory when sync starts, rather you can keep them in SQLiteDB to reduce the memory consumption?. I think SQLite is a very efficient Database. This way you won't have a issue in scaling when number of files grow to a very large size. (I am guess you are loading all into memory as I saw that in another post, may be I am wrong.) 

 

Thank you if you can further clarify.

 

BG

Link to post
Share on other sites
  • 3 months later...
  • 1 year later...

I know this is an old post but it's the top result in Google for "btsync indexing"

Has there been any update here - my 2TB(ish) NAS was stuck indexing for weeks so last week I replaced the disks and reinstalled and it's been stuck on indexing for 4 days now! I wouldn't mind put my CPU's are pegged at 100% so the box is pretty much unuseable while this is going on.

Link to post
Share on other sites
6 hours ago, OttoTheBusDriver said:

I know this is an old post but it's the top result in Google for "btsync indexing"

Has there been any update here - my 2TB(ish) NAS was stuck indexing for weeks so last week I replaced the disks and reinstalled and it's been stuck on indexing for 4 days now! I wouldn't mind put my CPU's are pegged at 100% so the box is pretty much unuseable while this is going on.

Hi busdriver, i kinda of think we should somehow try to reduce the numbers of files for syncing progress. One thing I like to do in my daily spare time is to check the files I got on my hard disk. For examples, I always liked to ZIP those old files (maybe thousands of them in one folder and subfolder) into one and named it with a pretty clear name. No matter the copy or sync process would accelerate a lot. 

Not the best way to ur problem, just for a tip. Unless the CPU or HDD is fast enough, or this is the bug of programming, no really way to solve the indexing for a large amounts of files. 

Link to post
Share on other sites
15 hours ago, YI WANG said:

Hi busdriver, i kinda of think we should somehow try to reduce the numbers of files for syncing progress. One thing I like to do in my daily spare time is to check the files I got on my hard disk. For examples, I always liked to ZIP those old files (maybe thousands of them in one folder and subfolder) into one and named it with a pretty clear name. No matter the copy or sync process would accelerate a lot. 

Not the best way to ur problem, just for a tip. Unless the CPU or HDD is fast enough, or this is the bug of programming, no really way to solve the indexing for a large amounts of files. 

Yes Yi WANG - I've thought this for a long time, most syncing solutions I have tried (Dropbox included) seem to have trouble with large numbers of files.  My NAS has hundreds of thousands of tiny files which I think causes issues.

Link to post
Share on other sites
  • 6 months later...
1 hour ago, coolrazor said:

I see this is an older post, but I'm experiencing the same problem I think.  Indexing on a server with local storage and 3TB of data is fine, but indexing on my laptop connected to a NAS is not.  Is this still a bug with SMB and Resilio (BTsync)?  

https://help.getsync.com/hc/en-us/articles/205458185-Setting-how-often-Sync-should-check-for-file-changes-

Is it possible that your laptop is too slow, so that the indexing takes so much time, that the next rescan cycle starts even though the first hasn't finished yet, resulting in endless indexing?

You might change the value to something like 10 hours (36000) to see if it's better. That's what I use on my laptop currently but I'll increase it to an even higher number, because I see little to no use in the rescan. For me it's more a security feature to not miss some depply stored files. Thus even longer periods could be used, depending on your file structure.

On the NAS you might set it to zero, because you probably don't change files on the NAS directly but rather only through sync.

Link to post
Share on other sites

I'll try setting the rescan metric really high and see if the index completes within a day or so.  Its a good first step.  But, earlier in this thread they talked about an issue with SMB specifically.  From reading the thread the bug may never have been fixed...

Link to post
Share on other sites
  • GreatMarko changed the title to Indexing After Re-Starting Sync Client
1 hour ago, coolrazor said:

I'll try setting the rescan metric really high and see if the index completes within a day or so.  Its a good first step.  But, earlier in this thread they talked about an issue with SMB specifically.  From reading the thread the bug may never have been fixed...

I thought the files you sync are stored on your laptop. But if you run Sync on your laptop to sync files stored on the NAS, why don't you simply run Sync on the NAS then, instead of your laptop?

Link to post
Share on other sites

I used to on the old BTSync, but there isn't a package for Resilio for the Thecus NAS that I'm aware of.  I also had a really tough time getting BTSync working on it originally anyway.  So, to upgrade, I now run it on my laptop (which is relegated to sitting behind a TV being a Media Center essentially).

Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.