ms2oo8

Indexing After Re-Starting Sync Client

Recommended Posts

Thanks

Sorry for being anxious about this bug. We bought about $2000 of hardware that we were planning to use with Sync, which is now sitting idle due to this bug.

I'll sign up for 2.0, see if it works then.

Share this post


Link to post
Share on other sites

How does btsync scale, if it don't keep the indexed information some where in a cache. If everything is in memory, every time btsync start, does btsync need to re-read all the data from disk and re-index? Will get worst, when someone try to sync folders larger than 4TB. It is command today to have disk sizes larger than 4TB or so. Is btsync not scalable?? May be I am missing something? Can someone explain?

Share this post


Link to post
Share on other sites

@boruguru

Sync caches all the information in SQLite DB. Every time it starts it does not re-read all the data in sync folders. It loads data from DB into memory and rescans sync folder for changes. While scanning, it does not actually read files content, but rather relies on file size and modification timestamp. If any of those has changed - Sync is going to update file's hash.

So, actual folder rescan depends more of files amount rather than of total size.

Share this post


Link to post
Share on other sites

Thank you and that clarifies it. Why all information is loaded into memory when sync starts, rather you can keep them in SQLiteDB to reduce the memory consumption?. I think SQLite is a very efficient Database. This way you won't have a issue in scaling when number of files grow to a very large size. (I am guess you are loading all into memory as I saw that in another post, may be I am wrong.) 

 

Thank you if you can further clarify.

 

BG

Share this post


Link to post
Share on other sites

@boruguru

Unloading it from memory slows down every file operation a little bit - which becomes pretty much noticeable on large amount of files. In any case, we plan to optimize in future.

Share this post


Link to post
Share on other sites

having the same issue, it's stuck on indexing for some weird reason.

 

I've tried removing files to do it progressively, did not help.

 

moving to sync thing anyway.

Share this post


Link to post
Share on other sites

I know this is an old post but it's the top result in Google for "btsync indexing"

Has there been any update here - my 2TB(ish) NAS was stuck indexing for weeks so last week I replaced the disks and reinstalled and it's been stuck on indexing for 4 days now! I wouldn't mind put my CPU's are pegged at 100% so the box is pretty much unuseable while this is going on.

Share this post


Link to post
Share on other sites
6 hours ago, OttoTheBusDriver said:

I know this is an old post but it's the top result in Google for "btsync indexing"

Has there been any update here - my 2TB(ish) NAS was stuck indexing for weeks so last week I replaced the disks and reinstalled and it's been stuck on indexing for 4 days now! I wouldn't mind put my CPU's are pegged at 100% so the box is pretty much unuseable while this is going on.

Hi busdriver, i kinda of think we should somehow try to reduce the numbers of files for syncing progress. One thing I like to do in my daily spare time is to check the files I got on my hard disk. For examples, I always liked to ZIP those old files (maybe thousands of them in one folder and subfolder) into one and named it with a pretty clear name. No matter the copy or sync process would accelerate a lot. 

Not the best way to ur problem, just for a tip. Unless the CPU or HDD is fast enough, or this is the bug of programming, no really way to solve the indexing for a large amounts of files. 

Share this post


Link to post
Share on other sites
15 hours ago, YI WANG said:

Hi busdriver, i kinda of think we should somehow try to reduce the numbers of files for syncing progress. One thing I like to do in my daily spare time is to check the files I got on my hard disk. For examples, I always liked to ZIP those old files (maybe thousands of them in one folder and subfolder) into one and named it with a pretty clear name. No matter the copy or sync process would accelerate a lot. 

Not the best way to ur problem, just for a tip. Unless the CPU or HDD is fast enough, or this is the bug of programming, no really way to solve the indexing for a large amounts of files. 

Yes Yi WANG - I've thought this for a long time, most syncing solutions I have tried (Dropbox included) seem to have trouble with large numbers of files.  My NAS has hundreds of thousands of tiny files which I think causes issues.

Share this post


Link to post
Share on other sites

just to reiterate the post here, we're currently syncing 3.79 TB of files (hundreds of thousands of files). Our Sync status is "Indexing" all the time, but as far as I can tell, it's keeping files up to date.

Share this post


Link to post
Share on other sites

I see this is an older post, but I'm experiencing the same problem I think.  Indexing on a server with local storage and 3TB of data is fine, but indexing on my laptop connected to a NAS is not.  Is this still a bug with SMB and Resilio (BTsync)?  

Share this post


Link to post
Share on other sites
1 hour ago, coolrazor said:

I see this is an older post, but I'm experiencing the same problem I think.  Indexing on a server with local storage and 3TB of data is fine, but indexing on my laptop connected to a NAS is not.  Is this still a bug with SMB and Resilio (BTsync)?  

https://help.getsync.com/hc/en-us/articles/205458185-Setting-how-often-Sync-should-check-for-file-changes-

Is it possible that your laptop is too slow, so that the indexing takes so much time, that the next rescan cycle starts even though the first hasn't finished yet, resulting in endless indexing?

You might change the value to something like 10 hours (36000) to see if it's better. That's what I use on my laptop currently but I'll increase it to an even higher number, because I see little to no use in the rescan. For me it's more a security feature to not miss some depply stored files. Thus even longer periods could be used, depending on your file structure.

On the NAS you might set it to zero, because you probably don't change files on the NAS directly but rather only through sync.

Share this post


Link to post
Share on other sites

I'll try setting the rescan metric really high and see if the index completes within a day or so.  Its a good first step.  But, earlier in this thread they talked about an issue with SMB specifically.  From reading the thread the bug may never have been fixed...

Share this post


Link to post
Share on other sites
1 hour ago, coolrazor said:

I'll try setting the rescan metric really high and see if the index completes within a day or so.  Its a good first step.  But, earlier in this thread they talked about an issue with SMB specifically.  From reading the thread the bug may never have been fixed...

I thought the files you sync are stored on your laptop. But if you run Sync on your laptop to sync files stored on the NAS, why don't you simply run Sync on the NAS then, instead of your laptop?

Share this post


Link to post
Share on other sites

I used to on the old BTSync, but there isn't a package for Resilio for the Thecus NAS that I'm aware of.  I also had a really tough time getting BTSync working on it originally anyway.  So, to upgrade, I now run it on my laptop (which is relegated to sitting behind a TV being a Media Center essentially).

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.