Sync Appears To Get Swamped Indexing

[Shaav]

I realize this is an issue that people bring up here a lot; I haven't really seen much by way of solutions---seems like a lot of them abandon btsync before it gets resolved. I'm hoping that I have more detail to offer about what happens than others...

 

Situation: 4 QNAP NAS drives syncing about 2.3TB of data; ~700K files. All 4 drives are using the latest 2 version. All the drives currently have ~99% identical content; just differences that have occurred from their use since the problems began. 2 of the drives have been functioning perfectly for about 8 months; the problem came when we tried to add 2 new drives. Left them to index for days and days without there seeming to be any change. Because of an accidental deletion, one of the original NAS's ended up dumping all 2.3TB into "Archive"; those were moved back in place, but of course, then that one needed to reindex everything too. All of them have their folder rescan set to 24hrs, which until now, I thought would be more than enough to handle the indexing of what I currently am shifting...

 

Not getting anywhere, I eventually stopped the original folder from syncing at all and set up a new sync folder and started shifting small amounts of data from the old to the new with the idea that if there's something corrupt about the share, then that will get fixed; if it's a problem with files or communication, that would be easier to diagnose with a smaller data set and if it's a problem with resources then it should be easier to manage and actually see tangible progress, get a handle on how long it takes to actually do stuff.

 

I started with 12GB --- went OK. Next I did about 30GB -- took *I think* about 4hrs to index and sync. Last night I did approximately 76GB and now more than 24hrs later, it looks like progress has halted. All four report different sizes, only one of which I think matches the size of the share (trying to take into account excluded files and rounding differences etc). Two report all 3 peers on line, 1 reports 2 peers online and 1 reports 0.

 

I have written some scripts to pull file lists from the drives and filter them with the excludes so that any differences are reflective of real differences and I can confirm that right now, no files are being updated.

 

I'm happy to provide log files --- I'm not making much sense of them --- but I'd rather not post them publicly. This is a client's system and they use a lot of identifiable personal information in filenames/paths. Is there another way I can upload them a little more privately?

 

There is definitely a lot of looping going on. Two have a lot of "rejecting until file info is updated". Another has a lot of "Failed to create empty suffix for file". The one that looks like it actually indexed everything appears the most normal, but still have a bunch of stuff like "Trash: requested file was not found".

 

 

 

Clearly, a big part of the problem is the combination of data volume and the processing capabilities of the NAS drives themselves. So one thing I'm wondering is whether it is possible to bootstrap that process by, say mounting the share on my hexacore behemoth as an nfs drive or something, and have it do the initial indexing? When I do an initial mirror of data from one to the other, could I copy the hashes too?

 

One thing that is profoundly frustrating about the UI, is that it is extremely difficult to determine what is actually happening. Especially when the devices are under a heavy load, it doesn't accurately reflect the actual state of the sync. Even trying to load the options, I sometimes have to wait 10s of seconds for the options to appear, particularly pre-defined hosts. The "indexing" indicator flashes on and off seemingly at random and you have no idea if it done or not done.  I'll try to pause/resume the syncing and sometimes it doesn't work. Often a 'du' of the folder is wildly out of whack with what it is reporting as the size in the UI, even when it doesn't seem to be indexing (but possibly is). Sometimes, the size changes and seems to reflect indexing going on; sometimes it just sits at the same amount for minutes/hours even when it seems to be indexing (but could be stuck).

 

It would be really useful if, especially for these massive initial indexes, there was a mode where you could just tell the device to just index the folder and provide a progress indicator and clear indication when it is done. If there's a lot to index --- maybe pick a threshold --- I wish it would put everything else on hold until it's done that process. Or at least it was a configurable option, especially when you add a folder that already has content.

 

Any help would be greatly appreciated.