Btsync Appears To Be Indexing Ignored Files

[jdyer]

So my directory tree has approximately 35000 files at the root ( /opt/customer_home) and each of these folders have a very predictable folder tree at their root [1]. 

 

I want to ignore a few folders of the root of each account which is why I put logs/* and reports/* in the Ignore list located at the root of the sync ( /opt/customer_home/.sync/IgnoreList ) [2].    
 
The problem is that the share never seems to finish indexing and transfers are very slow [4].  It has been running for over 36 hours now and is only at 1% with transfer speeds of less then 1mb [3].  My best guess is that because new log files are written every hour to the 35000 folders at the root then btsync is unable to keep up, even despite my configuration telling it specifically to ignore those logs.    I can tell you that btsync is not syncing those logs but it still appears to be indexing them.    I am not sure if this is related to the slow sync I am seeing in my other posting but just incase it wasn't I figured I would create a separate ticket [4]
 
 
 Environment

• CentOS 6
• BTSync 1.4.99
• Servers in Amazon VPC
• All private traffic opened between hosts
• Root of share /opt/customer_home
• Total files to sync 150gb
• config [5]

 

References
 
[1]

/opt||customer_home||-170129|| |--reports/| |--www/| |--logs|| |--2014-10-31-2200.txt.gz| |--2014-10-31-2300.txt.gz| |--2014-11-01-0000.txt.gz| |--2014-11-01-0100.txt.gz||-170130||--reports/|--www/|--logs||--2014-10-31-2200.txt.gz|--2014-10-31-2300.txt.gz|--2014-11-01-0000.txt.gz|--2014-11-01-0100.txt.gz 

 
[2]

[root@wh2 btsync]# cat /opt/customer_home/.sync/IgnoreList/*/logs/*/*/reports/*/*/records/**.tgz*.tar*.cdr.json*.cdr.csv*.txt.tgz*.tar.gz*.pdf*.exe*.dmg*.zip

 
[3]

[20141104 14:27:43.586] SyncFolderNotify: "2014-11-04-1400.txt", event = "IN_MODIFY"[20141104 14:27:43.586] [OnNotifyFileChange] "/opt/customer_home/5000377/logs/2014-11-04-1400.txt", source = "NULL"[20141104 14:27:43.586] SyncFolderNotify: "2014-11-04-1400.txt", event = "IN_MODIFY"[20141104 14:27:43.586] [OnNotifyFileChange] "/opt/customer_home/84575/logs/2014-11-04-1400.txt", source = "NULL"[20141104 14:27:43.586] SyncFolderNotify: "2014-11-04-1400.txt", event = "IN_MODIFY"[20141104 14:27:43.586] [OnNotifyFileChange] "/opt/customer_home/84575/logs/2014-11-04-1400.txt", source = "NULL"[20141104 14:27:43.586] SyncFolderNotify: "2014-11-04-1400.txt", event = "IN_MODIFY"[20141104 14:27:43.586] [OnNotifyFileChange] "/opt/customer_home/84575/logs/2014-11-04-1400.txt", source = "NULL"[20141104 14:27:43.586] SyncFolderNotify: "2014-11-04-1400.txt", event = "IN_MODIFY"[20141104 14:27:43.586] [OnNotifyFileChange] "/opt/customer_home/84575/logs/2014-11-04-1400.txt", source = "NULL"[20141104 14:27:43.586] SyncFolderNotify: "2014-11-04-1400.txt", event = "IN_MODIFY"[20141104 14:27:43.586] [OnNotifyFileChange] "/opt/customer_home/84575/logs/2014-11-04-1400.txt", source = "NULL"[20141104 14:27:43.586] SyncFolderNotify: "2014-11-04-1400.txt", event = "IN_MODIFY"[20141104 14:27:43.586] [OnNotifyFileChange] "/opt/customer_home/84575/logs/2014-11-04-1400.txt", source = "NULL"[20141104 14:27:43.586] SyncFolderNotify: "2014-11-04-1400.txt", event = "IN_MODIFY"[20141104 14:27:43.586] [OnNotifyFileChange] "/opt/customer_home/84575/logs/2014-11-04-1400.txt", source = "NULL"[20141104 14:27:43.586] SyncFolderNotify: "2014-11-04-1400.txt", event = "IN_MODIFY"[20141104 14:27:43.586] [OnNotifyFileChange] "/opt/customer_home/84575/logs/2014-11-04-1400.txt", source = "NULL"[20141104 14:27:43.587] SyncFolderNotify: "2014-11-04-1400.txt", event = "IN_MODIFY"[20141104 14:27:43.587] [OnNotifyFileChange] "/opt/customer_home/84575/logs/2014-11-04-1400.txt", source = "NULL"[20141104 14:27:43.587] SyncFolderNotify: "2014-11-04-1400.txt", event = "IN_MODIFY"[20141104 14:27:43.587] [OnNotifyFileChange] "/opt/customer_home/5001431/logs/2014-11-04-1400.txt", source = "NULL"[20141104 14:27:43.587] SyncFolderNotify: "2014-11-04-1400.txt", event = "IN_MODIFY"[20141104 14:27:43.587] [OnNotifyFileChange] "/opt/customer_home/5001431/logs/2014-11-04-1400.txt", source = "NULL"

 
[4]  http://forum.bittorrent.com/topic/31320-extremely-slow-sync/?p=94129

 

[5]

{  "device_name": "server1",  "listening_port" : 44444,  "storage_path" : "/home/btsync",  "pid_file" : "/home/btsync/btsync.pid",  "check_for_updates" : false,  "use_upnp" : false,  "download_limit" : 0,  "upload_limit" : 0,  "webui" :  {    "listen" : "0.0.0.0:8888",    "login" : "xxxx",    "password" : "xxxx"  },  "disk_low_priority" : false,  "config_refresh_interval" : 60}

[jdyer]

Still seeing this behavior w/ todays logging, here is an excerpt. 

[20141105 15:37:15.561] [OnNotifyFileChange] "/opt/customer_home/87162/logs/2014-11-05-1500.txt", source = "NULL"[20141105 15:37:15.562] SyncFolderNotify: "2014-11-05-1500.txt", event = "IN_MODIFY"[20141105 15:37:15.562] [OnNotifyFileChange] "/opt/customer_home/5001431/logs/2014-11-05-1500.txt", source = "NULL"[20141105 15:37:15.562] SyncFolderNotify: "2014-11-05-1500.txt", event = "IN_MODIFY"[20141105 15:37:15.562] [OnNotifyFileChange] "/opt/customer_home/5001431/logs/2014-11-05-1500.txt", source = "NULL"[20141105 15:37:15.562] SyncFolderNotify: "2014-11-05-1500.txt", event = "IN_MODIFY"[20141105 15:37:15.562] [OnNotifyFileChange] "/opt/customer_home/5001431/logs/2014-11-05-1500.txt", source = "NULL"[20141105 15:37:15.562] SyncFolderNotify: "2014-11-05-1500.txt", event = "IN_MODIFY"[20141105 15:37:15.562] [OnNotifyFileChange] "/opt/customer_home/5001431/logs/2014-11-05-1500.txt", source = "NULL"[20141105 15:37:15.562] SyncFolderNotify: "2014-11-05-1500.txt", event = "IN_MODIFY"[20141105 15:37:15.562] [OnNotifyFileChange] "/opt/customer_home/5001431/logs/2014-11-05-1500.txt", source = "NULL"[20141105 15:37:15.562] SyncFolderNotify: "2014-11-05-1500.txt", event = "IN_MODIFY"[20141105 15:37:15.562] [OnNotifyFileChange] "/opt/customer_home/5001431/logs/2014-11-05-1500.txt", source = "NULL"[20141105 15:37:15.563] SyncFolderNotify: "2014-11-05-1500.txt", event = "IN_MODIFY"[20141105 15:37:15.563] [OnNotifyFileChange] "/opt/customer_home/5001431/logs/2014-11-05-1500.txt", source = "NULL"[20141105 15:37:15.563] SyncFolderNotify: "2014-11-05-1500.txt", event = "IN_MODIFY"[20141105 15:37:15.563] [OnNotifyFileChange] "/opt/customer_home/5001431/logs/2014-11-05-1500.txt", source = "NULL"[20141105 15:37:15.563] SyncFolderNotify: "2014-11-05-1500.txt", event = "IN_MODIFY"[20141105 15:37:15.563] [OnNotifyFileChange] "/opt/customer_home/5001431/logs/2014-11-05-1500.txt", source = "NULL"[20141105 15:37:15.563] SyncFolderNotify: "2014-11-05-1500.txt", event = "IN_MODIFY"[20141105 15:37:15.563] [OnNotifyFileChange] "/opt/customer_home/5001431/logs/2014-11-05-1500.txt", source = "NULL"[20141105 15:37:15.563] SyncFolderNotify: "2014-11-05-1500.txt", event = "IN_MODIFY"[20141105 15:37:15.563] [OnNotifyFileChange] "/opt/customer_home/5001431/logs/2014-11-05-1500.txt", source = "NULL"[20141105 15:37:15.564] SyncFolderNotify: "2014-11-05-1500.txt", event = "IN_MODIFY"[20141105 15:37:15.564] [OnNotifyFileChange] "/opt/customer_home/62700/logs/2014-11-05-1500.txt", source = "NULL"^C

Does the IgnoreList not truly ignore files ? 

[jdyer]

Anyone?

[sfmcnally]

I've noticed this as well, but I think that it's just noting that the file exists but doesn't need to be synced. I don't know why it happens so often. It seems like a wast of resources to monitor files that often.

[jdyer]

My problem is that every hour there is 35,000 files being created.. So for example at 2014-11-06-2100 I would have 35000 files created 

 

Example

/opt/customer/10000/logs/2014-11-06-2100.txt/opt/customer/10001/logs/2014-11-06-2100.txt/opt/customer/10002/logs/2014-11-06-2100.txt

So this indexing takes a long time and seems to happen regularly and slow down the entire process. I am about to start looking into Syncthing because I am not getting a good feeling about bittorrent.  They support seems unresponsive to these forums and to direct email requests.... 

[RomanZ]

@jdyer

We are answering all tickets delivered to our ticketing system, though ours support is not working 24/7 - so please be patient and we'll do our best to assist you. Forums are not primary goal for support and mainly exists to give opportunity to community to help each other.

[Tommmii]

Just chiming in with a +1 on the issue.

Log is spammed with indexing of ignored files.