jdyer

Btsync Appears To Be Indexing Ignored Files

Recommended Posts

So my directory tree has approximately 35000 files at the root ( /opt/customer_home) and each of these folders have a very predictable folder tree at their root [1]. 

 

I want to ignore a few folders of the root of each account which is why I put logs/* and reports/* in the Ignore list located at the root of the sync ( /opt/customer_home/.sync/IgnoreList ) [2].    
 
The problem is that the share never seems to finish indexing and transfers are very slow [4].  It has been running for over 36 hours now and is only at 1% with transfer speeds of less then 1mb [3].  My best guess is that because new log files are written every hour to the 35000 folders at the root then btsync is unable to keep up, even despite my configuration telling it specifically to ignore those logs.    I can tell you that btsync is not syncing those logs but it still appears to be indexing them.    I am not sure if this is related to the slow sync I am seeing in my other posting but just incase it wasn't I figured I would create a separate ticket [4]
 
 
 Environment

  • CentOS 6
  • BTSync 1.4.99
  • Servers in Amazon VPC
  • All private traffic opened between hosts
  • Root of share /opt/customer_home
  • Total files to sync 150gb
  • config [5]

 

References
 
[1]

/opt||customer_home||-170129|| |--reports/| |--www/| |--logs|| |--2014-10-31-2200.txt.gz| |--2014-10-31-2300.txt.gz| |--2014-11-01-0000.txt.gz| |--2014-11-01-0100.txt.gz||-170130||--reports/|--www/|--logs||--2014-10-31-2200.txt.gz|--2014-10-31-2300.txt.gz|--2014-11-01-0000.txt.gz|--2014-11-01-0100.txt.gz 

 
[2]

[root@wh2 btsync]# cat /opt/customer_home/.sync/IgnoreList/*/logs/*/*/reports/*/*/records/**.tgz*.tar*.cdr.json*.cdr.csv*.txt.tgz*.tar.gz*.pdf*.exe*.dmg*.zip

 
[3]

[20141104 14:27:43.586] SyncFolderNotify: "2014-11-04-1400.txt", event = "IN_MODIFY"[20141104 14:27:43.586] [OnNotifyFileChange] "/opt/customer_home/5000377/logs/2014-11-04-1400.txt", source = "NULL"[20141104 14:27:43.586] SyncFolderNotify: "2014-11-04-1400.txt", event = "IN_MODIFY"[20141104 14:27:43.586] [OnNotifyFileChange] "/opt/customer_home/84575/logs/2014-11-04-1400.txt", source = "NULL"[20141104 14:27:43.586] SyncFolderNotify: "2014-11-04-1400.txt", event = "IN_MODIFY"[20141104 14:27:43.586] [OnNotifyFileChange] "/opt/customer_home/84575/logs/2014-11-04-1400.txt", source = "NULL"[20141104 14:27:43.586] SyncFolderNotify: "2014-11-04-1400.txt", event = "IN_MODIFY"[20141104 14:27:43.586] [OnNotifyFileChange] "/opt/customer_home/84575/logs/2014-11-04-1400.txt", source = "NULL"[20141104 14:27:43.586] SyncFolderNotify: "2014-11-04-1400.txt", event = "IN_MODIFY"[20141104 14:27:43.586] [OnNotifyFileChange] "/opt/customer_home/84575/logs/2014-11-04-1400.txt", source = "NULL"[20141104 14:27:43.586] SyncFolderNotify: "2014-11-04-1400.txt", event = "IN_MODIFY"[20141104 14:27:43.586] [OnNotifyFileChange] "/opt/customer_home/84575/logs/2014-11-04-1400.txt", source = "NULL"[20141104 14:27:43.586] SyncFolderNotify: "2014-11-04-1400.txt", event = "IN_MODIFY"[20141104 14:27:43.586] [OnNotifyFileChange] "/opt/customer_home/84575/logs/2014-11-04-1400.txt", source = "NULL"[20141104 14:27:43.586] SyncFolderNotify: "2014-11-04-1400.txt", event = "IN_MODIFY"[20141104 14:27:43.586] [OnNotifyFileChange] "/opt/customer_home/84575/logs/2014-11-04-1400.txt", source = "NULL"[20141104 14:27:43.586] SyncFolderNotify: "2014-11-04-1400.txt", event = "IN_MODIFY"[20141104 14:27:43.586] [OnNotifyFileChange] "/opt/customer_home/84575/logs/2014-11-04-1400.txt", source = "NULL"[20141104 14:27:43.587] SyncFolderNotify: "2014-11-04-1400.txt", event = "IN_MODIFY"[20141104 14:27:43.587] [OnNotifyFileChange] "/opt/customer_home/84575/logs/2014-11-04-1400.txt", source = "NULL"[20141104 14:27:43.587] SyncFolderNotify: "2014-11-04-1400.txt", event = "IN_MODIFY"[20141104 14:27:43.587] [OnNotifyFileChange] "/opt/customer_home/5001431/logs/2014-11-04-1400.txt", source = "NULL"[20141104 14:27:43.587] SyncFolderNotify: "2014-11-04-1400.txt", event = "IN_MODIFY"[20141104 14:27:43.587] [OnNotifyFileChange] "/opt/customer_home/5001431/logs/2014-11-04-1400.txt", source = "NULL"

 
[4]  http://forum.bittorrent.com/topic/31320-extremely-slow-sync/?p=94129

 

[5]

{  "device_name": "server1",  "listening_port" : 44444,  "storage_path" : "/home/btsync",  "pid_file" : "/home/btsync/btsync.pid",  "check_for_updates" : false,  "use_upnp" : false,  "download_limit" : 0,  "upload_limit" : 0,  "webui" :  {    "listen" : "0.0.0.0:8888",    "login" : "xxxx",    "password" : "xxxx"  },  "disk_low_priority" : false,  "config_refresh_interval" : 60}

Share this post


Link to post
Share on other sites

Still seeing this behavior w/ todays logging, here is an excerpt. 

[20141105 15:37:15.561] [OnNotifyFileChange] "/opt/customer_home/87162/logs/2014-11-05-1500.txt", source = "NULL"[20141105 15:37:15.562] SyncFolderNotify: "2014-11-05-1500.txt", event = "IN_MODIFY"[20141105 15:37:15.562] [OnNotifyFileChange] "/opt/customer_home/5001431/logs/2014-11-05-1500.txt", source = "NULL"[20141105 15:37:15.562] SyncFolderNotify: "2014-11-05-1500.txt", event = "IN_MODIFY"[20141105 15:37:15.562] [OnNotifyFileChange] "/opt/customer_home/5001431/logs/2014-11-05-1500.txt", source = "NULL"[20141105 15:37:15.562] SyncFolderNotify: "2014-11-05-1500.txt", event = "IN_MODIFY"[20141105 15:37:15.562] [OnNotifyFileChange] "/opt/customer_home/5001431/logs/2014-11-05-1500.txt", source = "NULL"[20141105 15:37:15.562] SyncFolderNotify: "2014-11-05-1500.txt", event = "IN_MODIFY"[20141105 15:37:15.562] [OnNotifyFileChange] "/opt/customer_home/5001431/logs/2014-11-05-1500.txt", source = "NULL"[20141105 15:37:15.562] SyncFolderNotify: "2014-11-05-1500.txt", event = "IN_MODIFY"[20141105 15:37:15.562] [OnNotifyFileChange] "/opt/customer_home/5001431/logs/2014-11-05-1500.txt", source = "NULL"[20141105 15:37:15.562] SyncFolderNotify: "2014-11-05-1500.txt", event = "IN_MODIFY"[20141105 15:37:15.562] [OnNotifyFileChange] "/opt/customer_home/5001431/logs/2014-11-05-1500.txt", source = "NULL"[20141105 15:37:15.563] SyncFolderNotify: "2014-11-05-1500.txt", event = "IN_MODIFY"[20141105 15:37:15.563] [OnNotifyFileChange] "/opt/customer_home/5001431/logs/2014-11-05-1500.txt", source = "NULL"[20141105 15:37:15.563] SyncFolderNotify: "2014-11-05-1500.txt", event = "IN_MODIFY"[20141105 15:37:15.563] [OnNotifyFileChange] "/opt/customer_home/5001431/logs/2014-11-05-1500.txt", source = "NULL"[20141105 15:37:15.563] SyncFolderNotify: "2014-11-05-1500.txt", event = "IN_MODIFY"[20141105 15:37:15.563] [OnNotifyFileChange] "/opt/customer_home/5001431/logs/2014-11-05-1500.txt", source = "NULL"[20141105 15:37:15.563] SyncFolderNotify: "2014-11-05-1500.txt", event = "IN_MODIFY"[20141105 15:37:15.563] [OnNotifyFileChange] "/opt/customer_home/5001431/logs/2014-11-05-1500.txt", source = "NULL"[20141105 15:37:15.563] SyncFolderNotify: "2014-11-05-1500.txt", event = "IN_MODIFY"[20141105 15:37:15.563] [OnNotifyFileChange] "/opt/customer_home/5001431/logs/2014-11-05-1500.txt", source = "NULL"[20141105 15:37:15.564] SyncFolderNotify: "2014-11-05-1500.txt", event = "IN_MODIFY"[20141105 15:37:15.564] [OnNotifyFileChange] "/opt/customer_home/62700/logs/2014-11-05-1500.txt", source = "NULL"^C

Does the IgnoreList not truly ignore files ? 

Share this post


Link to post
Share on other sites

I've noticed this as well, but I think that it's just noting that the file exists but doesn't need to be synced. I don't know why it happens so often. It seems like a wast of resources to monitor files that often.

Share this post


Link to post
Share on other sites

My problem is that every hour there is 35,000 files being created.. So for example at 2014-11-06-2100 I would have 35000 files created 

 

Example

/opt/customer/10000/logs/2014-11-06-2100.txt/opt/customer/10001/logs/2014-11-06-2100.txt/opt/customer/10002/logs/2014-11-06-2100.txt

So this indexing takes a long time and seems to happen regularly and slow down the entire process. I am about to start looking into Syncthing because I am not getting a good feeling about bittorrent.  They support seems unresponsive to these forums and to direct email requests.... 

Share this post


Link to post
Share on other sites

@jdyer

We are answering all tickets delivered to our ticketing system, though ours support is not working 24/7 - so please be patient and we'll do our best to assist you. Forums are not primary goal for support and mainly exists to give opportunity to community to help each other.

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.