jdyer Posted November 4, 2014 Report Share Posted November 4, 2014 So my directory tree has approximately 35000 files at the root ( /opt/customer_home) and each of these folders have a very predictable folder tree at their root [1]. I want to ignore a few folders of the root of each account which is why I put logs/* and reports/* in the Ignore list located at the root of the sync ( /opt/customer_home/.sync/IgnoreList ) [2]. The problem is that the share never seems to finish indexing and transfers are very slow [4]. It has been running for over 36 hours now and is only at 1% with transfer speeds of less then 1mb [3]. My best guess is that because new log files are written every hour to the 35000 folders at the root then btsync is unable to keep up, even despite my configuration telling it specifically to ignore those logs. I can tell you that btsync is not syncing those logs but it still appears to be indexing them. I am not sure if this is related to the slow sync I am seeing in my other posting but just incase it wasn't I figured I would create a separate ticket [4] EnvironmentCentOS 6BTSync 1.4.99Servers in Amazon VPCAll private traffic opened between hostsRoot of share /opt/customer_homeTotal files to sync 150gbconfig [5] References [1]/opt||customer_home||-170129|| |--reports/| |--www/| |--logs|| |--2014-10-31-2200.txt.gz| |--2014-10-31-2300.txt.gz| |--2014-11-01-0000.txt.gz| |--2014-11-01-0100.txt.gz||-170130||--reports/|--www/|--logs||--2014-10-31-2200.txt.gz|--2014-10-31-2300.txt.gz|--2014-11-01-0000.txt.gz|--2014-11-01-0100.txt.gz [2][root@wh2 btsync]# cat /opt/customer_home/.sync/IgnoreList/*/logs/*/*/reports/*/*/records/**.tgz*.tar*.cdr.json*.cdr.csv*.txt.tgz*.tar.gz*.pdf*.exe*.dmg*.zip [3][20141104 14:27:43.586] SyncFolderNotify: "2014-11-04-1400.txt", event = "IN_MODIFY"[20141104 14:27:43.586] [OnNotifyFileChange] "/opt/customer_home/5000377/logs/2014-11-04-1400.txt", source = "NULL"[20141104 14:27:43.586] SyncFolderNotify: "2014-11-04-1400.txt", event = "IN_MODIFY"[20141104 14:27:43.586] [OnNotifyFileChange] "/opt/customer_home/84575/logs/2014-11-04-1400.txt", source = "NULL"[20141104 14:27:43.586] SyncFolderNotify: "2014-11-04-1400.txt", event = "IN_MODIFY"[20141104 14:27:43.586] [OnNotifyFileChange] "/opt/customer_home/84575/logs/2014-11-04-1400.txt", source = "NULL"[20141104 14:27:43.586] SyncFolderNotify: "2014-11-04-1400.txt", event = "IN_MODIFY"[20141104 14:27:43.586] [OnNotifyFileChange] "/opt/customer_home/84575/logs/2014-11-04-1400.txt", source = "NULL"[20141104 14:27:43.586] SyncFolderNotify: "2014-11-04-1400.txt", event = "IN_MODIFY"[20141104 14:27:43.586] [OnNotifyFileChange] "/opt/customer_home/84575/logs/2014-11-04-1400.txt", source = "NULL"[20141104 14:27:43.586] SyncFolderNotify: "2014-11-04-1400.txt", event = "IN_MODIFY"[20141104 14:27:43.586] [OnNotifyFileChange] "/opt/customer_home/84575/logs/2014-11-04-1400.txt", source = "NULL"[20141104 14:27:43.586] SyncFolderNotify: "2014-11-04-1400.txt", event = "IN_MODIFY"[20141104 14:27:43.586] [OnNotifyFileChange] "/opt/customer_home/84575/logs/2014-11-04-1400.txt", source = "NULL"[20141104 14:27:43.586] SyncFolderNotify: "2014-11-04-1400.txt", event = "IN_MODIFY"[20141104 14:27:43.586] [OnNotifyFileChange] "/opt/customer_home/84575/logs/2014-11-04-1400.txt", source = "NULL"[20141104 14:27:43.586] SyncFolderNotify: "2014-11-04-1400.txt", event = "IN_MODIFY"[20141104 14:27:43.586] [OnNotifyFileChange] "/opt/customer_home/84575/logs/2014-11-04-1400.txt", source = "NULL"[20141104 14:27:43.587] SyncFolderNotify: "2014-11-04-1400.txt", event = "IN_MODIFY"[20141104 14:27:43.587] [OnNotifyFileChange] "/opt/customer_home/84575/logs/2014-11-04-1400.txt", source = "NULL"[20141104 14:27:43.587] SyncFolderNotify: "2014-11-04-1400.txt", event = "IN_MODIFY"[20141104 14:27:43.587] [OnNotifyFileChange] "/opt/customer_home/5001431/logs/2014-11-04-1400.txt", source = "NULL"[20141104 14:27:43.587] SyncFolderNotify: "2014-11-04-1400.txt", event = "IN_MODIFY"[20141104 14:27:43.587] [OnNotifyFileChange] "/opt/customer_home/5001431/logs/2014-11-04-1400.txt", source = "NULL" [4] http://forum.bittorrent.com/topic/31320-extremely-slow-sync/?p=94129 [5]{ "device_name": "server1", "listening_port" : 44444, "storage_path" : "/home/btsync", "pid_file" : "/home/btsync/btsync.pid", "check_for_updates" : false, "use_upnp" : false, "download_limit" : 0, "upload_limit" : 0, "webui" : { "listen" : "0.0.0.0:8888", "login" : "xxxx", "password" : "xxxx" }, "disk_low_priority" : false, "config_refresh_interval" : 60} Quote Link to comment Share on other sites More sharing options...
jdyer Posted November 5, 2014 Author Report Share Posted November 5, 2014 Still seeing this behavior w/ todays logging, here is an excerpt. [20141105 15:37:15.561] [OnNotifyFileChange] "/opt/customer_home/87162/logs/2014-11-05-1500.txt", source = "NULL"[20141105 15:37:15.562] SyncFolderNotify: "2014-11-05-1500.txt", event = "IN_MODIFY"[20141105 15:37:15.562] [OnNotifyFileChange] "/opt/customer_home/5001431/logs/2014-11-05-1500.txt", source = "NULL"[20141105 15:37:15.562] SyncFolderNotify: "2014-11-05-1500.txt", event = "IN_MODIFY"[20141105 15:37:15.562] [OnNotifyFileChange] "/opt/customer_home/5001431/logs/2014-11-05-1500.txt", source = "NULL"[20141105 15:37:15.562] SyncFolderNotify: "2014-11-05-1500.txt", event = "IN_MODIFY"[20141105 15:37:15.562] [OnNotifyFileChange] "/opt/customer_home/5001431/logs/2014-11-05-1500.txt", source = "NULL"[20141105 15:37:15.562] SyncFolderNotify: "2014-11-05-1500.txt", event = "IN_MODIFY"[20141105 15:37:15.562] [OnNotifyFileChange] "/opt/customer_home/5001431/logs/2014-11-05-1500.txt", source = "NULL"[20141105 15:37:15.562] SyncFolderNotify: "2014-11-05-1500.txt", event = "IN_MODIFY"[20141105 15:37:15.562] [OnNotifyFileChange] "/opt/customer_home/5001431/logs/2014-11-05-1500.txt", source = "NULL"[20141105 15:37:15.562] SyncFolderNotify: "2014-11-05-1500.txt", event = "IN_MODIFY"[20141105 15:37:15.562] [OnNotifyFileChange] "/opt/customer_home/5001431/logs/2014-11-05-1500.txt", source = "NULL"[20141105 15:37:15.563] SyncFolderNotify: "2014-11-05-1500.txt", event = "IN_MODIFY"[20141105 15:37:15.563] [OnNotifyFileChange] "/opt/customer_home/5001431/logs/2014-11-05-1500.txt", source = "NULL"[20141105 15:37:15.563] SyncFolderNotify: "2014-11-05-1500.txt", event = "IN_MODIFY"[20141105 15:37:15.563] [OnNotifyFileChange] "/opt/customer_home/5001431/logs/2014-11-05-1500.txt", source = "NULL"[20141105 15:37:15.563] SyncFolderNotify: "2014-11-05-1500.txt", event = "IN_MODIFY"[20141105 15:37:15.563] [OnNotifyFileChange] "/opt/customer_home/5001431/logs/2014-11-05-1500.txt", source = "NULL"[20141105 15:37:15.563] SyncFolderNotify: "2014-11-05-1500.txt", event = "IN_MODIFY"[20141105 15:37:15.563] [OnNotifyFileChange] "/opt/customer_home/5001431/logs/2014-11-05-1500.txt", source = "NULL"[20141105 15:37:15.563] SyncFolderNotify: "2014-11-05-1500.txt", event = "IN_MODIFY"[20141105 15:37:15.563] [OnNotifyFileChange] "/opt/customer_home/5001431/logs/2014-11-05-1500.txt", source = "NULL"[20141105 15:37:15.564] SyncFolderNotify: "2014-11-05-1500.txt", event = "IN_MODIFY"[20141105 15:37:15.564] [OnNotifyFileChange] "/opt/customer_home/62700/logs/2014-11-05-1500.txt", source = "NULL"^CDoes the IgnoreList not truly ignore files ? Quote Link to comment Share on other sites More sharing options...
jdyer Posted November 5, 2014 Author Report Share Posted November 5, 2014 Anyone? Quote Link to comment Share on other sites More sharing options...
sfmcnally Posted November 6, 2014 Report Share Posted November 6, 2014 I've noticed this as well, but I think that it's just noting that the file exists but doesn't need to be synced. I don't know why it happens so often. It seems like a wast of resources to monitor files that often. Quote Link to comment Share on other sites More sharing options...
jdyer Posted November 6, 2014 Author Report Share Posted November 6, 2014 My problem is that every hour there is 35,000 files being created.. So for example at 2014-11-06-2100 I would have 35000 files created Example/opt/customer/10000/logs/2014-11-06-2100.txt/opt/customer/10001/logs/2014-11-06-2100.txt/opt/customer/10002/logs/2014-11-06-2100.txtSo this indexing takes a long time and seems to happen regularly and slow down the entire process. I am about to start looking into Syncthing because I am not getting a good feeling about bittorrent. They support seems unresponsive to these forums and to direct email requests.... Quote Link to comment Share on other sites More sharing options...
RomanZ Posted November 6, 2014 Report Share Posted November 6, 2014 @jdyerWe are answering all tickets delivered to our ticketing system, though ours support is not working 24/7 - so please be patient and we'll do our best to assist you. Forums are not primary goal for support and mainly exists to give opportunity to community to help each other. Quote Link to comment Share on other sites More sharing options...
Tommmii Posted November 10, 2014 Report Share Posted November 10, 2014 Just chiming in with a +1 on the issue.Log is spammed with indexing of ignored files. Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.