jvavruska Posted March 14, 2014 Report Share Posted March 14, 2014 Hello, we deployed BTSync 1.2.91 (i386 linux version) on 4 linux machines, 1 CentOS and 3 NAS TS-269 (linux-based operating system QTS). I noticed that on various occasions CPU load as dislayed by the top utility shows btsync running at the top with high cpu loads over 85%, often peaking to 99.9%. Some machines reached a consistent 99.9% level after about a day. When the porcessed reached 99.9%, GUI responds slowly, some synced folders appear not to be synchronized (at least in GUI) and there seem to be hanging (infinitely pending) transfers. I noticed similar behavior with version 1.2.82, where CentOS process often crashed with segfault and QTS machines as if "accumulated" load until 99.9% when they stopped working, although the process itself did not crash. Temporary solution implemented is periodical restart of btsync every two hours. After restart, btsync starts at low CPU loads (1 to 10%) and is getting busier as time passes. Before it becomes completely dead it is gracefully restarted by a cron job. Could this be answer to some of you guys who reported insanely slow download speeds? Regards, Jindrich Vavruska Quote Link to comment Share on other sites More sharing options...
GreatMarko Posted March 14, 2014 Report Share Posted March 14, 2014 There is a newer version of Sync available, do you see the same behavior using the latest 1.2.92 build ? Quote Link to comment Share on other sites More sharing options...
RomanZ Posted March 17, 2014 Report Share Posted March 17, 2014 jvavruska, Couple of questions to clarify situation:- What happens next? Does CPU load ever drops or it is stays at 99%? - How many files / folders do you have? What is the summary volume?- What are the CPUs on your computers?- Does your CentOS suffers the same issue as NAS devices?- could you please make a debug logs for couple of peers so we can take a glance what happens inside during 100% CPU usage? Thanks! Quote Link to comment Share on other sites More sharing options...
jvavruska Posted March 24, 2014 Author Report Share Posted March 24, 2014 Hello, from what I could observe, once they reach this level (after a day) it stays high and blocks any transfers. - Since we have it on production machines and do not have better alternative, I put a crontab restart every two hours. After restart, btsync is temporarily busy as it loads all folders anew, but after few minutes, web gui starts working and the load drops to some 18-30%. Peaks occur occasionally, probably during reindexing, but stay temporary. - some 400K+ files, almost 200GB volume, total tree structure has some 18K+ folders. There are about 50 synchronized folders (btsync entries) ranging from few MB up to 34 GB total volume. (We are in a planning phase to put most of it out of sync to a permanent archive, but this is our current load) - CPU on QNAP is dual core Pentium (1.9 GHz IIRC), CentOS machine is a different flavor but also a dual core Pentium. I will have a look for the logs. I will switch off the restarts on weekend and try to collect more information. What we have now would probably not be representative because I installed a regular 2-hour restart in all units. In the meantime we switched off CentOS (it was receiving uploaded files from web server but we redirected the upload directly to the QNAP NAS) so I am afraid it will not provide any more data. Regards, Jindra Quote Link to comment Share on other sites More sharing options...
RomanZ Posted March 25, 2014 Report Share Posted March 25, 2014 Jindra, Thanks for details. To debug this issue and find a root cause we'll need some debugging information if possible:1) Debug logs as I asked above (preferably windows). Most likely debug logs from 1 PC would be enough, but to be on the safe side please collect from 2. One of them must show the high-CPU-behavior.2) Process dumps (also windows preferable). Could you please make a several process dumps? 3-5 sequential dumps of the same process would be great. Having the data above we have good chances to find the issue root cause. Thanks! Quote Link to comment Share on other sites More sharing options...
dl97ag Posted May 9, 2014 Report Share Posted May 9, 2014 I'm also seeing high CPU usage on Linux PowerPC (WD My Book Live). When btsync starts it pegs at ~100% CPU and eventually settles at ~30% and stays in that area forever, even after any syncing completed. Are logs still needed from a Windows machine for some reason, or can they be pulled from the Linux box? Quote Link to comment Share on other sites More sharing options...
jedie Posted May 12, 2014 Report Share Posted May 12, 2014 - some 400K+ files, almost 200GB volume, total tree structure has some 18K+ folders. There are about 50 synchronized folders (btsync entries) ranging from few MB up to 34 GB total volume.That's many, many files. Maybe to many for btsync? Somewhere was the info, that ~100Bytes RAM needed for every sync file, isn't it? I start bysync with nice and with ionice -n7 Quote Link to comment Share on other sites More sharing options...
RomanZ Posted May 13, 2014 Report Share Posted May 13, 2014 @dl97agYes, we'll need logs. If the Sync consumes a lot of CPU even when no other peers are online - then I'll need only your WD device logs, configured to full output. @jedieIt depends on a hardware. We've tested up to 1M files on modern desktops. One file\folder consumes around ~400 bytes in the memory, which brings us to 160Mb of memory for 400k files. It could be an issue for boxes like RPI, not sure about NAS storages - they vary greatly depending on the model. Quote Link to comment Share on other sites More sharing options...
dl97ag Posted May 18, 2014 Report Share Posted May 18, 2014 Thanks RomanZ - am submitting logs to Dale Cooper via ticket I'd opened via Zendesk. Thanks! Quote Link to comment Share on other sites More sharing options...
feanorknd Posted May 19, 2014 Report Share Posted May 19, 2014 Hello: I found same problem testing btsync... CPU constantly at 99%... I figured out it was the server daemon scanning my large /xxxx/media directories, with much hundred of files and folders... But I have a simply question... I did configure *media* at SyncExclude file... I saw the clients replicated this exclude and did not synchronized the /xxx/media directory... that is right.... If a subdirectory is excluded to be synchronized, is the server daemon proccessing its files or "watching out" possible changes inside?? This may explain that 100% CPU, because outside media folder there is no big number of files... If you exclude a subdirectory, daemon will ignore it and does not scan it for changes? Thanks so much! Quote Link to comment Share on other sites More sharing options...
RomanZ Posted May 21, 2014 Report Share Posted May 21, 2014 @feanorknd In Mac and Windows we do subscribe for notifications of folder changes for the folder you added to Bittorrent Sync and automatically get notifications for whole subtree - so our code has to explicitly drop ones that are ignored in SyncIgnore. In Linux it is different - we do subscribe for notifications separately for each folder and each subfolder - and do not subscribe to folders that are in SyncIgnore. So, for Linux Sync should not get any notifications if daemon does something in the ignored folder. Indexing thread in Sync also ignores and does not index ignored folders. Do you experience CPU load only when this daemon scanning something in sync folders? Quote Link to comment Share on other sites More sharing options...
feanorknd Posted May 22, 2014 Report Share Posted May 22, 2014 Hello: First of all, I did increase this one to exceed number of files synced... cat /proc/sys/fs/inotify/max_user_watches1048576 But..... have a look at this debug lines: ....[20140522 12:08:24.182] Lost peer for folder /home/WEBS 10F1AB73CF74B1D3CBEB4358E5F51CC3F83A2E04[20140522 12:08:45.097] [OnNotifyFileChange] /home/WEBS/MYDOMAIN.net/media/resize/285/245/1/media/usuarios/e/9/a3/e9a3c2c9e31cb45e3276c4351e9e12ad.jpg, source = NULL[20140522 12:10:24.011] [OnNotifyFileChange] /home/WEBS/MYDOMAIN.net/media/resize/670/260/1/media/eventos/e/1/36/e136997cffb16b0959d29527d6b0d655383b9a99.jpg, source = NULL.... and this is just while: cat /home/WEBS/.SyncIgnore.DS_Store.DS_Store?._*.Spotlight-V100.Trashesehthumbs.dbdesktop.iniThumbs.db# NO SINCRONIZAR ARCHIVOS MEDIA/*/media/**/media Yes... /home/WEBS/*/media is not being synchronized, and syncignore configuration is pretty good. I have looked for it at other peers and media is excluded... But if you have a look a debug logs at server, OnNotifyFileChange events for media appear........ this is for me a big problem, because /media should be forever excluded everywhere due to very very high number of files not to be synced never, and do not need / desire inotify events for files inside... it could be a intensive waste of CPU/RAM/kernel resources.... Also: There are not peers at all, server si running for hours, and CPU for btsync is now at this precise moment of 95% for minutes... have a look at debug log and see: [20140522 12:24:41.217] SyncFilesController [file updated]: Processing file /home/WEBS/MYDOMAIN.net/verificacion-cam.php 1400500417 251[20140522 12:24:41.217] SyncFilesController [file updated]: Processing file /home/WEBS/MYDOMAIN.net/ver-privado.php 1400500417 2286[20140522 12:24:41.217] SyncFilesController [file updated]: Processing file /home/WEBS/MYDOMAIN.net/validar_registro.php 1400500416 5680[20140522 12:24:41.225] SyncFilesController [file updated]: Processing file /home/WEBS/MYDOMAIN.net/unsubscribe.php 1400500416 222[20140522 12:24:41.225] SyncFilesController [file updated]: Processing file /home/WEBS/MYDOMAIN.net/redimensionar_pendientes.php 1400500416 707[20140522 12:24:41.225] SyncFilesController [file updated]: Processing file /home/WEBS/MYDOMAIN.net/redes-sociales.php 1400500416 607[20140522 12:24:41.225] SyncFilesController [file updated]: Processing file /home/WEBS/MYDOMAIN.net/posts-usuario.php 1400500416 1587 This files are not new at all, are not modified and were there since server start.... not really updated... Why btsync does explode CPU with lines like this "file updated" while they are not really updated at all? Thanks so much for your support/efforts with community! Regards @feanorknd In Mac and Windows we do subscribe for notifications of folder changes for the folder you added to Bittorrent Sync and automatically get notifications for whole subtree - so our code has to explicitly drop ones that are ignored in SyncIgnore. In Linux it is different - we do subscribe for notifications separately for each folder and each subfolder - and do not subscribe to folders that are in SyncIgnore. So, for Linux Sync should not get any notifications if daemon does something in the ignored folder. Indexing thread in Sync also ignores and does not index ignored folders. Do you experience CPU load only when this daemon scanning something in sync folders? Quote Link to comment Share on other sites More sharing options...
feanorknd Posted May 22, 2014 Report Share Posted May 22, 2014 If could ask.... Which way do you use inotifywait events?? I ask this because: root@ip-10-0-1-10:/home/WEBS# inotifywait -r -m -e modify,delete,create --exclude '*/media/*' /home/WEBS/Error in `exclude' regular expression.root@ip-10-0-1-10:/home/WEBS# inotifywait -r -m -e modify,delete,create --exclude './*/media/*' /home/WEBS/Setting up watches. Beware: since -r was given, this may take a while!Watches established. If use "*/media/*" exclude, btsync is doing right, but this exclude string is not supported by inotifywait tool.... I must use "./*/media/*" to get done. Maybe it is problem related? Also....... what events are you waiting for? Thanks. Quote Link to comment Share on other sites More sharing options...
feanorknd Posted May 26, 2014 Report Share Posted May 26, 2014 Someone? Quote Link to comment Share on other sites More sharing options...
sfmcnally Posted June 7, 2014 Report Share Posted June 7, 2014 I am also having this issue on 1.3.105. I'm syncing 4 folders and 200k files with a size of roughly 230GB. CPU load after 1 day is ~30% and the memory usage is way higher than the stated 400bytes per file. It's using about 1.5 gigs of memory after 1 day and eventually crashes. Quote Link to comment Share on other sites More sharing options...
sfmcnally Posted June 8, 2014 Report Share Posted June 8, 2014 It is easily reproducible. The longer the program runs, the more ram it uses until it finally crashes. Quote Link to comment Share on other sites More sharing options...
denis4inet Posted June 27, 2014 Report Share Posted June 27, 2014 @feanorknd,Please, make a coredump with the gcore and send compressed coredump us via syncapp.zendesk.com (leave a link to this topic). Thanks Quote Link to comment Share on other sites More sharing options...
denis4inet Posted August 28, 2014 Report Share Posted August 28, 2014 @feanorknd, try to install newer version of BitTorrent Sync. It contains some optimizations related to CPU usage. Quote Link to comment Share on other sites More sharing options...
KonstantinMiha Posted August 5, 2015 Report Share Posted August 5, 2015 (edited) I have the same problem on 2.1.1 version. I already test It on 3 different servers and get same result and GUI do not responding. I have more then 8.6 millions files (500 GB) for sync.Is any solutions for this problem? Edited August 5, 2015 by KonstantinMiha Quote Link to comment Share on other sites More sharing options...
GreatMarko Posted August 16, 2015 Report Share Posted August 16, 2015 @KonstantinMiha, Sync 2.1.2 contained a wealth of fixes, including one for "Large amount of folders causes high CPU usage...". Please update to the latest 2.1.x build (which at time of writing is 2.1.3) and see if this solves your issue! Quote Link to comment Share on other sites More sharing options...
KonstantinMiha Posted August 26, 2015 Report Share Posted August 26, 2015 (edited) @KonstantinMiha, Sync 2.1.2 contained a wealth of fixes, including one for "Large amount of folders causes high CPU usage...". Please update to the latest 2.1.x build (which at time of writing is 2.1.3) and see if this solves your issue! Yes, its begin work more stable but too slowly. I couldn't sync 60 Gb (1 millions files) for one week. Now its sync only 22 Gb. I frequently see status Indexing, on main server I see 40 Gb send to another device but on device I see 22 Gb and do not download anything. I have 5 days trial, and I do not know whether it is suitable for my task. Edited August 26, 2015 by KonstantinMiha Quote Link to comment Share on other sites More sharing options...
Helen Posted August 28, 2015 Report Share Posted August 28, 2015 KonstantinMiha,There is support ticket opened, let's proceed there, Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.