jvavruska

Btsync "accumulating" Cpu Load

Recommended Posts

Hello,

 

we deployed BTSync 1.2.91 (i386 linux version) on 4 linux machines, 1 CentOS and 3 NAS TS-269 (linux-based operating system QTS).

 

I noticed that on various occasions CPU load as dislayed by the top utility shows btsync running at the top with high cpu loads over 85%, often peaking to 99.9%. Some machines reached a consistent 99.9% level after about a day.

 

When the porcessed reached 99.9%, GUI responds slowly, some synced folders appear not to be synchronized (at least in GUI) and there seem to be hanging (infinitely pending) transfers.

 

I noticed similar behavior with version 1.2.82, where CentOS process often crashed with segfault and QTS machines as if "accumulated" load until 99.9% when they stopped working, although the process itself did not crash.

 

Temporary solution implemented is periodical restart of btsync every two hours. After restart, btsync starts at low CPU loads (1 to 10%) and is getting busier as time passes. Before it becomes completely dead it is gracefully restarted by a cron job.

 

Could this be answer to some of you guys who reported insanely slow download speeds?

 

Regards,

 

Jindrich Vavruska

Share this post


Link to post
Share on other sites

jvavruska,

 

Couple of questions to clarify situation:

- What happens next? Does CPU load ever drops or it is stays at 99%? 

- How many files / folders do you have? What is the summary volume?

- What are the CPUs on your computers?

- Does your CentOS suffers the same issue as NAS devices?

- could you please make a debug logs for couple of peers so we can take a glance what happens inside during 100% CPU usage?

 

Thanks!

Share this post


Link to post
Share on other sites

Hello,

 

from what I could observe, once they reach this level (after a day) it stays high and blocks any transfers.

 

 - Since we have it on production machines and do not have better alternative, I put a crontab restart every two hours. After restart, btsync is temporarily busy as it loads all folders anew, but after few minutes, web gui starts working and the load drops to some 18-30%. Peaks occur occasionally, probably during reindexing, but stay temporary.

 

- some 400K+ files, almost 200GB volume, total tree structure has some 18K+ folders. There are about 50 synchronized folders (btsync entries) ranging from few MB up to 34 GB total volume. (We are in a planning phase to put most of it out of sync to a permanent archive, but this is our current load)

 

- CPU on QNAP is dual core Pentium (1.9 GHz IIRC), CentOS machine is a different flavor but also a dual core Pentium.

 

I will have a look for the logs. I will switch off the restarts on weekend and try to collect more information. What we have now would probably not be representative because I installed a regular 2-hour restart in all units.

 

In the meantime we switched off CentOS (it was receiving uploaded files from web server but we redirected the upload directly to the QNAP NAS) so I am afraid it will not provide any more data.

 

Regards,

 

Jindra

Share this post


Link to post
Share on other sites

Jindra,

 

Thanks for details. To debug this issue and find a root cause we'll need some debugging information if possible:

1) Debug logs as I asked above (preferably windows). Most likely debug logs from 1 PC would be enough, but to be on the safe side please collect from 2. One of them must show the high-CPU-behavior.

2) Process dumps (also windows preferable). Could you please make a several process dumps? 3-5 sequential dumps of the same process would be great.

 

Having the data above we have good chances to find the issue root cause.

 

Thanks!

Share this post


Link to post
Share on other sites

I'm also seeing high CPU usage on Linux PowerPC (WD My Book Live).  When btsync starts it pegs at ~100% CPU and eventually settles at ~30% and stays in that area forever, even after any syncing completed.  Are logs still needed from a Windows machine for some reason, or can they be pulled from the Linux box?

Share this post


Link to post
Share on other sites
- some 400K+ files, almost 200GB volume, total tree structure has some 18K+ folders. There are about 50 synchronized folders (btsync entries) ranging from few MB up to 34 GB total volume.

That's many, many files. Maybe to many for btsync? Somewhere was the info, that ~100Bytes RAM needed for every sync file, isn't it?

 

I start bysync with nice and with ionice -n7

Share this post


Link to post
Share on other sites

@dl97ag

Yes, we'll need logs. If the Sync consumes a lot of CPU even when no other peers are online - then I'll need only your WD device logs, configured to full output.

 

@jedie

It depends on a hardware. We've tested up to 1M files on modern desktops. One file\folder consumes around ~400 bytes in the memory, which brings us to 160Mb of memory for 400k files. It could be an issue for boxes like RPI, not sure about NAS storages - they vary greatly depending on the model.

Share this post


Link to post
Share on other sites

Hello:

 

I found same problem testing btsync... CPU constantly at 99%...

 

I figured out it was the server daemon scanning my large /xxxx/media directories, with much hundred of files and folders...

 

But I have a simply question... I did configure *media* at SyncExclude file... I saw the clients replicated this exclude and did not synchronized the /xxx/media directory... that is right....

 

If a subdirectory is excluded to be synchronized, is the server daemon proccessing its files or "watching out" possible changes inside?? This may explain that 100% CPU, because outside media folder there is no big number of files...

 

If you exclude a subdirectory, daemon will ignore it and does not scan it for changes?

 

Thanks so much!

Share this post


Link to post
Share on other sites

@feanorknd

 

In Mac and Windows we do subscribe for notifications of folder changes for the folder you added to Bittorrent Sync and automatically get notifications for whole subtree - so our code has to explicitly drop ones that are ignored in SyncIgnore.

 

In Linux it is different - we do subscribe for notifications separately for each folder and each subfolder - and do not subscribe to folders that are in SyncIgnore. So, for Linux Sync should not get any notifications if daemon does something in the ignored folder.

 

Indexing thread in Sync also ignores and does not index ignored folders.

 

Do you experience CPU load only when this daemon scanning something in sync folders?

Share this post


Link to post
Share on other sites

Hello:

 

First of all, I did increase this one to exceed number of files synced...

 

cat /proc/sys/fs/inotify/max_user_watches
1048576

 

But..... have a look at this debug lines:

 

 

....

[20140522 12:08:24.182] Lost peer for folder /home/WEBS 10F1AB73CF74B1D3CBEB4358E5F51CC3F83A2E04
[20140522 12:08:45.097] [OnNotifyFileChange] /home/WEBS/MYDOMAIN.net/media/resize/285/245/1/media/usuarios/e/9/a3/e9a3c2c9e31cb45e3276c4351e9e12ad.jpg, source = NULL
[20140522 12:10:24.011] [OnNotifyFileChange] /home/WEBS/MYDOMAIN.net/media/resize/670/260/1/media/eventos/e/1/36/e136997cffb16b0959d29527d6b0d655383b9a99.jpg, source = NULL

....

 

and this is just while:

 

 

cat /home/WEBS/.SyncIgnore
.DS_Store
.DS_Store?
._*
.Spotlight-V100
.Trashes
ehthumbs.db
desktop.ini
Thumbs.db

# NO SINCRONIZAR ARCHIVOS MEDIA/
*/media/*
*/media
 

 

Yes... /home/WEBS/*/media is not being synchronized, and syncignore configuration is pretty good. I have looked for it at other peers and media is excluded...

 

But if you have a look a debug logs at server, OnNotifyFileChange events for media appear........ this is for me a big problem, because /media should be forever excluded everywhere due to very very high number of files not to be synced never, and do not need / desire inotify events for files inside... it could be a intensive waste of CPU/RAM/kernel resources....

 

Also:

 

There are not peers at all, server si running for hours, and CPU for btsync is now at this precise moment of 95% for minutes... have a look at debug log and see:

 

 

[20140522 12:24:41.217] SyncFilesController [file updated]: Processing file /home/WEBS/MYDOMAIN.net/verificacion-cam.php 1400500417 251
[20140522 12:24:41.217] SyncFilesController [file updated]: Processing file /home/WEBS/MYDOMAIN.net/ver-privado.php 1400500417 2286
[20140522 12:24:41.217] SyncFilesController [file updated]: Processing file /home/WEBS/MYDOMAIN.net/validar_registro.php 1400500416 5680
[20140522 12:24:41.225] SyncFilesController [file updated]: Processing file /home/WEBS/MYDOMAIN.net/unsubscribe.php 1400500416 222
[20140522 12:24:41.225] SyncFilesController [file updated]: Processing file /home/WEBS/MYDOMAIN.net/redimensionar_pendientes.php 1400500416 707
[20140522 12:24:41.225] SyncFilesController [file updated]: Processing file /home/WEBS/MYDOMAIN.net/redes-sociales.php 1400500416 607
[20140522 12:24:41.225] SyncFilesController [file updated]: Processing file /home/WEBS/MYDOMAIN.net/posts-usuario.php 1400500416 1587
 

 

 

This files are not new at all, are not modified and were there since server start.... not really updated...

 

Why btsync does explode CPU with lines like this "file updated" while they are not really updated at all?

 

Thanks so much for your support/efforts with community!

 

Regards

 

 

 

 

@feanorknd

 

In Mac and Windows we do subscribe for notifications of folder changes for the folder you added to Bittorrent Sync and automatically get notifications for whole subtree - so our code has to explicitly drop ones that are ignored in SyncIgnore.

 

In Linux it is different - we do subscribe for notifications separately for each folder and each subfolder - and do not subscribe to folders that are in SyncIgnore. So, for Linux Sync should not get any notifications if daemon does something in the ignored folder.

 

Indexing thread in Sync also ignores and does not index ignored folders.

 

Do you experience CPU load only when this daemon scanning something in sync folders?

Share this post


Link to post
Share on other sites

If could ask....

 

Which way do you use inotifywait events??

 

I ask this because:

 

 

root@ip-10-0-1-10:/home/WEBS# inotifywait -r -m -e modify,delete,create --exclude '*/media/*' /home/WEBS/
Error in `exclude' regular expression.

root@ip-10-0-1-10:/home/WEBS# inotifywait -r -m -e modify,delete,create --exclude './*/media/*' /home/WEBS/
Setting up watches.  Beware: since -r was given, this may take a while!
Watches established.

 

If use "*/media/*" exclude, btsync is doing right, but this exclude string is not supported by inotifywait tool.... I must use "./*/media/*" to get done.

 

Maybe it is problem related? Also....... what events are you waiting for?

 

Thanks.

Share this post


Link to post
Share on other sites

I am also having this issue on 1.3.105. I'm syncing 4 folders and 200k files with a size of roughly 230GB. CPU load after 1 day is ~30% and the memory usage is way higher than the stated 400bytes per file. It's using about 1.5 gigs of memory after 1 day and eventually crashes.

Share this post


Link to post
Share on other sites

I have the same problem on 2.1.1 version. I already test It on 3 different servers and get same result and GUI do not responding. I have more then 8.6 millions files (500 GB) for sync.

Is any solutions for this problem?

Edited by KonstantinMiha

Share this post


Link to post
Share on other sites

@KonstantinMiha, Sync 2.1.2 contained a wealth of fixes, including one for "Large amount of folders causes high CPU usage...".

 

Please update to the latest 2.1.x build (which at time of writing is 2.1.3) and see if this solves your issue!

Share this post


Link to post
Share on other sites

@KonstantinMiha, Sync 2.1.2 contained a wealth of fixes, including one for "Large amount of folders causes high CPU usage...".

 

Please update to the latest 2.1.x build (which at time of writing is 2.1.3) and see if this solves your issue!

 

Yes, its begin work more stable but too slowly. I couldn't sync 60 Gb (1 millions files) for one week. Now its sync only 22 Gb. I frequently see status Indexing, on main server I see 40 Gb send to another device but on device I see 22 Gb and do not download anything. I have 5 days trial, and I do not know whether it is suitable for my task.

Edited by KonstantinMiha

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.