Btsync Memory Usage

[pleonepola]

Hallo, I installed BTSync 1.2.82 on a RaspberryPi B (512MiB version), I'm testing it on three folders, one of them is a folder of 20GiB and 100000 files. After having btsync running for a while I stopped it because it used 70% of the memory of RPi. What is the memory requirements for large number of files?

 

Thanks, Pietro.

[GreatMarko]

From the Unofficial FAQ: "memory usage increases by around 300-400 bytes per file being monitored by BitTorrent Sync. Therefore, if you're monitoring/syncing 1 million files, you will need around 300-400 MB of free memory"

[arcol]

I just got an out of memory kill today on raspberry pi too.

[  188.777681] UpdateHalRAMask8188EUsb => mac_id:0, networkType:0x03, mask:0x00000fff[  188.777681] 	 ==> rssi_level:2, rate_bitmap:0x00000ff0[11027.178451] btsync invoked oom-killer: gfp_mask=0x200da, order=0, oom_adj=0, oom_score_adj=0[11027.178584] [<c0013b60>] (unwind_backtrace+0x0/0xf0) from [<c039c4e0>] (dump_header.isra.15+0x74/0x1a0)[11027.178678] [<c039c4e0>] (dump_header.isra.15+0x74/0x1a0) from [<c0092050>] (oom_kill_process+0x294/0x418)[11027.178719] [<c0092050>] (oom_kill_process+0x294/0x418) from [<c00925b8>] (out_of_memory+0x1cc/0x29c)[11027.178753] [<c00925b8>] (out_of_memory+0x1cc/0x29c) from [<c0095a84>] (__alloc_pages_nodemask+0x704/0x738)[11027.178788] [<c0095a84>] (__alloc_pages_nodemask+0x704/0x738) from [<c00ac0a8>] (handle_pte_fault+0x580/0x788)[11027.178817] [<c00ac0a8>] (handle_pte_fault+0x580/0x788) from [<c00ac348>] (handle_mm_fault+0x98/0xc8)[11027.178861] [<c00ac348>] (handle_mm_fault+0x98/0xc8) from [<c03a3048>] (do_page_fault+0x22c/0x3cc)[11027.178917] [<c03a3048>] (do_page_fault+0x22c/0x3cc) from [<c000832c>] (do_DataAbort+0x34/0x98)[11027.178947] [<c000832c>] (do_DataAbort+0x34/0x98) from [<c03a1a5c>] (__dabt_usr+0x3c/0x40)[11027.178961] Exception stack(0xd5df7fb0 to 0xd5df7ff8)[11027.178980] 7fa0:                                     a40d9000 9fb98fdc 00318f54 5abed59a[11027.179001] 7fc0: f695d16e d45c3cf0 b3662217 b110e2c8 623a3034 37643366 a3dc000c b54f3f9c[11027.179018] 7fe0: 38306538 b54f3c2c ffffff84 b6edb2ec 20000010 ffffffff[11027.179027] Mem-info:[11027.179036] Normal per-cpu:[11027.179047] CPU    0: hi:  186, btch:  31 usd:  30[11027.179074] active_anon:42411 inactive_anon:42415 isolated_anon:0[11027.179074]  active_file:179 inactive_file:226 isolated_file:0[11027.179074]  unevictable:0 dirty:0 writeback:528 unstable:0[11027.179074]  free:2026 slab_reclaimable:4403 slab_unreclaimable:1304[11027.179074]  mapped:185 shmem:203 pagetables:657 bounce:0[11027.179127] Normal free:8104kB min:8192kB low:10240kB high:12288kB active_anon:169644kB inactive_anon:169660kB active_file:716kB inactive_file:904kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:390144kB mlocked:0kB dirty:0kB writeback:2112kB mapped:740kB shmem:812kB slab_reclaimable:17612kB slab_unreclaimable:5216kB kernel_stack:1528kB pagetables:2628kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:2338 all_unreclaimable? yes[11027.181560] lowmem_reserve[]: 0 0[11027.181594] Normal: 84*4kB 25*8kB 23*16kB 9*32kB 8*64kB 6*128kB 4*256kB 1*512kB 0*1024kB 0*2048kB 1*4096kB = 8104kB[11027.181639] 1438 total pagecache pages[11027.181650] 830 pages in swap cache[11027.181659] Swap cache stats: add 27670, delete 26840, find 607/796[11027.181667] Free swap  = 0kB[11027.181673] Total swap = 102396kB[11027.209454] 98304 pages of RAM[11027.209481] 2403 free pages[11027.209488] 2376 reserved pages[11027.209494] 3425 slab pages[11027.209500] 264498 pages shared[11027.209507] 830 pages swap cached[11027.209517] [ pid ]   uid  tgid total_vm      rss nr_ptes swapents oom_score_adj name[11027.209555] [  154]     0   154      729       22       6      141         -1000 udevd[11027.209575] [ 1655]     0  1655      436       26       5       15         -1000 ifplugd[11027.209596] [ 1661]     0  1661      436       25       5       16         -1000 ifplugd[11027.209612] [ 1850]     0  1850     6992       98       7       40             0 rsyslogd[11027.209629] [ 1878] 65534  1878      503       32       5       11             0 thd[11027.209646] [ 1906]     0  1906   111641    79013     212    14929             0 btsync[11027.209663] [ 1918]     0  1918      948       35       6       23             0 cron[11027.209680] [ 1948]   104  1948      852       30       5      110             0 dbus-daemon[11027.209697] [ 1993]     0  1993     7673       26       9      118             0 lightdm[11027.209714] [ 2010]     0  2010     5181      442      11      611             0 Xorg[11027.209732] [ 2026]   102  2026     1377       77       6       60             0 ntpd[11027.209749] [ 2078]     0  2078     3830       26       7      132             0 lightdm[11027.209766] [ 2099]  1000  2099      438       16       4       17             0 xbmc-standalone[11027.209785] [ 2107]  1000  2107      438       16       4       20             0 xbmc[11027.209802] [ 2110]     0  2110     6630       37      10      261             0 console-kit-dae[11027.209820] [ 2177]     0  2177     5660       26      10      154             0 polkitd[11027.209837] [ 2189]  1000  2189     3198       45       6       40             0 lxsession[11027.209856] [ 2210]  1000  2210      880       19       4       47             0 ssh-agent[11027.209873] [ 2213]  1000  2213      842       18       5       57             0 dbus-launch[11027.209889] [ 2214]  1000  2214      827       18       4       77             0 dbus-daemon[11027.209907] [ 2220]  1000  2220     4074       76      12      648             0 openbox[11027.209924] [ 2222]  1000  2222    21093      338      27      885             0 lxpanel[11027.209942] [ 2223]  1000  2223    23799      192      29      676             0 pcmanfm[11027.209958] [ 2228]  1000  2228     6490       26      12      195             0 lxpolkit[11027.209977] [ 2232]  1000  2232     1613       54       7       68             0 menu-cached[11027.209994] [ 2234]  1000  2234     2175       24       8       92             0 gvfsd[11027.210010] [ 2248]     0  2248      935       18       6       32             0 getty[11027.210028] [ 2249]     0  2249      935       18       5       32             0 getty[11027.210045] [ 2250]     0  2250      935       19       6       31             0 getty[11027.210062] [ 2251]     0  2251      935       18       6       32             0 getty[11027.210079] [ 2252]     0  2252      935       18       6       32             0 getty[11027.210097] [ 2253]     0  2253      935       18       6       32             0 getty[11027.210115] [ 2254]     0  2254      515       19       5       30             0 getty[11027.210132] [ 2257]  1000  2257    67271     3174      75     4408             0 xbmc.bin[11027.210149] [ 2269]     0  2269     6596       25       9      164             0 upowerd[11027.210166] [ 2311]  1000  2311     8539       93      10      105             0 gvfs-gdu-volume[11027.210183] [ 2320]     0  2320     5681       59       8       99             0 udisks-daemon[11027.210201] [ 2321]     0  2321     1546       23       7       66             0 udisks-daemon[11027.210219] [ 2324]  1000  2324     4786       31       9       98             0 gvfs-afc-volume[11027.210237] [ 2327]  1000  2327     2228       24       8      113             0 gvfs-gphoto2-vo[11027.210255] [ 2367]     0  2367      436       41       5       11         -1000 ifplugd[11027.210273] [ 2384]     0  2384     1488       47       6       78         -1000 wpa_supplicant[11027.210293] [ 2445]     0  2445      569       19       5       24         -1000 wpa_cli[11027.210310] [ 2450]     0  2450      728       19       6      141         -1000 udevd[11027.210327] [ 2451]     0  2451      728       18       7      148         -1000 udevd[11027.210346] [ 2512]     0  2512     1223       13       5      429         -1000 dhclient[11027.210364] [ 2559]     0  2559     1552       33       6       92         -1000 sshd[11027.210380] [ 2589]     0  2589     2451      180       8        4         -1000 sshd[11027.210397] [ 2596]  1000  2596     2451      150       7       28         -1000 sshd[11027.210414] [ 2597]  1000  2597     1558      399       6       81         -1000 bash[11027.210430] Out of memory: Kill process 1906 (btsync) score 744 or sacrifice child[11027.210447] Killed process 1906 (btsync) total-vm:446564kB, anon-rss:315848kB, file-rss:204kB

What I would *LOVE* to see, is an option to not permanently monitor a folder or its subfolder, but rather scan it once a week.

 

The real culprit is git repositories, which are easily 40000+ files in the .git dir.

 

I do think a raspberry pi should be capable of keeping a 1TB hard drive in sync. But in my case it fails in 35.9 GB (in 101747 files). Which is only 4% of the storage.

 

 

An easy workaround would be to have a btsync/tmp dir, which is watched all the time.

I move stuff there, it syncs with the raspberry pi, then I move it into its target dir within the btsync folder (on the host computer).

The btsync on the host notice the file move, then instead of reuploading to the raspberry pi, it would simply send an event of the file move.

 

Also on the raspberry pi, it only scans the whole filesystem only once in a week.

 

 

What I think pretty lame right now, is the file moving or file renaming.

If you upload a directory full of video files (like your holiday trip), and you are sorting your files, housekeep a little, and you rename the dir (or you move around).

 

Then what happens on the other end? It gets reuploaded then the original file gets moved into .SyncArchive directory using twice the diskspace for 30 days!

 

I happened to me, when I renamed a heavy dir it used an additional 8GB space running out of the 32GB pendrive.

Now I bought a 1TB hard drive, to prevent it happening again in the future. But instead i get an out of memory error.

 

Otherwise it would be perfect, but if it can't be trusted, then I don't know what to do.

 

I do think the proposed workaround would be an awesome alternative, but I'm a simple user without any connection to the developers inside their castle.

 

ps: the btsync version is Version 1.2.82 ( up to date ), on both the host computer (linux), and on the raspberry pi (arm).

 

ps2: after three day of using btsync with 89.0 GB in 136342 files data, I can say for sure, that btsync is not suited for backing up full of your important data.

It uses 13-43% cpu all the time, with a (not so) short burst of 100% cpu, which is usually 30-40sec duration every 10 mins or so.

 

It is a dual core 4 year old laptop.

 

I had this problem since last year (august) with the 24GB btsync directory,  an order of magnitude less bad.

But with the additional files the issue skyrocketed.

Edited February 20, 2014 by arcol

[pleonepola]

I launched BTSync yesterday and today I have found RPi completely stuck, it answered to ping but I could not login using ssh. I suppose BTSync simply ate all the ram.

 

Today I have found that BTSync was trying to index all my files, not only the files contained in shared folders. I configured BTSync to share /home/ folder only, but now it is indexing /mnt too (in /mnt I have mounted 3 2TiB HDD). I suppose this is the reason why BTSync is using so much memory, it is trying to index 2 million files. Any suggestion for limiting BTSync to the shared folders?

 

Thanks, Pietro.