pleonepola Posted February 19, 2014 Report Share Posted February 19, 2014 Hallo, I installed BTSync 1.2.82 on a RaspberryPi B (512MiB version), I'm testing it on three folders, one of them is a folder of 20GiB and 100000 files. After having btsync running for a while I stopped it because it used 70% of the memory of RPi. What is the memory requirements for large number of files? Thanks, Pietro. Quote Link to comment Share on other sites More sharing options...
GreatMarko Posted February 19, 2014 Report Share Posted February 19, 2014 From the Unofficial FAQ: "memory usage increases by around 300-400 bytes per file being monitored by BitTorrent Sync. Therefore, if you're monitoring/syncing 1 million files, you will need around 300-400 MB of free memory" Quote Link to comment Share on other sites More sharing options...
arcol Posted February 19, 2014 Report Share Posted February 19, 2014 (edited) I just got an out of memory kill today on raspberry pi too.[ 188.777681] UpdateHalRAMask8188EUsb => mac_id:0, networkType:0x03, mask:0x00000fff[ 188.777681] ==> rssi_level:2, rate_bitmap:0x00000ff0[11027.178451] btsync invoked oom-killer: gfp_mask=0x200da, order=0, oom_adj=0, oom_score_adj=0[11027.178584] [<c0013b60>] (unwind_backtrace+0x0/0xf0) from [<c039c4e0>] (dump_header.isra.15+0x74/0x1a0)[11027.178678] [<c039c4e0>] (dump_header.isra.15+0x74/0x1a0) from [<c0092050>] (oom_kill_process+0x294/0x418)[11027.178719] [<c0092050>] (oom_kill_process+0x294/0x418) from [<c00925b8>] (out_of_memory+0x1cc/0x29c)[11027.178753] [<c00925b8>] (out_of_memory+0x1cc/0x29c) from [<c0095a84>] (__alloc_pages_nodemask+0x704/0x738)[11027.178788] [<c0095a84>] (__alloc_pages_nodemask+0x704/0x738) from [<c00ac0a8>] (handle_pte_fault+0x580/0x788)[11027.178817] [<c00ac0a8>] (handle_pte_fault+0x580/0x788) from [<c00ac348>] (handle_mm_fault+0x98/0xc8)[11027.178861] [<c00ac348>] (handle_mm_fault+0x98/0xc8) from [<c03a3048>] (do_page_fault+0x22c/0x3cc)[11027.178917] [<c03a3048>] (do_page_fault+0x22c/0x3cc) from [<c000832c>] (do_DataAbort+0x34/0x98)[11027.178947] [<c000832c>] (do_DataAbort+0x34/0x98) from [<c03a1a5c>] (__dabt_usr+0x3c/0x40)[11027.178961] Exception stack(0xd5df7fb0 to 0xd5df7ff8)[11027.178980] 7fa0: a40d9000 9fb98fdc 00318f54 5abed59a[11027.179001] 7fc0: f695d16e d45c3cf0 b3662217 b110e2c8 623a3034 37643366 a3dc000c b54f3f9c[11027.179018] 7fe0: 38306538 b54f3c2c ffffff84 b6edb2ec 20000010 ffffffff[11027.179027] Mem-info:[11027.179036] Normal per-cpu:[11027.179047] CPU 0: hi: 186, btch: 31 usd: 30[11027.179074] active_anon:42411 inactive_anon:42415 isolated_anon:0[11027.179074] active_file:179 inactive_file:226 isolated_file:0[11027.179074] unevictable:0 dirty:0 writeback:528 unstable:0[11027.179074] free:2026 slab_reclaimable:4403 slab_unreclaimable:1304[11027.179074] mapped:185 shmem:203 pagetables:657 bounce:0[11027.179127] Normal free:8104kB min:8192kB low:10240kB high:12288kB active_anon:169644kB inactive_anon:169660kB active_file:716kB inactive_file:904kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:390144kB mlocked:0kB dirty:0kB writeback:2112kB mapped:740kB shmem:812kB slab_reclaimable:17612kB slab_unreclaimable:5216kB kernel_stack:1528kB pagetables:2628kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:2338 all_unreclaimable? yes[11027.181560] lowmem_reserve[]: 0 0[11027.181594] Normal: 84*4kB 25*8kB 23*16kB 9*32kB 8*64kB 6*128kB 4*256kB 1*512kB 0*1024kB 0*2048kB 1*4096kB = 8104kB[11027.181639] 1438 total pagecache pages[11027.181650] 830 pages in swap cache[11027.181659] Swap cache stats: add 27670, delete 26840, find 607/796[11027.181667] Free swap = 0kB[11027.181673] Total swap = 102396kB[11027.209454] 98304 pages of RAM[11027.209481] 2403 free pages[11027.209488] 2376 reserved pages[11027.209494] 3425 slab pages[11027.209500] 264498 pages shared[11027.209507] 830 pages swap cached[11027.209517] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name[11027.209555] [ 154] 0 154 729 22 6 141 -1000 udevd[11027.209575] [ 1655] 0 1655 436 26 5 15 -1000 ifplugd[11027.209596] [ 1661] 0 1661 436 25 5 16 -1000 ifplugd[11027.209612] [ 1850] 0 1850 6992 98 7 40 0 rsyslogd[11027.209629] [ 1878] 65534 1878 503 32 5 11 0 thd[11027.209646] [ 1906] 0 1906 111641 79013 212 14929 0 btsync[11027.209663] [ 1918] 0 1918 948 35 6 23 0 cron[11027.209680] [ 1948] 104 1948 852 30 5 110 0 dbus-daemon[11027.209697] [ 1993] 0 1993 7673 26 9 118 0 lightdm[11027.209714] [ 2010] 0 2010 5181 442 11 611 0 Xorg[11027.209732] [ 2026] 102 2026 1377 77 6 60 0 ntpd[11027.209749] [ 2078] 0 2078 3830 26 7 132 0 lightdm[11027.209766] [ 2099] 1000 2099 438 16 4 17 0 xbmc-standalone[11027.209785] [ 2107] 1000 2107 438 16 4 20 0 xbmc[11027.209802] [ 2110] 0 2110 6630 37 10 261 0 console-kit-dae[11027.209820] [ 2177] 0 2177 5660 26 10 154 0 polkitd[11027.209837] [ 2189] 1000 2189 3198 45 6 40 0 lxsession[11027.209856] [ 2210] 1000 2210 880 19 4 47 0 ssh-agent[11027.209873] [ 2213] 1000 2213 842 18 5 57 0 dbus-launch[11027.209889] [ 2214] 1000 2214 827 18 4 77 0 dbus-daemon[11027.209907] [ 2220] 1000 2220 4074 76 12 648 0 openbox[11027.209924] [ 2222] 1000 2222 21093 338 27 885 0 lxpanel[11027.209942] [ 2223] 1000 2223 23799 192 29 676 0 pcmanfm[11027.209958] [ 2228] 1000 2228 6490 26 12 195 0 lxpolkit[11027.209977] [ 2232] 1000 2232 1613 54 7 68 0 menu-cached[11027.209994] [ 2234] 1000 2234 2175 24 8 92 0 gvfsd[11027.210010] [ 2248] 0 2248 935 18 6 32 0 getty[11027.210028] [ 2249] 0 2249 935 18 5 32 0 getty[11027.210045] [ 2250] 0 2250 935 19 6 31 0 getty[11027.210062] [ 2251] 0 2251 935 18 6 32 0 getty[11027.210079] [ 2252] 0 2252 935 18 6 32 0 getty[11027.210097] [ 2253] 0 2253 935 18 6 32 0 getty[11027.210115] [ 2254] 0 2254 515 19 5 30 0 getty[11027.210132] [ 2257] 1000 2257 67271 3174 75 4408 0 xbmc.bin[11027.210149] [ 2269] 0 2269 6596 25 9 164 0 upowerd[11027.210166] [ 2311] 1000 2311 8539 93 10 105 0 gvfs-gdu-volume[11027.210183] [ 2320] 0 2320 5681 59 8 99 0 udisks-daemon[11027.210201] [ 2321] 0 2321 1546 23 7 66 0 udisks-daemon[11027.210219] [ 2324] 1000 2324 4786 31 9 98 0 gvfs-afc-volume[11027.210237] [ 2327] 1000 2327 2228 24 8 113 0 gvfs-gphoto2-vo[11027.210255] [ 2367] 0 2367 436 41 5 11 -1000 ifplugd[11027.210273] [ 2384] 0 2384 1488 47 6 78 -1000 wpa_supplicant[11027.210293] [ 2445] 0 2445 569 19 5 24 -1000 wpa_cli[11027.210310] [ 2450] 0 2450 728 19 6 141 -1000 udevd[11027.210327] [ 2451] 0 2451 728 18 7 148 -1000 udevd[11027.210346] [ 2512] 0 2512 1223 13 5 429 -1000 dhclient[11027.210364] [ 2559] 0 2559 1552 33 6 92 -1000 sshd[11027.210380] [ 2589] 0 2589 2451 180 8 4 -1000 sshd[11027.210397] [ 2596] 1000 2596 2451 150 7 28 -1000 sshd[11027.210414] [ 2597] 1000 2597 1558 399 6 81 -1000 bash[11027.210430] Out of memory: Kill process 1906 (btsync) score 744 or sacrifice child[11027.210447] Killed process 1906 (btsync) total-vm:446564kB, anon-rss:315848kB, file-rss:204kBWhat I would *LOVE* to see, is an option to not permanently monitor a folder or its subfolder, but rather scan it once a week. The real culprit is git repositories, which are easily 40000+ files in the .git dir. I do think a raspberry pi should be capable of keeping a 1TB hard drive in sync. But in my case it fails in 35.9 GB (in 101747 files). Which is only 4% of the storage. An easy workaround would be to have a btsync/tmp dir, which is watched all the time.I move stuff there, it syncs with the raspberry pi, then I move it into its target dir within the btsync folder (on the host computer).The btsync on the host notice the file move, then instead of reuploading to the raspberry pi, it would simply send an event of the file move. Also on the raspberry pi, it only scans the whole filesystem only once in a week. What I think pretty lame right now, is the file moving or file renaming.If you upload a directory full of video files (like your holiday trip), and you are sorting your files, housekeep a little, and you rename the dir (or you move around). Then what happens on the other end? It gets reuploaded then the original file gets moved into .SyncArchive directory using twice the diskspace for 30 days! I happened to me, when I renamed a heavy dir it used an additional 8GB space running out of the 32GB pendrive.Now I bought a 1TB hard drive, to prevent it happening again in the future. But instead i get an out of memory error. Otherwise it would be perfect, but if it can't be trusted, then I don't know what to do. I do think the proposed workaround would be an awesome alternative, but I'm a simple user without any connection to the developers inside their castle. ps: the btsync version is Version 1.2.82 ( up to date ), on both the host computer (linux), and on the raspberry pi (arm). ps2: after three day of using btsync with 89.0 GB in 136342 files data, I can say for sure, that btsync is not suited for backing up full of your important data.It uses 13-43% cpu all the time, with a (not so) short burst of 100% cpu, which is usually 30-40sec duration every 10 mins or so. It is a dual core 4 year old laptop. I had this problem since last year (august) with the 24GB btsync directory, an order of magnitude less bad.But with the additional files the issue skyrocketed. Edited February 20, 2014 by arcol Quote Link to comment Share on other sites More sharing options...
pleonepola Posted February 20, 2014 Author Report Share Posted February 20, 2014 I launched BTSync yesterday and today I have found RPi completely stuck, it answered to ping but I could not login using ssh. I suppose BTSync simply ate all the ram. Today I have found that BTSync was trying to index all my files, not only the files contained in shared folders. I configured BTSync to share /home/ folder only, but now it is indexing /mnt too (in /mnt I have mounted 3 2TiB HDD). I suppose this is the reason why BTSync is using so much memory, it is trying to index 2 million files. Any suggestion for limiting BTSync to the shared folders? Thanks, Pietro. Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.