arcol

I just got an out of memory kill today on raspberry pi too. [ 188.777681] UpdateHalRAMask8188EUsb => mac_id:0, networkType:0x03, mask:0x00000fff[ 188.777681] ==> rssi_level:2, rate_bitmap:0x00000ff0[11027.178451] btsync invoked oom-killer: gfp_mask=0x200da, order=0, oom_adj=0, oom_score_adj=0[11027.178584] [<c0013b60>] (unwind_backtrace+0x0/0xf0) from [<c039c4e0>] (dump_header.isra.15+0x74/0x1a0)[11027.178678] [<c039c4e0>] (dump_header.isra.15+0x74/0x1a0) from [<c0092050>] (oom_kill_process+0x294/0x418)[11027.178719] [<c0092050>] (oom_kill_process+0x294/0x418) from [<c00925b8>] (out_of_memory+0x1cc/0x29c)[11027.178753] [<c00925b8>] (out_of_memory+0x1cc/0x29c) from [<c0095a84>] (__alloc_pages_nodemask+0x704/0x738)[11027.178788] [<c0095a84>] (__alloc_pages_nodemask+0x704/0x738) from [<c00ac0a8>] (handle_pte_fault+0x580/0x788)[11027.178817] [<c00ac0a8>] (handle_pte_fault+0x580/0x788) from [<c00ac348>] (handle_mm_fault+0x98/0xc8)[11027.178861] [<c00ac348>] (handle_mm_fault+0x98/0xc8) from [<c03a3048>] (do_page_fault+0x22c/0x3cc)[11027.178917] [<c03a3048>] (do_page_fault+0x22c/0x3cc) from [<c000832c>] (do_DataAbort+0x34/0x98)[11027.178947] [<c000832c>] (do_DataAbort+0x34/0x98) from [<c03a1a5c>] (__dabt_usr+0x3c/0x40)[11027.178961] Exception stack(0xd5df7fb0 to 0xd5df7ff8)[11027.178980] 7fa0: a40d9000 9fb98fdc 00318f54 5abed59a[11027.179001] 7fc0: f695d16e d45c3cf0 b3662217 b110e2c8 623a3034 37643366 a3dc000c b54f3f9c[11027.179018] 7fe0: 38306538 b54f3c2c ffffff84 b6edb2ec 20000010 ffffffff[11027.179027] Mem-info:[11027.179036] Normal per-cpu:[11027.179047] CPU 0: hi: 186, btch: 31 usd: 30[11027.179074] active_anon:42411 inactive_anon:42415 isolated_anon:0[11027.179074] active_file:179 inactive_file:226 isolated_file:0[11027.179074] unevictable:0 dirty:0 writeback:528 unstable:0[11027.179074] free:2026 slab_reclaimable:4403 slab_unreclaimable:1304[11027.179074] mapped:185 shmem:203 pagetables:657 bounce:0[11027.179127] Normal free:8104kB min:8192kB low:10240kB high:12288kB active_anon:169644kB inactive_anon:169660kB active_file:716kB inactive_file:904kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:390144kB mlocked:0kB dirty:0kB writeback:2112kB mapped:740kB shmem:812kB slab_reclaimable:17612kB slab_unreclaimable:5216kB kernel_stack:1528kB pagetables:2628kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:2338 all_unreclaimable? yes[11027.181560] lowmem_reserve[]: 0 0[11027.181594] Normal: 84*4kB 25*8kB 23*16kB 9*32kB 8*64kB 6*128kB 4*256kB 1*512kB 0*1024kB 0*2048kB 1*4096kB = 8104kB[11027.181639] 1438 total pagecache pages[11027.181650] 830 pages in swap cache[11027.181659] Swap cache stats: add 27670, delete 26840, find 607/796[11027.181667] Free swap = 0kB[11027.181673] Total swap = 102396kB[11027.209454] 98304 pages of RAM[11027.209481] 2403 free pages[11027.209488] 2376 reserved pages[11027.209494] 3425 slab pages[11027.209500] 264498 pages shared[11027.209507] 830 pages swap cached[11027.209517] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name[11027.209555] [ 154] 0 154 729 22 6 141 -1000 udevd[11027.209575] [ 1655] 0 1655 436 26 5 15 -1000 ifplugd[11027.209596] [ 1661] 0 1661 436 25 5 16 -1000 ifplugd[11027.209612] [ 1850] 0 1850 6992 98 7 40 0 rsyslogd[11027.209629] [ 1878] 65534 1878 503 32 5 11 0 thd[11027.209646] [ 1906] 0 1906 111641 79013 212 14929 0 btsync[11027.209663] [ 1918] 0 1918 948 35 6 23 0 cron[11027.209680] [ 1948] 104 1948 852 30 5 110 0 dbus-daemon[11027.209697] [ 1993] 0 1993 7673 26 9 118 0 lightdm[11027.209714] [ 2010] 0 2010 5181 442 11 611 0 Xorg[11027.209732] [ 2026] 102 2026 1377 77 6 60 0 ntpd[11027.209749] [ 2078] 0 2078 3830 26 7 132 0 lightdm[11027.209766] [ 2099] 1000 2099 438 16 4 17 0 xbmc-standalone[11027.209785] [ 2107] 1000 2107 438 16 4 20 0 xbmc[11027.209802] [ 2110] 0 2110 6630 37 10 261 0 console-kit-dae[11027.209820] [ 2177] 0 2177 5660 26 10 154 0 polkitd[11027.209837] [ 2189] 1000 2189 3198 45 6 40 0 lxsession[11027.209856] [ 2210] 1000 2210 880 19 4 47 0 ssh-agent[11027.209873] [ 2213] 1000 2213 842 18 5 57 0 dbus-launch[11027.209889] [ 2214] 1000 2214 827 18 4 77 0 dbus-daemon[11027.209907] [ 2220] 1000 2220 4074 76 12 648 0 openbox[11027.209924] [ 2222] 1000 2222 21093 338 27 885 0 lxpanel[11027.209942] [ 2223] 1000 2223 23799 192 29 676 0 pcmanfm[11027.209958] [ 2228] 1000 2228 6490 26 12 195 0 lxpolkit[11027.209977] [ 2232] 1000 2232 1613 54 7 68 0 menu-cached[11027.209994] [ 2234] 1000 2234 2175 24 8 92 0 gvfsd[11027.210010] [ 2248] 0 2248 935 18 6 32 0 getty[11027.210028] [ 2249] 0 2249 935 18 5 32 0 getty[11027.210045] [ 2250] 0 2250 935 19 6 31 0 getty[11027.210062] [ 2251] 0 2251 935 18 6 32 0 getty[11027.210079] [ 2252] 0 2252 935 18 6 32 0 getty[11027.210097] [ 2253] 0 2253 935 18 6 32 0 getty[11027.210115] [ 2254] 0 2254 515 19 5 30 0 getty[11027.210132] [ 2257] 1000 2257 67271 3174 75 4408 0 xbmc.bin[11027.210149] [ 2269] 0 2269 6596 25 9 164 0 upowerd[11027.210166] [ 2311] 1000 2311 8539 93 10 105 0 gvfs-gdu-volume[11027.210183] [ 2320] 0 2320 5681 59 8 99 0 udisks-daemon[11027.210201] [ 2321] 0 2321 1546 23 7 66 0 udisks-daemon[11027.210219] [ 2324] 1000 2324 4786 31 9 98 0 gvfs-afc-volume[11027.210237] [ 2327] 1000 2327 2228 24 8 113 0 gvfs-gphoto2-vo[11027.210255] [ 2367] 0 2367 436 41 5 11 -1000 ifplugd[11027.210273] [ 2384] 0 2384 1488 47 6 78 -1000 wpa_supplicant[11027.210293] [ 2445] 0 2445 569 19 5 24 -1000 wpa_cli[11027.210310] [ 2450] 0 2450 728 19 6 141 -1000 udevd[11027.210327] [ 2451] 0 2451 728 18 7 148 -1000 udevd[11027.210346] [ 2512] 0 2512 1223 13 5 429 -1000 dhclient[11027.210364] [ 2559] 0 2559 1552 33 6 92 -1000 sshd[11027.210380] [ 2589] 0 2589 2451 180 8 4 -1000 sshd[11027.210397] [ 2596] 1000 2596 2451 150 7 28 -1000 sshd[11027.210414] [ 2597] 1000 2597 1558 399 6 81 -1000 bash[11027.210430] Out of memory: Kill process 1906 (btsync) score 744 or sacrifice child[11027.210447] Killed process 1906 (btsync) total-vm:446564kB, anon-rss:315848kB, file-rss:204kBWhat I would *LOVE* to see, is an option to not permanently monitor a folder or its subfolder, but rather scan it once a week. The real culprit is git repositories, which are easily 40000+ files in the .git dir. I do think a raspberry pi should be capable of keeping a 1TB hard drive in sync. But in my case it fails in 35.9 GB (in 101747 files). Which is only 4% of the storage. An easy workaround would be to have a btsync/tmp dir, which is watched all the time. I move stuff there, it syncs with the raspberry pi, then I move it into its target dir within the btsync folder (on the host computer). The btsync on the host notice the file move, then instead of reuploading to the raspberry pi, it would simply send an event of the file move. Also on the raspberry pi, it only scans the whole filesystem only once in a week. What I think pretty lame right now, is the file moving or file renaming. If you upload a directory full of video files (like your holiday trip), and you are sorting your files, housekeep a little, and you rename the dir (or you move around). Then what happens on the other end? It gets reuploaded then the original file gets moved into .SyncArchive directory using twice the diskspace for 30 days! I happened to me, when I renamed a heavy dir it used an additional 8GB space running out of the 32GB pendrive. Now I bought a 1TB hard drive, to prevent it happening again in the future. But instead i get an out of memory error. Otherwise it would be perfect, but if it can't be trusted, then I don't know what to do. I do think the proposed workaround would be an awesome alternative, but I'm a simple user without any connection to the developers inside their castle. ps: the btsync version is Version 1.2.82 ( up to date ), on both the host computer (linux), and on the raspberry pi (arm). ps2: after three day of using btsync with 89.0 GB in 136342 files data, I can say for sure, that btsync is not suited for backing up full of your important data. It uses 13-43% cpu all the time, with a (not so) short burst of 100% cpu, which is usually 30-40sec duration every 10 mins or so. It is a dual core 4 year old laptop. I had this problem since last year (august) with the 24GB btsync directory, an order of magnitude less bad. But with the additional files the issue skyrocketed.

I'm really interested into this archiving solution, the best would be an official solution. If not, then I think the following scenario would be excellent: have a daemon on the device where the btsync is running. The daemon would treat the "/btsync/archive" directory special, and would operate like this: move a file outside of /btsync scope, ie. /archive, then create an identically named file inside the btsync/archive directory appending something in the end (.archive for example), and put relevant info inside: * original filename, * original file's date * checksum of the original file * size * time of archiving * some metadata, tags (future development, useful for searching), * comment field, where the user can add some useful texts, not affecting the operation at all. Example: /btsync/archive/family_video_april/mts001.avi -> /archive/family_video_april/mts001.avi -> /btsync/archive/family_video_april/mts001.avi.archive And one could restore the original file remotely, by renaming the archive file to restore. So the daemon would restore the file. Example: mv /btsync/archive/family_video_april/mts001.avi.archive /btsync/archive/family_video_april/mts001.avi.restore It would resolve the diskspace issue, while leaving you directory structure intact. One problem also needs to be solved, namely the problem of small files. Like a .git repository, where many files only a few bytes, so archiving would be too slow. In that case in the archive directory it should compress the many files into a single file, then archive that one. Or some other solution. But I would LOVE an official solution. The above method could be implemented inside btsync easily. Best, arcol