very big ram footprint

[operationally]

I searched on this forum, and i do  see this question came up before, but I still think this worth mentioning.

I am not here to question why it would current implementation takes that much memory.

My Sync Folder contains about 56632 files, and folders are:  575GB in size.  Running on Ubuntu 16.04 and rslsync uses about 9.1 GB RAM. (CPU fluctuates between 5-8% during running)

For a comparison, i also use Dropbox Pro, I have 26602 files and 79GB in size. Dropbox uses about  329MB RAM. (0 % CPU when idle)

In this case, since server is powerful enough that I don't care about the RAM and CPU. But on a less powerful machine, it is also very resource intensive.

 

I have a laptop running windows, upon start, Sync eats 5 GB RAM, (this is given the fact only one folder (less than 200MB) is in full sync mode, most others are selective sync and with very few files synced).

When Sync finishes initialization, it stables at 2GB RAM. (At this point I have effectively 2.25GB data synced,  Folder properties shown 113k Files).  And CPU usage flucuates around 20-50% (this is a much weaker processor i7-4510U)

This is a LOT of resources for such small number data being synced in selective sync mode. 

 

Again I am not here asking why it is so, or I am trying to say this is a bug. I think this huge resources footprint will greatly limit Sync's user base/ usage. And it simply cannot compete with Dropbox as a standard family sync tool. (All your photos, 4k videos etc, old backups of your OSes). 

 

 

[xfelix]

i agree.

i have to set ram limit for the docker container that runs ResilioSync otherwise it will eat up all the host RAM.

On a resources limited NAS, it will take ages to open the ResilioSync webui because of not enough RAM.

If dev can find ways to improve the memory efficiency, that will be great.

[Frank Maier]

Can't agree here.

Using it on Windows 10 on two computers, one with 8GB RAM, Desktop, Core i5, 2500, the other a with 4GB RAM Core 2 Duo U7600 (so pretty old and slow).
On both, Resilio Sync consumes around 170MB RAM and barely uses the CPU.

On my NAS (QNAP TS-451, 16GB RAM) it consumes 200MB RAM when idle.

I sync around 200GB and 140k files.

 

If the CPU is used by about 50%, are you sure, that Sync is idle and not indexing? 
https://help.resilio.com/hc/en-us/articles/205458185-Setting-how-often-Sync-should-check-for-file-changes-

[Helen]

On 7/9/2017 at 10:14 PM, operationally said:

Again I am not here asking why it is so

And still I'll explain it, which may help to find a solution. This has been already explained multiple times on forum, but here it goes. 
RAM is taken by database files. Database keeps record of each entry in the sync share - currently present files and the ones that have ever been in the share. Entry - is the file itself, and also each subdir level in its path. Each entry takes up around 2KB of RAM. The bigger folder tree you have, the more nested folders, the more file changes/deletions, the bigger DB grows, as Sync needs to remember all those entries. 

SO you can remove the share from Sync on all peers and add it back, so that at least old entries are cleared and only current ones are recorded. 

I'm not claiming it's a perfect way to work, and ways to optimise it are being worked on, but it's not a one hour fix.

RAM jumps from 5Gb to 2, apparently, cause of folder rescanning that is performed on sync start. which sync version you use? 

 

[operationally]

On 7/11/2017 at 10:41 AM, Helen said:

RAM is taken by database files. Database keeps record of each entry in the sync share - currently present files and the ones that have ever been in the share. Entry - is the file itself, and also each subdir level in its path. Each entry takes up around 2KB of RAM. The bigger folder tree you have, the more nested folders, the more file changes/deletions, the bigger DB grows, as Sync needs to remember all those entries.

Doing a disconnect for one of my largest folder reduced RAM to 4.9GB. But reconnect seems refuse to allow me to sync the same folder and it is frustrating, if I am not the owner of the folder do I need to re-sync everything?

And If it is changes/deletions that is in RAM I think that is a not so good design:

a. these can easily put into a local db like sqlite and query upon request

b. with the UI there is no way to restore a copy (albeit there is a archive Dir)  so no reason to keep in RAM, unless this is a planned feature?

Of course I don't know enough design details here to make sound judgement, so please enlighten me.

 

On 7/10/2017 at 2:57 AM, Frank Maier said:

Can't agree here.

Using it on Windows 10 on two computers, one with 8GB RAM, Desktop, Core i5, 2500, the other a with 4GB RAM Core 2 Duo U7600 (so pretty old and slow).
On both, Resilio Sync consumes around 170MB RAM and barely uses the CPU.

On my NAS (QNAP TS-451, 16GB RAM) it consumes 200MB RAM when idle.

I sync around 200GB and 140k files.

 

If the CPU is used by about 50%, are you sure, that Sync is idle and not indexing? 
https://help.resilio.com/hc/en-us/articles/205458185-Setting-how-often-Sync-should-check-for-file-changes-

Well tweaking check for file changes is considered indexing? (I am assuming no file change means no index required).

Also I don't know why checking if file changed requires lots of CPU or memory? This is not done via something like inotify on Linux?

Also do you use full sync or selective sync, i am very surprised by your memory / cpu usage. Because I have limited usage on Windows but huge resource hog.

[Helen]

On 7/15/2017 at 3:44 PM, operationally said:

do I need to re-sync everything?

no, you can use the folders with already synced files. Take the link to share from share Owner. 

Sync keeps db in RAM to reduce disk IO operations. 

On 7/15/2017 at 3:44 PM, operationally said:

This is not done via something like inotify on Linux?

it is.

[Frank Maier]

On 15.7.2017 at 2:44 PM, operationally said:

Well tweaking check for file changes is considered indexing? (I am assuming no file change means no index required).

Also I don't know why checking if file changed requires lots of CPU or memory? This is not done via something like inotify on Linux?

Also do you use full sync or selective sync, i am very surprised by your memory / cpu usage. Because I have limited usage on Windows but huge resource hog.

No, Sync checks for file changes the whole time. (that's not what I meant with indexing) But once in a while Sync scans all files again to see if a file got changed but was missed by the auomatic detection, that's the rescan interval. The rescan takes pretty long on my laptop, causes high CPU and HDD usage. Thus I increased the time to once every two days, had no issues yet. But it's possible that a file change might get detected in two days at the earliest.

I own the pro license, tried selective sync but don't use selective sync any longer because it does not fit my use case.

[operationally]

14 hours ago, Helen said:

no, you can use the folders with already synced files. Take the link to share from share Owner. 

Sync keeps db in RAM to reduce disk IO operations. 

it is.

Thanks, after reconnecting it does help reduce RAM usage.

If inotify is used, on a 64 bit machine, millions of files takes about 1GB RAM, whereas my usage must have been one of the worst case then?

 

8 hours ago, Frank Maier said:

No, Sync checks for file changes the whole time. (that's not what I meant with indexing) But once in a while Sync scans all files again to see if a file got changed but was missed by the auomatic detection, that's the rescan interval. The rescan takes pretty long on my laptop, causes high CPU and HDD usage. Thus I increased the time to once every two days, had no issues yet. But it's possible that a file change might get detected in two days at the earliest.

I own the pro license, tried selective sync but don't use selective sync any longer because it does not fit my use case.

Thanks for your reply.

 

Just so that everyone else know, i updated Linux client to 2.5.6 (1043), so far it does looks like the RAM usage is drastically down, after reconnect as suggested by @Helen 

RAM went down to 5GB, now RAM is at 901MB (still growing though), I will follow up on this.

But thanks for all the community support and replies. 

[hotbelgo]

I sync directories of code and added `node_modules` to my IgnoreList. That still leaves quite a few files, but sync is using 840Mb on an 8Gb machine. Anything else I can do?

[jpap]

I'm seeing massive RAM usage on macOS.  On a MacBook 12" with 8GB of RAM, I'm seeing Resilio Sync use 1.31GB of memory for a two-folder share with 1,654,405 files.  That's aside from the massive disk usage for the SQLite files.

I'm sure there is a lot of room for improvement here.  I'm debating whether Resilio Sync is worth the overhead compared to a periodic rsync in my case.

I wonder if some of the simple suggestions on StackOverflow for limiting SQLite memory usage could be helpful to the devs.

[operationally]

@jpap I think the ram usage is your case is reasonable for 1.6m files compared to my original use case/problem.  In fact for this many files on Linux just storing watching events will take 820+MB RAM.  

As this tool is closed source, so cannot comment on what other information the software is storing and saving.

[jpap]

@operationally Linux is very different to macOS in that there is very little filesystem overhead to monitor large number of files in a directory tree: you don't have to have an inotify object for each file/folder being monitored. In this case, the 1.31GB overhead is almost completely application-specific memory overhead, which seems excessive to me.