Sync fails for any folder that contains "node_modules" subfolder


Mason

Recommended Posts

I've been using Sync for maybe 6 months, evaluating it as a replacement for Dropbox and Google Drive.

This has been promising, and a nice feature of Sync has been its support for symbolic links. This had let me sync some types of things that neither of the other two could handle, including Node JS project directories. However, this recently broke. I think it broke in version 2.5.6, but it could have been a previous version — I didn't have time to debug the failing sync until recently.

The symptoms are similar to other some posts here (e.g. this one): sync never finishes, or gets hung at 99% with high CPU usage, or hangs forever at "Loading list of files" (the tooltip next to the forever-spinning loading icon next to the folder in the folder list). Some debug logs of the failed syncs are also similar (see below).

To debug, I created new sync folders and tried to sync them, and I found out which folder causes the problem: the node_modules folder inside one of my projects.

The node_modules folder is notorious for being problematic; it typically contains tons of symlinks, many of which may be broken, or point to different locations on one machine than another. However, that is why I thought it was a good stress test for a new sync system (Resilio Sync being fairly new to me). And indeed, for several months Sync did a great job.

I was not able to find a solution to make this folder sync successfully. And, I have reproduced the issue by setting up a new synced folder containing only the problematic Node JS project (once I figured out which subfolder was making the sync fail). Removing the problematic folder causes sync to work again.

However, I have been able to work around the issue by adding "node_modules" to the ".sync/IgnoreList" file at the root of the synced folder on all of my machines (for now, all Macs running the latest release of macOS). 

But it seems to me it should not be necessary to do this. If Resilio can sync symbolic links, including symbolic links that are broken, then shouldn't it be able to sync this folder? I would expect it to maybe make syncing slow, since there are (in this particular case) 55,843 files and 296 symlinks in the "node_modules" subfolder. I might even expect CPU usage to spike during the sync.

But I wouldn't expect sync to fail outright (or hang "forever", meaning 72 hours in my testing without showing any sign of ever finishing). And, Sync was previously able to sync my ~10GB test folder which contained this same node project.

The debug logs were filled with lines like this, which reference files inside the problematic "node_modules" subfolder. 

[20170803 20:53:04.984] JOURNAL[63B5]: Setup entry job "VerifyFileSignatureMessage" for path "", next state is "SKIP", queue size 0 11 4591 2139 
[20170803 20:53:04.984] FC[63B5]: Failed to get node for parent(SORACOM/user-console/bower_components/angular-strap/node_modules/undertaker-lib-tasks/node_modules/browser-sync/node_modules/socket.io/node_modules/engine.io/node_modules/ws/node_modules/utf-8-validate/build/Release) of file SORACOM/user-console/bower_components/angular-strap/node_modules/undertaker-lib-tasks/node_modules/browser-sync/node_modules/socket.io/node_modules/engine.io/node_modules/ws/node_modules/utf-8-validate/build/Release/obj.target
[20170803 20:53:04.984] FC[63B5]: Max count of remote files processing achieved. Requeueing VerifyFileSignatureMessage job.
[20170803 20:53:04.984] JOURNAL[63B5]: Setup entry job "VerifyFileSignatureMessage" for path "", next state is "SKIP", queue size 0 11 4591 2139 
[20170803 20:53:04.984] FC[63B5]: Failed to get node for parent(SORACOM/user-console/bower_components/angular-strap/node_modules/undertaker-lib-tasks/node_modules/browser-sync/node_modules/socket.io/node_modules/socket.io-client/node_modules/engine.io-client/node_modules/ws/node_modules/bufferutil/node_modules/nan) of file SORACOM/user-console/bower_components/angular-strap/node_modules/undertaker-lib-tasks/node_modules/browser-sync/node_modules/socket.io/node_modules/socket.io-client/node_modules/engine.io-client/node_modules/ws/node_modules/bufferutil/node_modules/nan/nan_persistent_12_inl.h
[20170803 20:53:04.984] FC[63B5]: Max count of remote files processing achieved. Requeueing VerifyFileSignatureMessage job.
[20170803 20:53:04.984] JOURNAL[63B5]: Setup entry job "VerifyFileSignatureMessage" for path "", next state is "SKIP", queue size 0 11 4591 2139 
[20170803 20:53:04.984] FC[63B5]: Failed to get node for parent(SORACOM/user-console/bower_components/angular-strap/node_modules/ng-factory/node_modules/undertaker/node_modules/es6-weak-map/node_modules/es5-ext/object) of file SORACOM/user-console/bower_components/angular-strap/node_modules/ng-factory/node_modules/undertaker/node_modules/es6-weak-map/node_modules/es5-ext/object/safe-traverse.js
[20170803 20:53:04.984] FC[63B5]: Max count of remote files processing achieved. Requeueing VerifyFileSignatureMessage job.

As mentioned, I can work around this issue, but it has somewhat decreased my confidence in Resilio Sync in general — if I hit issues like this with only 20GB or so in my Resilio Sync synced folder, it it really going to be able to handle moving all the contents of my 800GB Dropbox folder?

Should I change my expectations of the kinds and quantity of files I expect Sync be able to handle? Should I break up one big synced folder into multiple folders, even though that increases management overhead?

Does anybody have any other advice for workaround/solution?

Finally, if this is a bug, but one that is hard to reproduce, I don't think my node_modules folder contains anything confidential; I would be happy to provide a copy of it if that help debug this problem. It's about 150MB compressed.

 

Link to comment
Share on other sites

  • 4 weeks later...

I was able to reproduce this exact problem. Instead of a node_modules folder, I have a *lot* of files in several shares. The repeatable test case was to remove the folder from Resilio Sync, wipe the folder from the drive, create a new one, then sync again. Repeatedly failed. I'm using the Linux (amd64) version, both 2.5.6-1 and 2.5.7-1. Didn't change when I removed the folder and added it again. Downgrading to 2.5.5-1 worked for me. The package I used is here: 

http://linux-packages.resilio.com/resilio-sync/deb/pool/non-free/r/resilio-sync/resilio-sync_2.5.5-1_amd64.deb

I wouldn't give up with this software. For instance, one of my many large folders is 77 gigs consumed by 72k files. I've got bigger ones that I sync elsewhere as well and ones with far more files. One of the shares is nearly 200 gigs, another has 200k files. Typically there's no snag. I believe that there's a bug that was introduced with version 2.5.6.

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.