Lloyd Summers

Resilio Sync - Deleted Millions of Files (doing First Sync)

Recommended Posts

Hi All,

History - You can skip this if you like

I came to Resilio from Dropbox as they informed me will not provide support to clients with over 300,000 files, resulting in the application crashing on 3 of our computers for the last few years.

On the advice of others, I bought Resilio without being able to do much testing (with over 500GB of data, and 2.5 million files, the 5GB limited version does not tell me anything of value).  So, I did the initial setup, copied all the folders to where I needed, got all the sync's running and folders connected.  I did at first try to use SymLinks to make it easier to split the data, which obviously just didn't do anything.

Anyway, I won't harp on the terrible memory use yet - but I find it is partially synced and already using ~2 to 4GB of RAM - I will roll with that for now (all my machines have 32GB minimum) and it is not my chief concern but I do feel like a DBA should be consulted, you do not need to have the entire folder DB loaded in RAM all the time...  However, my current pressing issue is that about a week in now and I started noticing about 2 days ago that I had files missing...  I took a look today, and discovered that it wasn't just some files, but had become approximately 75% of all of my files were deleted by Resilio Sync. 

In total, I am missing around ~350GB (millions of files) - however, the folders and subfolders are intact.  Just the files were deleted. 

Question #1 - I can't for the life of me find a good article on how to properly restore files. 

Is there an article on proper restore methods using Resilio?  I did find some of the missing files in .sync/Archive. 

  • So do I now need to look in that folder on all of my computers, manually copy those files out, back into the Resilio Sync folder? 
  • Do I then delete the Archive copies? Or is it smart enough to do this on its own?  I suspect, it is not.
  • How do I stop it from permanently deleting files >1GB?  During this process, it does appear to have permanently deleted several large files.

Question #2 - Should I, or Can I, trust Resilio?

I ask this, not to be rude, but out of knowing little about this, as I purchased the program on the recommendation of others.  I honestly want to know, is this an actual commercial product, or a funky proof of concept, and not ready for consumption?

I'm much more nervous and concerned about my data with Resilio and I'm on the fence of asking for a refund - I'm starting to get buyers remorse really quickly.  I absolutely do not want to find myself missing this many files again - it is actually worse than a crashing application, it is actually intentionally destroying data - and I can't find anything to tell me why. 

Just to clarify on this point, fortunately Resilio is not my last point of backup, but if it deletes a mass load of files and I don't realize it for 30 days, it could begin to impact my other points of backup.  And that will be a real problem.

Things I'd really love to know ...

Am I wrong to be trusting Resilio or should I be staying with mainstream cloud sync providers at this time?  Is the only solution to create my own third party program to monitor the archive folder for files that Resilio is deleting?  Finally, all 6 or so computers are STILL syncing...  If I restore these files now, is this program just going to keep deleting them again until this sync finishes? 

I suppose, information wise, if there is a KB article or something I missed - this could help me a lot.  The only links I could find mentioned on the forums and posts are 404's.  For further examination, I would for future love to know the easiest way to do a restore when the archive has a bunch of renamed copies of files.  I really don't want to have to create my own tool / program here for monitoring and handling Resilio archives ... but the program seems like it is incomplete in the archiving area.

Support Ticket

I have not been able to create a support ticket yet.  The login system is asking for my License ID, which when I copy it from the program, it just says it doesn't recognize.  I'll give this thread a few days, and hope for a response.

Share this post


Link to post
Share on other sites

Some findings

#1 - I was correct, the application does not appear smart enough to register that something was copied from the Archive back to the Main.  It is maintaining dual copies.  Only time will tell if it "duplicates" all the copied items in the archive ...

#2 - I believe I found the reason Resilio decided to delete the millions of files - and I honestly expect it will continue to delete every file until the sync is complete.  This is a great example of why documentation is important, as the order of operations is quite possibly the reason it eradicated the file structure - and now it has me concerned that there are bigger issues in Resilio.

Near as I can tell, what appears to have happened is this:

  • The Share was created on Computer A.
  • Computers A,B,C,D,E,F were connected.  Everyone was empty.
  • Computer A was setup with Symlinks - however the program registers those as empty and nothing would Sync.  So the symlinks were removed.
  • All 6 Computers (including the owner) were then firmly seeded with content (no more symlinks)

What is expected:

  • Resilio sees the new content on all 6 machines, and begins indexing and validating the data

What appears to have happened:

  • Somehow Resilio decided that the new files on owner conflicted with the original empty state.  It then proceeded to delete all the files on owner, resulting in it Syncing deletions to all children.

What is interesting, is the deletions did not start occurring until about 24hrs to 48hrs in. 

If this is a true accounting of what happened, this means there is a very real possibility anytime I copy a file into Resilio from any of the computers, that it will consider it a conflict of some past state of the Owner, and will just silently delete the files.  But if that is true, then why aren't the forums on fire from everyone having missing files?

Share this post


Link to post
Share on other sites

Lloyd,

You're not alone.  We were using reslio on a largish share that wasn't as large as yours (some 100k or so files) and it got very confused, deleting more recent files and replacing them with an older rev when a peer that had not been connected in a while came back online.

Resilio sync seems *very* sensitive to the local times on the peers.  It does not seem to verify that the local time is correct (I suggested this once a while back but was rebuffed).  That's just a guess on my part though.  We never really got to the bottom of the issue.

 

Rob

Share this post


Link to post
Share on other sites

Sync relies on "modified time" timestamp to know which file is newer. "Newer" file is usually overwritten if it's opened on two peers at the same time, edited and closed on peer1 (file updates mtime), and only after that closed on peer2 (file updates mtime here as well, file gets newer timestamp). From person's POV, file1 is newer, from timestamp POV - file2, which gets synced, overwriting the newer one. This is the most frequent use case in "newer file was overwritten"  report. 

Share this post


Link to post
Share on other sites

I have encountered this exact issue, massive numbers of files disappearing, but in my case it was all contents. Based on Helen's response above about Sync relying on the time stamp I wonder if sync can reliably be used across time zones. If client A marks a file as modified a 1:00pm Pacific and client B marks the file modified at 1:30PM eastern (2.5 hours before than the pacific time stamp) will Resilio update with the older file because it has a later modified? I ask because in my experience Resilio has been pretty bullet proof with computers on my local network and even from home to the office. But every time I try to sync in with our out of state user all files end up getting dumped into the Archive folder as if they were deleted on one side or the other. Could this be the same issue, or are time zones accounted for? And if so, why are my syncs working perfectly in California but every time (about 4 different attempts) I try to syn with a computer in Tennessee I have the folders empty themselves completely?

 

 

Share this post


Link to post
Share on other sites

Sync uses GMT unix timestamps, these are calculated with peer's local time and timezone settings. So regardless of time zones your peers are in, timestamps are still unified. To ensure that time is calculated all right, peers shall have correct time settings. Otherwise, Sync will give the error: 
https://help.resilio.com/hc/en-us/articles/204753599--Excessive-time-difference-error

Share this post


Link to post
Share on other sites

The time zone calculation makes sense. But it still leaves me at a loss as to why all of my syncs in California work great, but every time I try to set up a sync with the computer in Tennessee it removes all of the files on both ends of the sync after about 24-48 hours during the first sync. Are there issues that can occur on an individual computer that would cause this sort of behavior? The computer is a Mac laptop (OS 10, but not sure which release), and the folder I'm wanting to sync is on the internal HDD. The sync seems to start fine, but with in a couple of days both sides of the sync start removing files to the archive. 

Share this post


Link to post
Share on other sites

better contact support and give them the logs with this case recorded there - right after reproduction. Mention the files affected and take the logs from at least two peers - the one where files are moved to Archive and the one that is mentioned in history as "Peer_name removed files" 

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.