file deletion Unexpected File Deletion

[zpeverman]

We have been using Sync for 3 years now and still, once every couple of months need to audit the system to make sure that files have not been randomly deleted. I frequently have designers and producers come to my office and ask if files were moved. Entire folders and the content are being deleted and we can not figure out what is causing it. We can sometimes narrow it down to the machine that caused it, but not why they are deleted. Generally, If caught early enough, I can recover the files from the Archive folder, but it requires hours of cross-referencing to make sure we have everything that was supposed to be there and the files that were supposed to be deleted are still deleted. In worst case scenarios, we have had to pull from our cloud storage on Drive to get it back.

Ultimately there have been three variables we have found, but have no idea if it is the real cause. And if it is, how to fix it.

1. Software versions are not all the same across the network: While we do our best to keep them all up to date, there are times where they may not be matched across every machine. This should not cause an issue with syncing, but generally when we are troubleshooting this issue, this is a common thing we find.

2. Machine has been offline for several days: When a machine is offline for more than a few days, it seems to cause confusion when it comes back online. Files that had not been synced seem to delete from other peers rather than downloading to the machine that needs updating.

3. External Drive Disconnection causes File Deletion: If a machine uses an external drive and that external drive is not fully synced it will cause the unsynced files to delete from peers if the drive is removed. This is typical with employees who use mobile workstations that they take with them at the end of the day.

BONUS ISSUE - Sync installed as Service doesn't always launch: We use machines that have users manages on a domain, every user needs access to the same media so we have set it up in a way that a Sync is running as a service in order to stay up to date no matter what user is logged in or if any user is logged in. This service does not always launch there is no visual indication of it running without loading the GUI or checking the system activity monitor. Because it can do this for days sometimes, it creates issue #2 from above.

What I need in order to continue using this service is to learn how to keep this system more stable and reliable without needing to babysit the processes and all of the machines every day. What are the steps I should take in order to achieve this? Thanks!

[Chris Cho]

We are having a similar issue with our setup with a FreeNAS box and a win10 machine.  We get a "Failed to Download" event in Resilio, then the system removes the file.  I submitted a ticket early this month and haven't gotten a reply yet.

[Lentil-Soup]

Same issue - catastrophic data loss.

Have had to fall back 2 tiers into our backup strategy to restore data.

Files that were copied over and working great between several nodes are being removed (with seemingly no rhyme or reason), with no questions or warnings, and users are being granted with Failure messages instead of usable files.

Can no longer trust Resilio at this point... a shame, because I really want to!

Am paying for Pro. I am starting to question why, but will troubleshoot just a bit more before giving up.

[zpeverman]

This is absurd. I'm sorry you guys are having these issues, but at the same time glad it isn't just our set up. I have no idea where to start on troubleshooting this issue or how to prevent it from happening in the future. We have an entire network of data that is actively syncing 8TB of Data on and off-site to a remote employee. It is all syncing between 10-12 machines. When it works it is great, but when this happens it is devastating and cripples our workflow for so long while we recover the data. And quite honestly, we have had situations where recovery didn't work despite our cloud backups as well.

I hope more people chime in here to Resilio understands the magnitude of this issue or can provide guidance for us to resolve it.

[maccra]

This just happened to me also... What an absolute disaster. I shutdown Resilio last night, It was only shutdown about 16 hours.

I just placed some files in one of my sync folders while Resilio for shutdown. Started Resilio and then Resilio immediately deleted the files instead of syncing them to the other peers. Luckily the files were in the archive folder, but really it should never have happened.

Yes...Resilio,, where is the guidance or what special tips do you have for us... "Never shutdown Resilio"... really!!!

[zpeverman]

I'm pretty dissapointed in the level of attention to this forum by Resilio. I am actively testing other options that can do this same thing. It may not be as fast, but we are looking at using Google Drive paired with File Stream on each of the machines. Anyone else have any other suggestions?

[ekim501]

So this happened to me as well.  I have a 450 GB folder synced to several computers and half of it just went missing.  I was able to restore it all using the archive but it was such a pain - and it coped over multiple version of files that were changed throughout the day.  This makes me extremely nervous - I feel like I'm going to have to check my archive everyday to make sure there is nothing in their that shouldn't be.

[colin-gnec]

good to know it is not me either, I can't figure it out, wonder if some tool like splunk might help,

[robertpersons]

I have started using Resilio, but these stories give me pause.... especially if files get deleted simply because one PC was down a few days.

Is everyone having this problem running Resilio 2.6.3?

Are these problems limited to running Resilio on machines other than Windows? I'm wondering if an all Windows network might be more stable than syncing between other OS, like Unix which doesn't have the same level of time/date stamps.

[jocko]

Hi

Except for Alex's message on February 20, it seems the staff is making a long break...: no answer from the support on any new help topic since January

So not really nice🙁

 

[Ole Wissing]

Just lost 4-500Gb high-quality scans of old-school positive film from  and 80'. 1960 - 1980. 100s of hours vasted!!!!!!

If it was not because I have seen it before, and don't trust Resilio completely (it is getting better) and have the data distributed to multiple hosts. But like with you other guys, it is a pain to get it all together again. And Resilio won't even start up. Says it is missing a service file, and then restarts, crashes, restart, crashes and so on.

Running version 2.6.3 on a fully updated Mac (10.15.3)

Annoying!

[AlexC]

@Ole Wissing Did you check .sync/Archive folder? What do you see in history tab?

[zpeverman]

@Ole Wissing We just started using SyncThing. There is a bit of setup to do with it, but it has been pretty solid so far. And it's free. Shoot me an email if you would like to know a bit about how we have it set up. zach(a)thepixellab.co. I'm not really the IT type to know every detail, but I can give you a high level overview of how we are using it.

We gave up on Resilio because it was failing to index and sync almost 600GB of data. Resilio kept telling me it was temp files but it was 100% media that our editors were not getting synced to their machines. 

[Ole Wissing]

I now realise, what has happened. And it is not Resilio fault, the folders are missing. I had a systemcrash, where I reinstalled OSX. I did not think it touched the data too, because everything else is here. I guess it must have been that.

But that does not explain, why Resilio is not starting and repairing the broken system, but instead crashes.

[Andy+]

I recommend testing Syncthing well because there are a few things to consider as well.

I mainly use 3 Synology servers, 5 Windows clients, 3 Smartphones with Resilio. I haven't had any data loss so far, but there are inconsistencies, some of which are mentioned here. Because of this, I have now also installed Syncthing in parallel on all devices and with appropriate ignorelists there are overlaps in the same directories possible. In comparison, Resilio is the reference for me, although Syncthing is catching up and running just as well. Because the Resilio APP does not start reliably on smartphones, syncthing is now installed everywhere. Overall, it have pros and cons and each package has its advantages and disadvantages, so I use both in parallel.

However, one thing is clear, the support at Syncthing is completely different than here, everyone can get an idea of it "https://forum.syncthing.net".

How such deletions can occur is at least a mystery to me. I have never had anything like this, but I consistently and everywhere use the latest version, now v2.6.4 (I could now make a choke and say that with these update cycles this is not a problem either ... 😎).

Maybe first something about the history and my reasons for using both:

I have been using Resilio since version 1.4.111 (then BTsync), the previous versions were not yet so stable. I pre-matched the 3 Synology servers at different locations and then set them up there. With Resilio, I manage around 2.11 TB of data in over 50 peers. Resilio has been running well since v2.x.x and currently v2.6.4 is actually reliable, without major errors and is fast. I experimented with Syncthing and developed an ignorelist so that Resilio and Syncthing can work in the same directories:

// Resilio files
.sync
.syncID
.syncIgnore
.syncArchive
* .bts
! sync
* .syncPart
* .syncTemp
* .syncOld

// syncthing files
.stfolder
.stignore
.stversion
.syncthing *
.syncthing. *
.syncthing. *. *
~ Syncthing ~ *. *. Tmp
* .Identifier

Syncthing has a better relay handling with external connections than Resilio. If the 5-digit ports cannot be used, e.g. because the ports in the router are closed or cannot be opened as UPnP or because you are in a strict company network where only http and https work, channels at the http and https level are used, i.e. ports 80 and 443 connections. In such an environment, Resilio is unworkable. Then I use syncthing, that is for me the main reason. Other minor advantages are several peers can point to one directory in Syncthing, for example to establish a kind of star connection. This is not possible with Resilio. The "Send + Receive", "Send Only" and "Receive Only" modes are easy to set up. Resilio can do that too, but the peer type has to be changed and set up with each change.

Then there is the performance: One of the servers is on a 2000 (!) Line (yes, that still exists today). Because Resilio is really very performant, this performance needs quite a bit of basic bandwidth. As soon as there are already 5 larger peers, almost nothing works anymore. If the peers are small, so if there are small directories to match, a few more peers go, but the performance problem remains. With Syncthing I have over 40 peers in sync and it still works. The reaction speed to changes is then slower, but this does not bother with this server. These 40 peers go to the same directories on my productive server, which are compared with all other servers and clients with Resilio, thanks to my ignore list no problem.

My experience shows that Resilio has a very strong competitor, but is still a reference. Both packages have bugs and minor problems. I use open words and Resilio should read along well, because more and more people deal with the "other".