[Raspberry Pi] Database Error

[Mouserider]

Hi,

I am experiencing a very annoying issue on my Raspberry Pi with Raspberry Pi OS and Sync 2.7.3.

I have 25 synced folders in my list and whenever I start my Pi (or restart the sync service), some of them show "Database Error". The database is not actually corrupted, because when I restart the Pi or the sync service, it will be other folders that show the message, sometimes more, sometimes less. But I can rarely ever get all of the folders to load correctly.

I think this might have to do with Sync trying to open too many files at the same time and that this could be fixed by retrying the failed folders after the other ones have been processed. Unfortunately there is no such option.

Yes, I have seen the help center page. Re-adding won't help as the folders are not actually corrupted.

Don't wonder about the "folder not found" message, this is expected.

[eltopo]

If you use SD card for Sync storage, you might have hardware problems, use dmesg to check system/kernel errors. 

I suggest that you get a good brand (SanDisk/Samsung, etc.) portable USB SSD drive for Sync.

[Mouserider]

All the data is stored on an HDD. The metadata (database) seems to be stored on the SD, as the messages appear even before the HDD is connected. And as I said before, after restarting the service, the cards are being reshuffled. Some broken ones will work again, other ones will break. Sometimes almost all of the folders have the message, sometimes it's only 1-2 of the 25.

[eltopo]

I would say it's a sign that your SD is not reliable. Move "storage_path" to your HDD from SD card (using config mode)

See:  Sync Storage folder – Sync (resilio.com)

[Mouserider]

12 minutes ago, eltopo said:

I would say it's a sign that your SD is not reliable. Move "storage_path" to your HDD from SD card (using config mode)

See:  Sync Storage folder – Sync (resilio.com)

In this case, can I still attach the HDD at a later point?

Btw, I also see an error message "Too many open files" after attaching the HDD. In this case it will recover automatically after a while. But this is also a reason why I was suspecting some OS/concurrent access limitations.

[eltopo]

If you frequently attach/detach your HDD, you probably should modify your script to check if the HDD is attached before you start rslsync, and stop rslsync before you detach it.

For "too many open files" message: you need to increase "open files" number, check "ulimit -n" for your current number, change it to a bigger number, such as "ulimit -n 50000", in your rslsync script.

See: How to Solve the “Too Many Open Files” Error on Linux (howtogeek.com)

[Mouserider]

Update about this issue. I did not change any settings, but I noticed at some point, that I had this issue only in one physical location, but not in the other one. It seemed to never load all folders correctly at place A (dozens of restarts necessary), while it always worked right at the first time at place B, which seemed very weird to me.

When it finally worked after many restarts at place A, I noticed that my computer was off at that point. I found out that I needed to stop Resilio Sync on my computer at place A in order to start the service on my Raspberry correctly. At place B I also have a computer with Sync running on the network, but this one seems to not cause any issues.

Even the Ethernet switch model that is connecting both of the devices is the same in both places.

[Jyro Hyte]

I've been trying to use a raspberry pi as my backup disk for years and never had any success.

I need to preface this and say that i back up a lot of data, i use it for all my photography and digital art etc, so a few terabytes of data in hundreds of thousands of files. 

At every install the sync would install and work great until i reach some threshold of number of files at which point it would stall. This stalling would not be very noticeable, for example it would just pause. The in-browser resylio window would just keep displaying the same thing, one folder is in the middle of sync, and a few are being scanned. So i would leave it there for a few weeks to see if anything would move and nothing. no matter how many times I restart and check things it just would not move forwards. 
recently i had luck with the raspberry pi OS, but not its hardware in that I installed the OS on a normal (but old) computer and its now working fine. 

So TL:DR, i think the raspberry pi hardware just cant handle the load or resilio with more then a few thousand files.