Most likely cause: Using a cheap Synology NAS(816+) that does not have the resources to properly index all of the files required by Resilio. The Synology NAS was also not properly syncing Google Drive via its "Cloud Sync" app either. This was the clue that allowed me to figure it out. I don't know conclusively
The solution: Commissioned a Windows 10 computer to replace the Synology NAS for samba share, google drive syncing and resilio, along with Plex and other things I was using the NAS for.
I have three licenses, one Server and two regular licenses two regular, I'll call LocalSync and RemoteSync. Server is WinServer2016, LocalSync is Windows10Pro, and RemoteSync is a Synology NAS. WinServer2016 is the "owner" (or more actually the source of all data) of 4 folders configured as encrypted folders. LocalSync syncs those 4 folders in Read only modeand is located one gigabit switch away from WinServer2016. Additionall LocalSync is also the source of 2 encrypted folders. So LocalSync has 4 Read Only syncs, and 2 read/write syncs. RemoteSync is located 30 miles away from my business in my home. I have two pfsense routers on either side with a permenant OpenVPN connection. RemoteSync has all six folders configured as encrypted. All three machines think they are on a local LAN (and two are..) and thus go full speed, limited by my businesses upload speed of 20mbps. In all three locations in all folders, I DO NOT have "Store Deleted files in folder archive", i.e. delete ANYTHING that should not be there so I can compare folder size and number of files more easily (except the couple of extra files in ".sync"..np). I also have "Overwrite any changed files", again so I can more easily compare folder size and number of files.
The four folders between WinServer2016 and LocalSync are always 100% synced, no problem.
Three of the Four folders from WinServer2016 sync fine to RemoteSync.
One out of Two folders from LocalSync have finished syncing to RemoteSync
The last folder from LocalSync I've had paused to allow the other ones to finish syncing because they are very large volume backups (hundreds of gigabytes in a couple of files), so this one is not relevant...yet...unless it doesn't finish in a few days, then I'll be back
So my problem is the two folders, one from WinServer2016 and one from LocalSync that just seem to be stuck.
On RemoteSync I took some screenshots to remember where the sync was on each day:
On 1/21 the folder from WinServer2016 was 98% complete, and the folder from LocalSync was 60%
On 1/22 the folder from WinServer2016 was still 98% complete, and the LocalSync folder was 83%
On 1/23 (today) the WinServer2016 was 98% and Local sync was still 83% (Note that on LocalSync, it thinks its only 42% complete.
LocalSync folder should have 58,121 files. On RemoteSync it only has 57,950.
A picture is worth 1000 words so I made a diagram
What type of troubleshooting steps can I take to figure out what the problem is? One thing I'm thinking of doing is configuring a windows 10 Server at home for syncing, instead of my Synology NAS, as I think the problem may be my NAS. Although its not out of resources or anything and isn't throwing any errors.