Multiple problems with Bittorrent Sync between two Synology NAS

[vrrp]

Hi,

Having had a ton of issues that I've been able to surmount (namely permissions issues that I could only fix by copying all my files to a new share on the same source NAS), I'm now unable to get this working, and here I came for help.

The purpose of my bitsync install is to keep a working server in my company synced with a backup server which I'll keep home after the initial sync (~3.3Tb) has taken place locally.

My target NAS is using bittorent sync 1.4, as it is the latest available version for qoriq based Synology NAS, but I have been able to install everything, setup a shared folder and start the sync process, so to my untrained eye this does not seem to be the source of my problems. Source NAS is running the latest version available through synocommunity. Target NAS only has read permissions (thus my source/target names for the NAS').

Problem 1: My source folder is ~3.31 Tb; Sync only "sees" 2.94 Tb. It shows "indexing" for days, but it does not seem to be able to index the latest ~370Gb or so. (picture 1). Permissions' issues with the original share were what made me use the nuclear option, create a share and define all the permissions from scratch, sos I'm pretty sure this isn't a permissions issue as outlined in the FAQ and in many forum posts.

Problem 2: Target NAS synced folder stopped at around 260Gb, and it stopped increasing. Although both NAS are currently connected to the same gigabit switch, transfer rates went from ~10Mb/s with peaks of ~30Mb/s to 1-2kb/s. I have since removed the shared folder in the target NAS, in the hopes that this would decrease the load on the server and improve the speed with which it was being indexed. Hasn't happened. Any ideas on why it stopped syncing with more than 90% to go?

Problem 3: Bittorent Sync's web interface stopped generating the QR codes for syncing with mobile apps. (picture 2) This was working before, and I have been able to successfully link the shared folder with my iPhone.

Two related questions:

1 - Bittorrent Sync generates a ton of logs and journals, any hints on what I should try to search for in order to try to understand what the problems are? Also, why would I have a B07A0AB68F26F6E0CCEEF0CFBBC7FBBFEFAB3455.5711241.db file at /usr/local/bittorentsync/var that is ~3.6Gb and growing? It seems quite large, what is supposedly stored in this database?

2 - I've had a (quite bizarre, IMHO) behaviour where the files in a test source folder were deleted when I removed the shared folder in the source NAS through Sync's web interface. I believe this is not intended behaviour, but this is in a way hold me back trying the "nuke" option of uninstall/reinstall and starting from scratch.

I realise these may not be the cleverest questions, but I've pretty much exhausted all the how-to's and forum material that I could find and I'm stuck.

Any help towards a despairing noob would be dearly appreciated.

Thanks in advance,

Vasco

[RomanZ]

Hi Vasco,

I'll start with your questions:

Quote

1 - Bittorrent Sync generates a ton of logs and journals, any hints on what I should try to search for in order to try to understand what the problems are? Also, why would I have a B07A0AB68F26F6E0CCEEF0CFBBC7FBBFEFAB3455.5711241.db file at /usr/local/bittorentsync/var that is ~3.6Gb and growing? It seems quite large, what is supposedly stored in this database?

Logs and journals could be turned off by writing "0" to debug.txt and setting "enable_journaling" power option to "false". Though, I'd advise to keep it on for now as your setup has some issues.

DB contains the tree of all files that you want to keep in Sync. It's size depends on 2 things: 1) amount of files and subfolders in synced folder 2) amount of operations you do with files. I suspect that you got something around 2.2M files inside synced folder. Is that true?

Quote

2 - I've had a (quite bizarre, IMHO) behaviour where the files in a test source folder were deleted when I removed the shared folder in the source NAS through Sync's web interface. I believe this is not intended behaviour, but this is in a way hold me back trying the "nuke" option of uninstall/reinstall and starting from scratch.

What is the version of Sync you use on source? We had such possibility in 2.0, but as many users removed their files accidentally, we completely removed that feature from Sync 2.2 and newer.

Quote

I realise these may not be the cleverest questions, but I've pretty much exhausted all the how-to's and forum material that I could find and I'm stuck. Any help towards a despairing noob would be dearly appreciated.

 

No worries, we are here to help. We'll do our best to deal with your issues. I'll ask you to send us some debugs below, the simplest way would be to open support ticket with us so me and my colleagues can work altogether on your issues. Mention this forum topic in message body.

And now to your problems:

Quote

Problem 1: My source folder is ~3.31 Tb; Sync only "sees" 2.94 Tb. It shows "indexing" for days, but it does not seem to be able to index the latest ~370Gb or so. (picture 1). Permissions' issues with the original share were what made me use the nuclear option, create a share and define all the permissions from scratch, sos I'm pretty sure this isn't a permissions issue as outlined in the FAQ and in many forum posts.

 

There are 2 common things that can cause such behavior: a) permissions. Although, as you checked all permissions manually it is unlikely reason. b) ignored files. Sync ignores some files according to IgnoreList set of filters. I suggest checking if there is something that is ignored in your case.

Quote

Problem 2: Target NAS synced folder stopped at around 260Gb, and it stopped increasing. Although both NAS are currently connected to the same gigabit switch, transfer rates went from ~10Mb/s with peaks of ~30Mb/s to 1-2kb/s. I have since removed the shared folder in the target NAS, in the hopes that this would decrease the load on the server and improve the speed with which it was being indexed. Hasn't happened. Any ideas on why it stopped syncing with more than 90% to go?

There is some issue that prevents Sync from transferring data, so it's not about your bandwidth. We need your debug logs to find out what happens. Your huge DB makes me thinking that you got enormous files tree which consumed all resources (both RAM and CPU) on your NAS, which results in inability to Sync.

Quote

Problem 3: Bittorent Sync's web interface stopped generating the QR codes for syncing with mobile apps. (picture 2) This was working before, and I have been able to successfully link the shared folder with my iPhone.

I also suspect lack of resources. Does the Sync UI opens fine in general? Quick workaround here is just to copy your link produced by Sync from NAS to any online QR code generator. Works fine.

 

[vrrp]

Question 1: around 2.2M files, yes. Thank you for clearing this up.

Question 2: I'm running 2.2.5 (131) on the server. I'm glad this behaviour is not possible anymore, as I'm considering removing the share folder and adding it again if all else fails.

Problem 1: This is the content of my IgnoreList, it does not seem to be the case that I am excluding important stuff out. What should I look for in the journal and/or logs for permissions issues with specific files/folders? (although, again, I do not believe this to be the case, but it appears there aren't any other possible causes for this issue).

Problem 2/3: Indeed, logging into the sync web interface on the server (after user/password has been submitted) takes something like 15 minutes (used to be much faster, of course). First thing I'll do is increase the RAM in my NAS, hopefully the QR code generation problem will be solved too.

I have submitted a support ticket with the latest logs and journal.

Thank you so much for your help.

[vrrp]

I forgot to add the (default) contents of my IgnoreList:

# OS generated files #
.DS_Store
.Spotlight-V100
.Trashes
ehthumbs.db
desktop.ini
Thumbs.db
# Temporary files #
~*
*~
.~lock.*
*.part
*.crdownload
@eaDir
@SynoResource
.@__thumb
._*

Thank you.

[RomanZ]

Vasco,

Let's continue in our help desk.

[Terence]

RomanZ, when you guys have figured out and fixed the problem, would you follow up here on what you had to do to get this configuration to work?

I am about to setup a similar configuration with a NAS and a Windows server replicating 6TB of data between each other. There are millions of files and hundreds of thousands of folders. I'd like to know what problems I would have to deal with to get this to work. Thanks.

[vrrp]

I'll leave some feedback here later on. I'm "listening" on this post, so feel free to ask questions, I'll do my best to help.

[RomanZ]

@Terence We usually post outcome on forum, though not sure if it is going to be useful for you as Vasco is using mixed environment of 2.2.5 and 1.4 clients. I can only advise to take care of hardware - syncing millions of files requires really lots of memory (count roughly 1.5kb per file/folder).