[QNAP] Synchronization, with encryption, very slow!

[drego85]

Hi guys,

for two weeks now I'm synchronizing data on my NAS to a remote server. Sync is terribly slow, but not a bandwidth problem.

The data transfer is inconsistent, I'll explain it better.

I'm currently synchronizing, with encryption, a folder with 975,000 files of about 668Gb. They are all RAW images.

Just look at the story/log to see what happens, NAS transfers files every 6, 8, or 10 minutes. Not continuously!

 

If I control the data traffic of my network it is clear that the upload stream (the blue graph) is not constant but there are only peaks.

Now I've thought that the Resilio process takes a long time to encrypt data, so I connected to my NAS via ssh and the rslsync process never occupies more than 25% of the CPU.

The priority of the rslsync process was at 0, so I took priority to -10 and then -19. Nothing has changed.

After that I thought that the reading and temp writing of the data to be encrypted is slow, so I checked the processes that read and write with iotop.

But surprisingly I have seen that it often occupies 0% of the resources.

In two weeks I'm not even 50% of the transfer, I do not understand. Other smaller folders (100Gb) went smoothly and with full regularity.

Some data on my current configuration:

• NAS:
  • QNAP TS-453A with 8Gb of RAM (50% free around)
  • QNAP QTS: 4.3.3.0229 build 20170624
  • Resilio version: 2.5.4 x64 (Manual installation)
• Remote Server:
  • HP i7 with 16Gb of RAM
  • Debian Jessie
  • Resilio version: 2.5.4 x64

 

Can you please help me to figure out what's happening?

If you need logs or other data to write to me, I am a regular home licensed user.

Thanks

Andrea

[drego85]

This morning I installed the official version of the Resilio package for QNAP (Sync for x86_64), the qpkg file. However, saving the old storagepath to avoid new synchronization.

The situation does not change, the web interface is very slow to open and also synchronization of data.

Looking at the sync.log file I know the repetition of these messages which I believe to be error:

Quote

[20170701 09:34:37.086] ZIP: Can't locate [css/style.css] in zip, error -100.
[20170701 09:34:53.625] SRPEH[undefined]: bad shareID (STATE_BEGIN)
[20170701 09:36:17.159] D! 10SyncTcpReq::set_error[0x00007f16e0016b80][129] 32 (Broken pipe)
[20170701 09:37:35.599] D! 10SyncTcpReq::set_error[0x00007f16f8082330][-1] 103 (application protocol level timeout)
[20170701 09:37:35.599] D! 10SyncTcpReq[0x00007f16f8082330]: EOF - error: 103 (application protocol level timeout)
[20170701 09:41:48.766] D! 14PeerConnection::set_error[0x00007f16f8850750][-1] 103 (EOF)
[20170701 09:41:48.766] PC[0x00007f16f8850750][0x0000000000000000][xxx.xxxx.xxx.xxx:3300/TUNNELL] disconnect - reason: "Connection closed", rb: 0, wb: 0, error: 103 (EOF) ref: 1
[20170701 09:41:48.802] D! 10SyncTcpReq::set_error[0x00007f16e0016b80][129] 32 (Broken pipe)

In particular, the message "Broken pipe" repeats itself very much!

 

[drego85]

Uhm maybe I solved, I have 13 shared folders. Of them 3 had not finished synchronizing, and they are the three largest folders by file and size. I disconnected two of them, the web interface quickly became fast and now about 30 minutes the sending speed is constant, 20Mb/s.

Keep you up to date...

[drego85]

Okay, this afternoon has ended the transfer of a shared folder. 250Gb in two weeks due to bug, 400Gb in three days!

 

Now start the synchronization of one of the two remaining folders!

[threebythreeandsix]

Hey Drego85 - i am assuming you are syncing w/ encryption via Resilio?

Can you provide any updates?

Are you still doing this or have you tried other methods? (for either encrypt as REST/ and or TRANSIT? ie. VPN tunnelling)

[drego85]

7 hours ago, threebythreeandsix said:

Hey Drego85 - i am assuming you are syncing w/ encryption via Resilio?

Can you provide any updates?

Are you still doing this or have you tried other methods? (for either encrypt as REST/ and or TRANSIT? ie. VPN tunnelling) 

Hi,
yes I'm still using Resilio with cryptography. All problems are solved and the system is now perfect.

Some tips I can give you:
* OVH cuts the band to p2p transfers, you can find another discussion here on the forum, so do not use it as a remote server.
* The NAS must have at least 4GB of RAM and must be 64bit.
* Initially transfer one directory at a time, too many at the same time overload the CPU.

[Frank Maier]

Your NAS has a quad core CPU. Resilio Sync seems to be a single threaded application. So 25% CPU load signals that the CPU was the limiting factor, because on core was fully loaded.