drego85

[QNAP] Synchronization, with encryption, very slow!

Recommended Posts

Hi guys,

for two weeks now I'm synchronizing data on my NAS to a remote server. Sync is terribly slow, but not a bandwidth problem.

The data transfer is inconsistent, I'll explain it better.

I'm currently synchronizing, with encryption, a folder with 975,000 files of about 668Gb. They are all RAW images.

Just look at the story/log to see what happens, NAS transfers files every 6, 8, or 10 minutes. Not continuously!

2017_06.30_Screenshot_2158140.thumb.png.ecfc10e468142063ca3e999dd555ab89.png

 

If I control the data traffic of my network it is clear that the upload stream (the blue graph) is not constant but there are only peaks.

2017_06_30_Screenshot_2201190.png

Now I've thought that the Resilio process takes a long time to encrypt data, so I connected to my NAS via ssh and the rslsync process never occupies more than 25% of the CPU.

2017_06_30_Screenshot_2208530.png

The priority of the rslsync process was at 0, so I took priority to -10 and then -19. Nothing has changed.

After that I thought that the reading and temp writing of the data to be encrypted is slow, so I checked the processes that read and write with iotop.

But surprisingly I have seen that it often occupies 0% of the resources.

2017_06_30_Screenshot_2217110.png

In two weeks I'm not even 50% of the transfer, I do not understand. Other smaller folders (100Gb) went smoothly and with full regularity.

Some data on my current configuration:

  • NAS:
    • QNAP TS-453A with 8Gb of RAM (50% free around)
    • QNAP QTS: 4.3.3.0229 build 20170624
    • Resilio version: 2.5.4 x64 (Manual installation)
  • Remote Server:
    • HP i7 with 16Gb of RAM
    • Debian Jessie
    • Resilio version: 2.5.4 x64

 

Can you please help me to figure out what's happening?

If you need logs or other data to write to me, I am a regular home licensed user.

Thanks

Andrea

Share this post


Link to post
Share on other sites

This morning I installed the official version of the Resilio package for QNAP (Sync for x86_64), the qpkg file. However, saving the old storagepath to avoid new synchronization.

The situation does not change, the web interface is very slow to open and also synchronization of data.

Looking at the sync.log file I know the repetition of these messages which I believe to be error:

Quote

[20170701 09:34:37.086] ZIP: Can't locate [css/style.css] in zip, error -100.
[20170701 09:34:53.625] SRPEH[undefined]: bad shareID (STATE_BEGIN)
[20170701 09:36:17.159] D! 10SyncTcpReq::set_error[0x00007f16e0016b80][129] 32 (Broken pipe)
[20170701 09:37:35.599] D! 10SyncTcpReq::set_error[0x00007f16f8082330][-1] 103 (application protocol level timeout)
[20170701 09:37:35.599] D! 10SyncTcpReq[0x00007f16f8082330]: EOF - error: 103 (application protocol level timeout)
[20170701 09:41:48.766] D! 14PeerConnection::set_error[0x00007f16f8850750][-1] 103 (EOF)
[20170701 09:41:48.766] PC[0x00007f16f8850750][0x0000000000000000][xxx.xxxx.xxx.xxx:3300/TUNNELL] disconnect - reason: "Connection closed", rb: 0, wb: 0, error: 103 (EOF) ref: 1
[20170701 09:41:48.802] D! 10SyncTcpReq::set_error[0x00007f16e0016b80][129] 32 (Broken pipe)

In particular, the message "Broken pipe" repeats itself very much!

 

Share this post


Link to post
Share on other sites

Uhm maybe I solved, I have 13 shared folders. Of them 3 had not finished synchronizing, and they are the three largest folders by file and size. I disconnected two of them, the web interface quickly became fast and now about 30 minutes the sending speed is constant, 20Mb/s.

Keep you up to date...

Share this post


Link to post
Share on other sites

Okay, this afternoon has ended the transfer of a shared folder. 250Gb in two weeks due to bug, 400Gb in three days!

 

Now start the synchronization of one of the two remaining folders!

Share this post


Link to post
Share on other sites

Hey Drego85 - i am assuming you are syncing w/ encryption via Resilio?

Can you provide any updates?

Are you still doing this or have you tried other methods? (for either encrypt as REST/ and or TRANSIT? ie. VPN tunnelling)

Share this post


Link to post
Share on other sites
7 hours ago, threebythreeandsix said:

Hey Drego85 - i am assuming you are syncing w/ encryption via Resilio?

Can you provide any updates?

Are you still doing this or have you tried other methods? (for either encrypt as REST/ and or TRANSIT? ie. VPN tunnelling) 

Hi,
yes I'm still using Resilio with cryptography. All problems are solved and the system is now perfect.

Some tips I can give you:
* OVH cuts the band to p2p transfers, you can find another discussion here on the forum, so do not use it as a remote server.
* The NAS must have at least 4GB of RAM and must be 64bit.
* Initially transfer one directory at a time, too many at the same time overload the CPU.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now