Inconsistent Speed

[adinunzio10]

@adinunzio10

Thanks for your feedback. We'll address the issue when we collect enough info. If we need more information - i'll drop requests to this topic or directly to you.

 

Sounds good! Thanks for the help, and I am always willing to assist / provide feedback. Drop a PM though as now that my issue is resolved, I probably wont be returning to this thread often. PM will go directly to my email. 

[ManChicken]

I just setup Sync 2.0.82 on two machines on a LAN (one Linux, one Windows) and have been getting abysmal speeds, of between 3-4MB/sec on a Gigabit LAN.  Every now and then it will spike up to ~10-12MB/sec for maybe a second but otherwise maintains a slow average. These are large ~400MB files but would normally be several gigabytes in size (large uncompressed video files.)

 

???

[RomanZ]

@ManChicken

Let's try to do the following:

1. Shut down Sync

2. Open your storage folder (%appdata%\BitTorrent Sync for Windows, .sync subfolder for Linux), thre will be a bunch of Sync files (DBs, configs, etc).

3. Create debug.txt file

4. Put next data to it:

00000000<return>

0000000A

5. Start Sync

6. Do the same on second peer.

 

This will force Sync to use TCP connection instead of UDP. Let me know if it helps.

[ManChicken]

Thanks for the response.

 

I did the above but it pretty much is exactly the same (and I did verify it's doing a TCP now via netstat)

[RomanZ]

@ManChicken

Okay, so it's not the issue of slow processing of UDP packets. I suggest collecting speed profile on both your peers - we'll try to identify the bottleneck.

[sebijisi]

Hi, 

just read the thread. I have also been having serious speed issues syncing to a host across the internet. I tried everything and cursed BTsync to the moon and back.  I think I just confirmed in my case it was not BTsync's fault.It was really is a problem with UDP traffic being throttled (heavily!) along the way

 

UDP

iperf -c <ip> -u------------------------------------------------------------Client connecting to <IP>, UDP port 5001Sending 1470 byte datagramsUDP buffer size:  208 KByte (default)------------------------------------------------------------[  3] local <ip> port 44731 connected with <ip> port 5001[ ID] Interval       Transfer     Bandwidth[  3]  0.0-10.0 sec  1.25 MBytes  1.05 Mbits/sec[  3] Sent 893 datagrams[  3] Server Report:[  3]  0.0-10.0 sec  1.25 MBytes  1.05 Mbits/sec   0.035 ms    0/  893 (0%)

The same pair of hosts using TCP

iperf -c <IP>------------------------------------------------------------Client connecting to <IP>, TCP port 5001TCP window size: 85.0 KByte (default)------------------------------------------------------------[  3] local <IP> port 50024 connected with <IP> port 5001[ ID] Interval       Transfer     Bandwidth[  3]  0.0-10.0 sec   112 MBytes  93.8 Mbits/sec

 So my first question is:

 

Is there a hidden option to force the sender to use TCP (or the receiver to request TCP connections from senders?), or is it at the moment only available in custom test builds?

 

I think it is of general interest, because in this case it is a limitation of the network the receiving server is in (I got the same iperf results sending from two different networks) . It is hosted in an OVH datacenter (a french provider, pretty big in the EU), and I guess it might be related to their DDoS protection stuff. However, this is just a lucky guess as this is my first OVH hosted host I am not very familiar with their network.

 

Anyway, I suppose there will be quite a few people renting el-cheapo servers as secondary offsite backup, that might run into this problem. I hope my post will make it at least googleable.  But can almighty RomanZ maybe even solve it ?

[sebijisi]

Ok, it turns out I do not know how tu use iperf... UDP requires to set a sending rate. As in 

iperf -c <IP> -u -b 20m

if you want to check 20 MBit. Turns out the host is still slower in UDP: Around 20-25 MBit, afterwards the dropped packet rate skyrockets. However, this does not explain BTSyncs << 10Mbit speeds. 

 

As for now I am trying with a TCP-OpenVPN that tunnels BTsync traffic directly to the machine. However as the Mac cleint exploded on me, and after crashing restarted indexing (and even asked a new identity and restarted the 30 day trial), I can not yet say what the results will be.

 

For the time being the receiver still throws millions of

[20150316 15:55:27.718] 10.8.0.6:59696: did not pick any blocks. blocking peer temporarily[20150316 15:55:57.593] Disconnect: Is seed[20150316 15:55:41.725] Disconnect: Timeout due to inactivity

into the log

[sebijisi]

I get the feeling this is a limitation of BTSync not being able to sync large amount of files. The best I manage to do with a medium sized share (16GB, 16k Files, various sizes) was around 2 MB/sec, which will occasionally drop to less then 1 MB/sec in the "sawtooth" pattern described here. No matter it is local or remote or TCP or UDP.

 

However when I add the share I originally want to sync (400GB, 420k Files, Most 8MB), it never goes over 1 MB/sec. Also, when the large share is added to BTsync, even when syncing is paused, the medium sized share mentioned above will also not go over 1 MB/sec.

 

I am not sure I am CPU bound. One machine is an oldish Core 2 DualCore, the other a Dual Core (4T) Cedarview Atom. While BTsync has significant impact on CPU, the CPU usage for both machines is not nearly maxed out under the conditions lined above.

 

Is there something in BTSyncs architecture that prevents it from syncing large collections? Some event-processing/thread locking limitation? Some scaling problem with a large number of files? Did anybody ever saw BTSync achieving or surpassing 100MBit/s when the files are large enough?

 

Also I did notice some largish sqlite3 database (536 MB) created by btsync. And it seems to have no indices. However, as I do not know whether they are read or written more often I do not know whether this could be a performance problem. Just worked a lot with sqlite3 before, and noticed that it can go from awesome! to yuck! pretty quickly once there is a lot data in it....

 

I did try the "speed profile" thing for a few minutes, but it just collected 10 MB of data I can not interpret by myself

[RomanZ]

@sebijisi

Indeed, there is some issue in processing large amount of small files by Sync - and they can affect data transfer speed significantly. The sqlite DB is not a bottleneck here. We plan to optimize it in future.

 

For the TCP connections instead of UDP ones - there is a debug flag which you can set to force Sync to use TCP to connect. Although note, that UDP is much more flexible in establishing direct connection, so forcing TCP may result in inability to directly connect in some cases.

 

So, to force TCP-only

- create a debug.txt file in storage folder (see this article for possible locations depending on OS)

- put the value of "00000000<newline>00000002" into this file

- save file, restart Sync. Do on all peers involved.

 

Disclaimer (to everyone who did not read the warning above)

Use with care. May cause inability to connect peers directly.

 

Let me know if it helped to improve your syncing speed.

[Herdo]

I've created an account to try and solve this issue.  I've followed your instructions, but I'm not sure I've done everything correctly.

 

I've found the debug.txt file on both machines.  There is only one line of text inside that reads "FFFFFFFF0".

 

Do I delete this line, or leave it where it is?

 

Also, your instructions say, put the value of "00000000<newline>00000002" into this file.

 

Does that mean like this:

 

00000000

00000002

 

?

 

I've tried this, but I'm not sure how to check if it's forcing TCP.  It doesn't seem to have fixed the problem.

 

Thanks in advance.

[RomanZ]

@Herdo

You can leave the first line as is. It is only responsible for what to log and what not to log. Setting it to zero lets Sync know that you no longer want to write logs. Indeed, debug.txt should consist of 2 lines:

00000000 (or FFFFFFFF)00000002

The second line turns off UDP traffic, which in certain cases causes traffic slow down. Although, I would recommend first disabling the "hdd_low_priority" advanced setting. In majority of cases HDD is the bottleneck.

 

Also, could you please share a bit more details about your case? Are peers connected via LAN or internet? What are the peers OS and hardware? Channels bandwidth?

[harryPotter]

I have recently discovered this same issue. 

[zcleaver]

I am currently experiencing what i believe to be the same issue. Here is my post:

 

[LucasVS]

So, no progress or update since 2015?

What is the status of this, as I am seeing this at present, and no fixes listed

[rayden]

Hi guys

I know this is an old topic, but I have just stumbled on it and I think i have figured it out that it may be a problem with UTP protocol which limits the data flow when it figures the network is "overloaded", I have changed "tunnel protocols" so that TCP is first and the speeds are what they are what they're supposed to be (max link speed)

[duckturduckenduck]

On 5/24/2020 at 3:35 PM, rayden said:

Hi guys

I know this is an old topic, but I have just stumbled on it and I think i have figured it out that it may be a problem with UTP protocol which limits the data flow when it figures the network is "overloaded", I have changed "tunnel protocols" so that TCP is first and the speeds are what they are what they're supposed to be (max link speed)

Did this end up being a consistent solution for you? I think it worked for maybe a day for me and then speed went back to weird. I've had some luck cycling listening ports, but always a temporary fix. I don't think I'm being throttled on the ISP end, since all other types of connections seem to be fast. The fact that under a fortuitous combination of things I'm able to reach speeds of 25Mbps+ seems to indicate that something is weird on the RS side. This has been going on for years now, but still holding on for some kind of actual solution...