Inconsistent Speed

[crazyk4952]

My setup is as follows:

 

Server 1: 

Ubuntu 12.04 server on a symmetric Gigabit connection (I usually get about 500Mbps up/down).

BTSync running

Static IP address

 

Server 2:

Linux Mint on a DSL 40Mbps down/20Mbps up connection

BTSync running

Dynamic public IP address behind a NAT with a static private IP address

BTSync has been assigned a specific port. Port forwarding rules have been set in the router for the specified port to the private IP address.

 

I sync one folder between the two computers. When I add files to server 1, the files start downloading on server 2 (as expected). However, the download rate will fluctuate. It will seem to climb up to about 40Mbps, then drop to zero. This repeats until the sync is complete. I have a graph of this below.

 

The file I synced in the graph below is about 300MB.

 

I have tried setting a download limit on server 2, but this does not help.

 

Is this normal behavior?

 

[grongl]

Fantastic that someone posted this. I'm still seeing a very similar pattern when transferring large files (around 2.5GB each in this case) between two remote locations (upload: Mac, 20M; download: RaspberryPi, 6M). Nothing else is going on at both ends when the shots attached were taken, both running 1.2.73.

[danvari]

Yeah, I get this too, syncing from a debian box to a Mac results in a very similar pattern.

Version: 1.2.82 (Mac10.9/Linux64)

[cubik]

Running into same situation:

server A behind pfsense firewall in LA

server B behind software firewall in NC

 

Windows 2008 boxes, running 1.2.82 build

only port forward is the listening port for each, to each other, TCP/UDP

 

other than default, I have:

disk_low_priority = false

lan_encrypt_data = false

lan_use_tcp = true 

 

for folder preferences I have only the 'use predefined hosts' checked with the other ip/port in correctly

 

Is it because a single port is open, and it could use more than one to even things out?

I thought it would open more using pnp, but according to a netstat dump, just one port active.

 

Edited January 18, 2014 by cubik

[RomanZ]

Hi all,

 

Can you please let me know the time scale on your graphs? What is the period between two peaks, is it a matter of hours, minutes, seconds? Did anyone see the same bandwidth usage pattern in the LAN with lan_use_tcp enabled?

 

cubik,

Your graph is 6 minutes long, your download rate jumps slingtly which is okay for UDP. Also note, that BTSync behaves very gently and if it sees that it overloads the channel while someone else is using it - it will automatically decrease speed. lan_use_tcp is not working in your case as you are using 2 remote connections.

[crazyk4952]

Hi all,

 

Can you please let me know the time scale on your graphs? What is the period between two peaks, is it a matter of hours, minutes, seconds? Did anyone see the same bandwidth usage pattern in the LAN with lan_use_tcp enabled?

 

For the graph that I posted, the entire window represents 10 minutes. So, each vertical line would be about 2 minutes.

 

I do not have a graph for the bandwidth usage pattern during a LAN sync. I will post one later today.

[crazyk4952]

Below is a bandwidth graph of a sync over my LAN (with the lan_use_tcp enabled):

 

 

 

Here is another sync over the WAN:

 

Just to compare, here is a file copied via SCP (over the WAN) between the same two computers as above:

 

[Guest proactiveservices]

Maybe BT Sync is waiting for hashes of files to compute/verify or is suffering a CPU/disk bottleneck somewhere. The graphs are so regular it does seem to be purposeful, whether bug or normal.

[RomanZ]

crazyk4952,

 

Thanks for the graphs. The WAN graph is very representative. It is very complex to debug bandwidth issues remotely, so we are trying to reproduce it in our lab.

 

Can you please share as much details as possible so we can build a repro? What are the OSes on both your peers, bandwidth (both peers), router types and configurations, any security software installed? + Any information you think might be valuable in building reproduction.

 

Thanks!

[crazyk4952]

crazyk4952,

 

Thanks for the graphs. The WAN graph is very representative. It is very complex to debug bandwidth issues remotely, so we are trying to reproduce it in our lab.

 

Can you please share as much details as possible so we can build a repro? What are the OSes on both your peers, bandwidth (both peers), router types and configurations, any security software installed? + Any information you think might be valuable in building reproduction.

 

Thanks!

 

 

Sure thing, I'm glad to help! BTSync was installed on both computers from the tuxpoldo/btsync PPA.

 

The two computers are as follows:

 

Server 1:

- Ubuntu 12.04 - 2GB RAM - 200GB Hard Drive

- Virtual Private Server

- Intel® Xeon® CPU X3440 @ 2.53GHz, 4 cores

- 1 Gigabit up/down connection to internet

- BTSync 1.2.2.2

- Static IP address

 

Server 2: 

- Linux Mint 13 MATE - 2GB RAM - 250GB Hard Drive

- Dedicated Server

- Intel® Atom CPU N550 @ 1.50GHz, 4 cores

- 40Mbps down/20Mbps up DSL residential internet connection

- BTSync 1.2.2.2

- Dynamic public IP address

- Server is behind a NAT (with a static private IP address)

- Router is an ASUS RT-N66U running Tomato 1.28 firmware by Shibby

- Port forwarding is enabled on the router for the port that BTSync uses

 

I have also tried sharing a folder between Server 1 and a friend's computer running Windows on a different ISP. The bandwidth pattern seems to go up and down when transferring files between computers also (although I don't have a graph of that usage).

[RomanZ]

crazyk4952,

 

Thanks a lot for the info. I'll update as soon as we have some news on saw-formed BTSync traffic issue.

[RomanZ]

crazyk4952,,

 

Few comments while we are working on reproduction.

1. BTSync version you mentioned is 1.2.2.2, while latest available is 1.2.82 (and it has a bunch of speed optimization). Could you please upgrade and see if issue persists?

2. There is another good test I'd like to ask you to do. Could you please try to sync with all the same parameters a folder with files 1..2Mbytes each, amount of files should be enough to make a graph. In this case BTSync is going to maintain different amount of simultaneous connections.

 

Thank you for your help.

[antol111]

Hi,

 

I signed up to the forums just to reply to this thread and  let you know that I am experiencing the same issue.  I have 3 clients synching:

 

Linux x64 running 1.2.82

Windows 8 x86 running 1.2.82

Windows 7 x64 running 1.2.82

 

Files are added on the linux client and simultaneously synched to the two windows clients.  The download speeds on the windows clients match the pattern OP is seeing. 

 

I don't think I see this fast/slow pattern when only one windows client is synching with the linux client. (i.e. i have three folders on the linux client    A Only    B Only    A+B.   I can't double check now as I have a sync in progress but it is currenlty doing the fast/slow pattern because I am syncing to both windows clients.

 

The windows clients are on the same ISP (Comcast) and within 20 miles of each other.  The linux client is hundreds of miles away.

 

My guess is has to do with some calculations on the next block to download and each client needs to figure out where to download it from.  I wonder how adding sync client compounds this issue?

Edited January 30, 2014 by antol111

[RomanZ]

antol111,

 

I could agree on calculation issue if we are talking about low-performance CPUs, but if you are using desktop PCs with x86/x64 architecture - calculating hashes and signing wouldn't be an issue, the bottleneck would be the communication channel. 

 

There is another way to make multiple connections between peers rather than adding more peers. Could you please try to make a folder containing large amount of files (enough to make a speed measurement and a graph) sized from 1 to 2Mb? In this case both peers would transfer files simultaneously using several connections.

 

Thanks!

[Guest proactiveservices]

I'm getting really low speed via LAN - usually not much more than 500Kbps. Should I add my two pennies worth on this thread (which seems to be about WAN) or start anew?

[RomanZ]

proactiveservices,

 

As it fits the topic name, join the thread. Could you please share more details? Devices, OSes, versions of BTSync, type of data synced (at least - avg size and amount of files)?

 

Only one suggestion I can make without having info above - try turning on "lan_use_tcp" advanced setting to force using TCP instead of UDP in your LAN.

[Guest proactiveservices]

I just went to collect relevant network information and other tests and found out that my switch is misbehaving, causing in terrible LAN speeds. Don't I feel stupid? :-)

I shall now go and enjoy a slice of humble pie.

[crazyk4952]

crazyk4952,,

 

Few comments while we are working on reproduction.

1. BTSync version you mentioned is 1.2.2.2, while latest available is 1.2.82 (and it has a bunch of speed optimization). Could you please upgrade and see if issue persists?

2. There is another good test I'd like to ask you to do. Could you please try to sync with all the same parameters a folder with files 1..2Mbytes each, amount of files should be enough to make a graph. In this case BTSync is going to maintain different amount of simultaneous connections.

 

Thank you for your help.

It looks like I was wrong about the version number that I had. I am actually using version 1.2.82.

 

I have not had a chance to sync files that are 1.2MB yet. I will post a graph of this when I have a chance.

[antol111]

Isn't testing with a bunch of small files a different scenario since BTsync probably isn't breaking those files up into separate segments/blocks/pieces since they're so small?  I know that when I have a folder with 75 zip files, each around 50MB, the slow/fast pattern occurs.

 

Could this be an ISP throttling issue?  I'm pretty sure I noticed the same pattern when I setup a private bittorrent tracker to share a file between my two windows computers.  If so, how do I get around it?  I need to do some more testing and report back.

 

Also from what I noticed, BTSync will not download more than one file per folder.  If I add one new folder with 50 files it will only sync one file at a time.  If I add two or more folders then BTSync will download multiple files at the same time but still only one file per folder.

Edited January 31, 2014 by antol111

[sty]

I have a similar issue, speeds being very inconsistent since I started using btsync in September (I think).

Running currently 1.2.82.

 

Both ends are linux servers and have very big files (30G+ each) on the sync folder. There's about 9000km of physical distance between the boxes but they have both static public IP's and no rate limiting/qos on. 

 

I did some testing and found out that if I have a folder inside the sync folder that has multiple files of different sizes between 3-15MB each, whenever btsync syncs sub-10MB files it send 2 files concurrently, resulting in more constant speed. Whenever it was time to sync a 10MB+ file it only did this on a single thread, causing the bandwidth to throttle after every 10 seconds or so and then pick up again until it throttled it again. On a network utilisation graphs this looks like a 'saw' pattern that was on the first post(s).

 

When the sync moved on to a folder where I have these 30G+ files, it will never transfer more than a single file at the time which is very frustrating and underutilises the bandwidth available a lot. If I restart the btsync process on the sending side, it picks up (for sending) one of the files not synced yet randomly and continues normally.

 

It would be nice to have a setting where we can set how many concurrent connections, threads or blocks to try to handle at the same time, going into tens of blocks per file and multiple files at the same time should be configurable.

[RomanZ]

Isn't testing with a bunch of small files a different scenario since BTsync probably isn't breaking those files up into separate segments/blocks/pieces since they're so small?  I know that when I have a folder with 75 zip files, each around 50MB, the slow/fast pattern occurs.

 

Could this be an ISP throttling issue?  I'm pretty sure I noticed the same pattern when I setup a private bittorrent tracker to share a file between my two windows computers.  If so, how do I get around it?  I need to do some more testing and report back.

 

Also from what I noticed, BTSync will not download more than one file per folder.  If I add one new folder with 50 files it will only sync one file at a time.  If I add two or more folders then BTSync will download multiple files at the same time but still only one file per folder.

 

When you are sending files below ~12 megs, BTSync is going to open several connections instead of only one and transfer several files simultaneously. Standard piece being send is around 1 meg, so these files would consist of at least 2 pieces. 

 

I suspect it could be ISP throttling. For now it looks like someone (router? ISP? some relay server) intentionally drops a packet or tho every time period and limiting the traffic by doing so. If this is true, then opening several connections is going to occupy full bandwidth.

From other side, if there is a bottleneck on, say, your router because of amount of UDP packets - splitting connections is not going to help as amount of packets should stay the same.

 

I did some testing and found out that if I have a folder inside the sync folder that has multiple files of different sizes between 3-15MB each, whenever btsync syncs sub-10MB files it send 2 files concurrently, resulting in more constant speed. Whenever it was time to sync a 10MB+ file it only did this on a single thread, causing the bandwidth to throttle after every 10 seconds or so and then pick up again until it throttled it again. On a network utilisation graphs this looks like a 'saw' pattern that was on the first post(s).

 

[...]

 

It would be nice to have a setting where we can set how many concurrent connections, threads or blocks to try to handle at the same time, going into tens of blocks per file and multiple files at the same time should be configurable.

sty,

 

Thanks for the information. In your case it looks like ISP is limiting the speed. I'll check how (and if) this can be workarounded. Also, thanks for the proposal - it's worth of posting in wishlist forum

[antol111]

Ok so I did another quick test and I do not think it is an ISP throttling issue.  This fast/slow pattern only happens when multiple locations are syncing at the same time.  I can sync any two computers in any combination (i.e. A to B, or A to C or B to C) and btsync speeds are constant.  When I add in the third computer (A syncs to B and C) the fast/slow issue starts happening so it has to be something with BTSync and how it handles this scenario.

[crazyk4952]

The fast/slow pattern happens to me when syncing only two computers. I have tried setting up a Virtual Private Server through DigitalOcean and the problem still happens, so I doubt it's an ISP issue for me, too.

[RomanZ]

Guys,

 

Anyone else with "saw-shaped-traffic" issue managed to do the test with 1-2megs numerous file sync between 2 peers?

[crazyk4952]

Guys,

 

Anyone else with "saw-shaped-traffic" issue managed to do the test with 1-2megs numerous file sync between 2 peers?

 

Sorry this took so long to post.

 

Here is a graph of a few hundred 1.5MB files being synced between two peers: