Theory Of Operation: How Does The Upload Balancing Work?

[JimKnopf]

Hi,

 

can some of you explain to me how a btsync client decides what upload bandwidth will be used to the other members sharing the same secret?

 

Background: When my team (consisting of 5 clients) shares data over the internet we can see at the seeder client, the system with the lowest upload bandwidth gets the most bandwidth from the seeder. "iftop" is a nice tool to check this on a Linux client.

 

We have here the effect while transferring large data files that the clients with the smallest upload capability get ready first and the systems with largest possible upload bandwidth finish at last and this in a strict order!

This is obviously the wrong order since it would be of course optimal to fill the client with the highest upload capability first so this resource can be fully used while distributing the data instead of waiting for the clients with the smallest capability to upload data to the others.

 

But perhaps this simply bases on the understanding of the algorithm used and can be influenced somehow, so that's why I'm asking this question, here.

 

Greetings from Germany

[JimKnopf]

Perhaps I'm alone with my question and the most people don't care how this works.

 

In case you're also interested in the answer to the questions I placed, please, express you curiosity by making a simple post to this thread.

 

Thx in advance,

[jaithn]

+1

[RomanZ]

@JimKnopf

 

BTSync does not do any load-balancing. Maybe in future we'll do some load-balancing adjustments, but now the traffic goes as fast as it can

 

In your post you correlate the upload bandwidth to the fact how quickly the peer gets data. Does it correlate in some way with download bandwidth? It actually should have more influence.

[JimKnopf]

Hi RomanZ:

 

Many thanks for your answer. No the download bandwidth cannot be a limiting factor since even the slowest internet connection has 14 Mbit download bandwidth capacity whereas the fastest internet connection only has 10 Mbit upload bandwidth.

 

But your answer let me check the latency between my internet connection and the others. And really:

 

The slowest internet connection shows the best ping round-trip times (used average of 100 pings). The member with the fastest internet connection showed the slowest ping response times.

 

Comparing with the behaviour seen, the client order of getting finish with a very large file synchronization exactly matches with the order of ping latencies between the seeder and the clients.

 

Sounds logic and is clearly a correlation but the effect is quite dramatic in real. Would be very cool and helpful if one could influence the upload bandwidth of the own btsync client to other clients.

Best would be if the protocol would automatically determine the upload capacity of the other clients and trying to balance the local upload to them so the data distribution gets done as fast as possible.

[rtrpm-bittorrentsync]

+1

 

I wondered this myself because I have 5 clients and often I'm creating large files on one client for sharing out.  I see the same sort of thing in the upload distribution.

 

I was also wondering how a file was uploaded.  For example, do the other devices ever demand the same chunks of the new file so causing the seed device to send out duplicate data?  Or do they somehow sort out different chunks?

 

RTRPM.

[RomanZ]

@rtrpm-bittorrentsync

 

If BTSync client has multiple known peers, he'll request different chunks of data from them. The only case when it can ask for the same piece of data from another peer is if it was unable to receive a piece (damaged, peer closed connection, etc.).

[JimKnopf]

@RomanZ:

 

I understood rtrpm-bittorrentsync that way that we he wants to avoid the situation that the seeder (the client with original file) has to send the same data chunks to more than one client in the network.

 

I totally agree with this since this would be an optimum process with highest efficiency. But if the current logic of btsync just base on a pure polling mechanism where a client without the data dictates what chunks other clients have to send than this cannot be guaranteed.

 

This in fact is the behavior I can see. In a perfect network the seeder would have to send the amount of data of a file only once by distributing the pieces in a disjunctive way to the clients in the network and these clients exchange the missing data among them without the seeder.

With current implementation the seeder runs much longer than the required minimum of a single transfer time of the file.

My experience is that if you compare the time a seeder would need to distribute a file in a pure sequential way (so any client would get the data by FTP i.e.) with the runtime btsync currently offers then the time saving is around 30% - 50% ... but this very depends on the file size and number of files to distribute.

Edited July 15, 2014 by JimKnopf

[coreycauble]

@JimKnopf I agree 100% that would be the ideal situation.

 

@RomanZ Would it not be possible for the seeder to service only those requests that would meet the goal of no duplication of chucks to its peers until a full distribution of the file is achieved. Then service the repeat requests? because in theory the client peers are should be requesting all of the chunks anyway, particularly when it is an initial seed.

 

On to my question: I would like to know if during an initial seed the leaches do not share chunks. In the tests I have done chunks from multiple sources do not appear until they have finished downloading the entire file. 

 

It would be great if we could get a representation in the clients of what chunks are being moved much like standard bittorrent clients as it would vastly improve debugging.

[JimKnopf]

@coreycauble: You might have been run into this: http://forum.bittorrent.com/topic/29870-suboptimal-performance-btsync-1394/

 

Workaround for me: Until this is not changed/fixed use small files instead of large ones. 7-zip is a nice tool to split such files into small parts.