Maximizing Speed In An Enterprise Environment

[johatfie]

I'm looking at an enterprise application for Bittorrent Sync on our LAN.  Every 18 hours, we've got a new 70GB build that has to be simultaneously copied to 6-12 other machines from a single file server.  The slowest link we have among all the machines is 2Gb and we are upgrading our network to 10Gb everywhere but on the actual machines themselves.

 

I've looked through the forums here for ways to make the transfers go faster.  I've disabled encryption.  I've set disk_low_priority to false.  I'm getting direct transfers to all the machines. I tried setting the send and receive queues to 500 on all the machines.  Though, that may have been ignored, since apparently only values between 1 and 100 are valid.  If so, the gui interface should have told me that when I tried to enter an invalid value.

 

In any case, we want the transfers to happen as fast as the hardware will allow.  We don't care how much CPU, memory, network, or other resources they consume.  We want them to happen as fast as the hardware can go.

 

Is there any thing else I can do to get Bittorrent Sync to be more aggressive? 

Edited May 5, 2014 by johatfie

[RomanZ]

@johatfie,

 

How much of your bandwidth is occupied? Do you see some obvious bottlenecks - like, CPU raises to 100%?

 

Usually, BTSync tries to occupy whole bandwidth, but if it sees that it interferes some other application it frees the channel. Is there any other apps running and utilizing a lot of bandwidth?

[2disbetter]

Would Dropbox, Onedrive, or the Transporter clients cause BT sync to ease up on its bandwidth consumption?

2d

[RomanZ]

@johatfie

Just another idea - the bottleneck might be your HDD. If you've got 2Gb LAN connection, it is roughly around 240 Mb/s bandwidth, which is nearly 2.5 times faster than your HDD allows.

 

@2disbetter

Yes, it might happen.

[johatfie]

I was seeing around 5% or so of the bandwidth occupied.  Robocopy uses up to 5 times that much.  No obvious bottlenecks.  Nothing else was running at the time.  And I've been able to get multiple instances of robocopy to use up to a sustained 26% of the available bandwidth.  I suspect that is a fast as the drive controller can go.

 

As far as the speed of the hard drive, I have yet to determine that.  Most of our machines have a data "drive" that is a 1+0 raid array of 23 drives (I believe one is a spare) and all the drives are 10k rpm dual port SAS drives.  And, as I said, I've been able to get up to 5x the speed with robocopy.

 

Now, about Bittorrent Sync "playing nice" with other applications, is there a way to turn that off?  For it to be useful to us, we need it to use as many resources as are available regardless of any other demands.

 

Thank you,

Jon Hatfield

Another question.  Bittorrent sync seems to use a predefined order for synchronizing files.  Is there a way to randomize that?  What I saw in my testing was that two peers initally were downloading from the file server and each other, but after a time they stopped syncing from each other because they already had all of the other peer's files.  So, after a time, they were only downloading from the file server.  That limits it's usefulness, but I know that it is an easy fix, perhaps for your next build.

[RomanZ]

Just a 5% of a bandwidth - it is definitely not a HDD bottleneck. And BTSync is not that nice, it should try to consume more of your bandwidth.

 

Unfortunately it looks like you encountered a speed issue we are trying to resolve right now: in rare occasions BTSync does not get the whole bandwidth with no obvious reason. I suggest to keep an eye on forum topic below - I'll publish news about speed issue resolution over there. 

 

http://forum.bittorrent.com/topic/25217-inconsistent-speed/page-2?hl=%20inconsistent%20%20speed

 

Answering your questions:

 - "Playing nice" to other apps can't be adjusted. But, as I mentioned above - it is not that nice, you have a different issue.

 - BTSync uses the order of files defined by OS itself. Thanks for the randomization idea - we'll consider it for future releases.