Send And Receive Buffer Sizes, Transfer Speed

[no1453]

I've been trying to get BT Sync to transfer files quickly between machines on my LAN ever since it was first made available.  But speeds have been quite erratic.

 

Finally I found a setup that is performing well, and I just thought I'd share it.

 

Not sure how vital these changes were:

   I enabled "Use predefined hosts" on both machines, set "lan_encrypt_data" to False. (Both machines)

 

What seemed to be vital to getting speeds up: finding the right values for "recv_buf_size" and "send_buf_size".   With the defaults (10/10), it kept settling at ~1 MB/s.  So I tried 32/32, but the improvement was minimal.  I then tried out 24/24, and the results were dramatic:  I started getting 6-10 MB/s.

 

The particular values might be specific to my setup, but with experimentation I'm sure others can find their own "butter zone" values.  The main lesson I learned that applied to BT Sync buffer size was: bigger isn't always better.

[trevellyan]

What seemed to be vital to getting speeds up: finding the right values for "recv_buf_size" and "send_buf_size".   With the defaults (10/10), it kept settling at ~1 MB/s.  So I tried 32/32, but the improvement was minimal.  I then tried out 24/24, and the results were dramatic:  I started getting 6-10 MB/s.

 

Are you syncing mostly large files, or mostly small files? The only time I see performance far below available bandwidth is with lots of small files, and it's still much better than every alternative I've tried.

[no1453]

I've run into problems with other transfer methods with lots of small files, so I usually bundle the data into a monolithic rar file. FTP behaves poorly (not reaching bandwidth available), Http does very well, but I haven't felt like installing a server on all my machines; simple drag and drop over the network is unreliable with large files.

This particular transfer was from a Windows 8.1 machine to a Mac Mini with wired connection through my router, maybe the cross-platform difference caused some of the "slowness" (though 1.1 MB/s isn't completely slow).

With default buffer sizes, it would transfer quickly for the first few minutes, but then slow down to 600KB/s-1.1MB/s, it seemed that the presence of other network traffic (Netflix, for example) would put BT Sync "off its game".

I have had good speeds transferring over the Internet, but on my LAN the only thing that gave me top speeds was changing the buffer sizes.

(And my impatience might be a factor, I've noticed lately it seems to take a while for BT Sync to ramp up to speed.)

[trevellyan]

I've run into problems with other transfer methods with lots of small files, so I usually bundle the data into a monolithic rar file. 

 

Interesting. Mostly I sync lots of small files, and for those my LAN rates tend to hover around 1MB/s with Sync. When I sync large files, I easily get into double digit MB/s. Most of the time at least one peer is on WiFi (802.11n). They're all Macs except for the obligatory always-on Linux box. All settings are default except folder_rescan_interval and UPnP port mapping.

 

I'm comparing with other automatic synchronization applications, not direct transfer, FTP or similar. Most worked OK with large files, but they all bogged down when I hit them with thousands of small files all at once. Some were orders of magnitude slower than Sync in that scenario.

 

it seemed that the presence of other network traffic (Netflix, for example) would put BT Sync "off its game".

 

I would regard that as by design, Sync playing nicely with higher priority traffic. I believe the protocol is designed to do that. In fact, at one point I was limiting upstream bandwidth for fear of saturating my connection, but I stopped doing so and it never seems to be a problem.

Edited November 2, 2014 by trevellyan

[RomanZ]

@no1453

That's really interesting!

 

In general, increasing buffer size should help with the speed for some time if you have some other processes loading HDD intermittently. I wonder, if you can also set "hdd_low_priority" advanced preference to "false" and see how it affects your performance?