joepc

Syncing 600GB (10 million Files) - Recommend Settings?

Recommended Posts

Hi,

 

I have been trying to sync a windows directory to another server for a few days now and all the usual file comparison / sync apps fail due to the number of files.

 

I set-up BitTorrent sync and it appears to be working great but I'm a little concerned the default settings might not be ideal for this data?

 

Does anyone have experience of syncing a large amount of small files? LAN to LAN

 

Thanks, Joe

Share this post


Link to post
Share on other sites

I have sync'd 2 shares for a total of roughly 400gb (50,000 files), default settings worked fine. This was 1 site pushing data to 2 sites over internet, took a while, LAN should be much faster.

Share this post


Link to post
Share on other sites

From unofficial FAQ we know that "if you're monitoring/syncing 1 million files, you will need around 300-400 MB of free memory". BTSync is 32-bit, so, it can only use ~2 GB of RAM. Therefore, the maximum amount of files is limited to ~6 million.

Share this post


Link to post
Share on other sites

Hi Joe,

 

 

Regarding the default settings, you could turn off DHT, tracker and relay server and turn on search LAN, if you only intend to use it within a LAN, maybe even insert your hosts in the predefined hosts section.

 

@deem: I use the x64 version of btsync and this the output of file I get on the btsync executable, so I think BTsync is 64 bit capable on   supported platforms, thereby expanding the maximum number.

btsync: ELF 64-bit LSB executable, x86-64, version 1 (FreeBSD), dynamically linked (uses shared libs), stripped

Share this post


Link to post
Share on other sites

I have very good experiences for large volume sycncing with "Syncovery" (http://www.syncovery.com/)

 

Hi,

 

I have been trying to sync a windows directory to another server for a few days now and all the usual file comparison / sync apps fail due to the number of files.

 

I set-up BitTorrent sync and it appears to be working great but I'm a little concerned the default settings might not be ideal for this data?

 

Does anyone have experience of syncing a large amount of small files? LAN to LAN

 

Thanks, Joe

Share this post


Link to post
Share on other sites

@nils: Ok, BTSync for windows is 32-bit and the author said: "sync a windows directory". Make conclusions. The author can dance with other settings, but it won't help. Problem is solved. You can't sync so many files on 32-bit app.

Share this post


Link to post
Share on other sites

Thanks for the comments, after 50 hours it synced to 800K (90GB) files and slowed to a crawl - the process was consuming nearly 1GB ram so deem is correct. I will try Syncovery.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now