joepc

Syncing 600GB (10 million Files) - Recommend Settings?

Recommended Posts

Hi,

 

I have been trying to sync a windows directory to another server for a few days now and all the usual file comparison / sync apps fail due to the number of files.

 

I set-up BitTorrent sync and it appears to be working great but I'm a little concerned the default settings might not be ideal for this data?

 

Does anyone have experience of syncing a large amount of small files? LAN to LAN

 

Thanks, Joe

Share this post


Link to post
Share on other sites

I have sync'd 2 shares for a total of roughly 400gb (50,000 files), default settings worked fine. This was 1 site pushing data to 2 sites over internet, took a while, LAN should be much faster.

Share this post


Link to post
Share on other sites

From unofficial FAQ we know that "if you're monitoring/syncing 1 million files, you will need around 300-400 MB of free memory". BTSync is 32-bit, so, it can only use ~2 GB of RAM. Therefore, the maximum amount of files is limited to ~6 million.

Share this post


Link to post
Share on other sites

Hi Joe,

 

 

Regarding the default settings, you could turn off DHT, tracker and relay server and turn on search LAN, if you only intend to use it within a LAN, maybe even insert your hosts in the predefined hosts section.

 

@deem: I use the x64 version of btsync and this the output of file I get on the btsync executable, so I think BTsync is 64 bit capable on   supported platforms, thereby expanding the maximum number.

btsync: ELF 64-bit LSB executable, x86-64, version 1 (FreeBSD), dynamically linked (uses shared libs), stripped

Share this post


Link to post
Share on other sites

I have very good experiences for large volume sycncing with "Syncovery" (http://www.syncovery.com/)

 

Hi,

 

I have been trying to sync a windows directory to another server for a few days now and all the usual file comparison / sync apps fail due to the number of files.

 

I set-up BitTorrent sync and it appears to be working great but I'm a little concerned the default settings might not be ideal for this data?

 

Does anyone have experience of syncing a large amount of small files? LAN to LAN

 

Thanks, Joe

Share this post


Link to post
Share on other sites

@nils: Ok, BTSync for windows is 32-bit and the author said: "sync a windows directory". Make conclusions. The author can dance with other settings, but it won't help. Problem is solved. You can't sync so many files on 32-bit app.

Share this post


Link to post
Share on other sites

Thanks for the comments, after 50 hours it synced to 800K (90GB) files and slowed to a crawl - the process was consuming nearly 1GB ram so deem is correct. I will try Syncovery.

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.