Sign in to follow this  
Followers 0
ms2oo8

Syncing over 1 million files

2 posts in this topic

So we've been using Sync since the 1.4 days and had experienced some frustrations as we learned how to deal with the ins and outs of syncing such a large amount of data. Many of our initial issues were caused by inadequate amount of CPU / RAM resource needed to sync this amount of files as well as a bug in indexing SMB shares. Initially we had tried installing Sync on our severely under powered NAS boxes with 512Mb of RAM. Instead we now have a dedicated desktop machine with 8GB of RAM indexing and syncing each of our 3 peers.

We're currently running 2.4.2 and syncing 4.21 TB of files across three geographical locations. It is working very well and we're really happy with it. It allows the CSR's at our multi-location print / graphic design company to always have the latest graphic files regardless of which location last editing artwork. We're able to spread out our workload across whichever staff at whichever location have time vs assigning projects based primarily on location of artwork files.

The total amount of files we're syncing is 1,347,334 files in 87,698 folders with the file size around 4.21 TB of data.

 

files.JPG

Share this post


Link to post
Share on other sites

It's good to see someone talking about a large amount of files, and highlighting that 8GB RAM is required for this amount of data.

I have a question though, are you only using Windows based machines? We're currently sync around half the files you have (2TB), mainly on OSX but we seem to be hitting some temporary issues with Sync picking up and trying to reconcile com.apple.resourcefork files.

I've read that i can comment this out in the StreamsList file to stop this from happening but i'm not sure if it's safe to do so. Wondering if you've come across something similar?

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now
Sign in to follow this  
Followers 0