silkygreen

Crash Trying To Sync Millions Of Files

Recommended Posts

Hello,

 

I'm migrating 4.6 million files (125 GB) from one server to another using BT Sync. I've made it to about 75 GB and it's now crashing on one of the servers. The "old" server where it's crashing while trying to index the directory is Windows Server 2003. The new server is Windows Server 2012 and it's not crashing (yet).

 

The crash is a non-specific Windows error about the program blowing up. No other programs are affected.

 

Any suggestions for solving the crash? Any suggestions for an alternate tool?

 

Thanks.

Share this post


Link to post
Share on other sites

How much free memory does your server have? 4.6 million files would require nearly 2GB of free memory in order for Sync to process (memory usage typically increases by around 300-400 bytes per file being monitored by Sync)
 

Any suggestions for an alternate tool?


You may be far better suited with a disc "cloning" tool if all you're doing is a one time "move"/migration of files to a new server, rather than actually wanting to keep 4.6 million files in real-time sync between devices!

Share this post


Link to post
Share on other sites

How much free memory does your server have? 4.6 million files would require nearly 2GB of free memory in order for Sync to process (memory usage typically increases by around 300-400 bytes per file being monitored by Sync)

You may be far better suited with a disc "cloning" tool if all you're doing is a one time "move"/migration of files to a new server, rather than actually wanting to keep 4.6 million files in real-time sync between devices!

 

Ahh, I bet it's a memory issue. The old server is a bit memory-strapped.

 

It's looking more and more like a zip-like-tool is going to be what's needed. Thanks for the tip about memory--I bet that's it. Now I know I should pursue another route.

 

Originally the intent with using BT Sync was that it could start weeks in advance and keep up with changes so that when deployment day came, it was basically done and up to date on this directory. And it was great for getting around the firewall.

Share this post


Link to post
Share on other sites

You could also zip the many small files into a single/few large, and then sync that via BTSync.  It should reduce memory usage quite a bit. 

 

Yeah, that's my fallback if my latest process doesn't work. Unfortunately I'd like to avoid it because the sheer number of files will make zipping (and then unzipping) a very time-consuming process. Thanks for the thought, though.

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.