Practical File Count Limitations - 100M Files

[quantkiwi]

I have 100M files to sync between servers over the internet, is this feasible with BT sync? 

 

Is there a practical file limitation? How long do you think a sync like this would take (assuming a solid internet connection)

 

[Moe]

The only limitation would be RAM... IIRC sync uses 1KB per file in RAM.

But sync should definitely be able to sync that much data...

[GreatMarko]

IIRC sync uses 1KB per file in RAM.

 

It's actually a little lower than that - around 300-400 bytes per file

 

Is there a practical file limitation?

There is no "enforced" limitation on the number of files you can sync, however, it will be affected by factors such as free disc space, available memory, etc.

 

I have 100M files to sync between servers over the internet ... How long do you think a sync like this would take (assuming a solid internet connection)

It's impossible to say, especially without knowing things like the size of files - as 100 million 1KB files would take a different total time to sync than 100 million 100MB files(!). Generally speaking though, it's quicker to sync when the two devices are on the same LAN, rather than across the internet, so you may achieve better sync speeds if you setup a VPN between your two servers so that they appear on the same physical network.

[quantkiwi]

Thank you for your quick replies -- 

 

The files are roughly 1kb-1mb, average is 10kb. 300 bytes per file is fine as long as it doesn't need to hold them all in memory at once? Does it should move through the list of files one by one? 300 bytes * 100M files is 30GB of ram needed :\

 

I don't mind if the initial sync takes a few days, but I add to the "master" server every day with 200mb of files, which I hopefully want sync'd to all the slaves within 2-3 hours.

 

Thank you for your help,

Best

Edited October 8, 2015 by quantkiwi

[RomanZ]

@quantkiwi, @GreatMarko

I'm afraid it is closer to 1Kb per file in latest versions. So, it would be approx 100Gb consumed to keep the tree of all files in memory. Sync will do its best to keep these files in Sync, although I can see couple of obstacles which will won't let you get desired 2-3 hours sync delay to all devices:

 

1. Memory. Either you need to get really huge volume of memory, or set up humongous swap file. If swap won't get to the SSD drive - it will be really slow.

2. Sync gets info about changed files in 2 ways: notifications from OS and via folder rescan.

 - Notifications from OS are unreliable way. They have a number of limitations (different for different OS) - like, Windows won't send you notifications for the very deep folders, while Linux may miss it if you got large amount of subfolders.

 - Rescan heavily depends on HDD speed. While rescanning Sync only reads file names, mtime and size, though this is a random access and it loads a lot classic HDDs (while is still pretty fast for SSD drives). And you can't make rescan period lesser than at least time of rescan, of course.

[kamborio]

Hello,

 

I am syncing 1.433 million (234GB) using 1.8 GB RAM with version 2.2.5

 

So far so good ( thanks @RomanZ !)