Practical limits to BTS size/number?

[krtaylor]

I am using BTS for what amounts to a family backup network. We each have a large music collection, large collection of digital photos, and other miscellanous files wanting to be backed up. By establishing BTS folder shares, everyone's files are automatically backed up to each of the other nodes so a fire can't destroy all someone's data. We all already pay for broadband connections, generally leave our computers on 24/7, and hard drive space is cheap.

The trouble is, I am noticing that BTS proc and memory requirements seem to scale to the point where it's not actually capable of ever finishing indexing, especially now with each new version requiring a fresh indexing from scratch.

Currently we have 12 shared folders ranging from 5MB up to 150GB and 3 files up to 30,000. The total data synced is around 370GB.

Oh, and, most (not all) of the shares are read-only at all nodes other than their source. I still encounter the old bug of the last couple files not ever showing as synced, but rarely.

But the two newest shares always just show 0 for files and size (not true) as that node is relatively old and slow and never finishes indexing everything higher in the list. So we appear to have reached a practical limit of some kind.

a) Does anyone else out there deal in volumes like this, either in number of folders, number of files, or data volume?

Is anyone else running into this kind of limit? I assume there hasn't been formal testing of limits nor anything hard-coded?

[GreatMarko]

What version of Sync are you running? There were CPU & Memory issues in previous alpha builds. If you're not using the latest 1.1.26 build, please update.

Also, please see "Is there a limit on the number/size of files I can sync?" in the Unofficial FAQ

[Camaban]

Regarding updating to the latest version, it's worth mentioning that I'm running two backup VPS's with 512MB RAM each.

On each of these VPS's is 249,025 files taking up 334 GB

Largest single share is 300GB

Just checked one of the servers (Ubuntu), and it's using 384MB of RAM and about 20-30% CPU (Single core VM)

CentOS box is using roughly the same amount of resources. These figures include what the OS itself requires to run a fairly minimal install.

Running version 1.1.26 and I can't find much cause for complaint.

If you're running 1.0.X however, then it was truly awful when it came to resource usage. I was using my full RAM capacity plus 500MB-1GB of swap space.

[krtaylor]

I am using 1.1.26. All systems involved have a minimum of 2GB of memory and running Windows 7.

The oldest system is running about 75% memory usage but the procs are flat out - 100%. I'm not totally sure it's the fault of BTS but nothing else is running other than normal system services.

I'm noticing a new problem though. My nodes have mostly completed syncing, but they are now stuck on an endless loop that appears in the Transfer window. They each show the same 3 files transferring but with 0 for both upload and download. The window does change, as sometimes it says there are no transfers, other times 1, 2, or 3 of the files in question. There is other data that needs to be transferred but it never gets a chance. Unfortunately, it does not display the path of the files in question so I don't know exactly what they are.

I'll send in a log file shortly.

[kos13]

Thank you. We will need logs, since from your description this is easy case for Sync. It doesn't really matter how big are files, it is more about amount of files.

[Camaban]

My nodes have mostly completed syncing, but they are now stuck on an endless loop that appears in the Transfer window. They each show the same 3 files transferring but with 0 for both upload and download. The window does change, as sometimes it says there are no transfers, other times 1, 2, or 3 of the files in question.

When I think that, I had something marginally similar to this on two occasions.

Please note for both of these that I run Cygwin, it's always possible that I had manipulated my files in an unapproved manner.

First was a case of no read/write permissions on those files.

Second was a case of the date stamp being corrupt.

[krtaylor]

These are all Windows and have local admin rights so I don't think that's it, but it's certainly possibly a date got corrupted. Not sure how to fix that as the "Transfer" window does not display the full path so I've got no way to find exactly which file they are. Good thought thought, it may very well be connected.

[Camaban]

These are all Windows and have local admin rights so I don't think that's it, but it's certainly possibly a date got corrupted. Not sure how to fix that as the "Transfer" window does not display the full path so I've got no way to find exactly which file they are. Good thought thought, it may very well be connected.

Mine is a Windows box that I'm fairly sure was doing a one way transfer.

And the transfer window does display the filename. Should be able to search for it. Can't be too many files with the same name (You'd hope)

[marvinandro]

Just joined the sync party at v 1.1.48. To test, installed 2 of those on Win7-64 and set one directory to sync, then on the other computer used the read-only key to syn the same dir.

It's been 48 hours, two machines on the same network:

The "source" one says that another 800MB need to be transferred up out of 52300 files.

The "target" one says "Synced on Aug 2nd" (some time yesterday) but is has only 52100 files.

Any reason why I'm missing 200 files on the target? Hit a max/limit? Need to debug after basic test 1 on day2? If I need to debug, which side?