Amazing Idea - But It Must Use Less Memory For 1,000,000 File Sets

[Bonsai]

I've spent about a week playing with Sync, and its almost perfect, there are 2 things that hold it back for me right now:

 

1) Memory usage grows with the number of files of being being tracked. This makes it a hog on my desktop machines, and makes it quit on my little ARM NAS (GoFlex)

2) Multiple locations on one machine for the same Secret, e.g. an internal drive, and a removable external drive, when you plug the external in, it syncs with the internal.

 

There are couple more nice to haves too:

3) Subdirectory shares, being able to share part of larger sync, this is where DropBox is very useful, but a read only sub-share in Sync would be excellent.

4) Encrypted stores - the ability to setup a share that doesn't decrypt the files.

 

Right now, I've limited Sync to smaller file sets, that really DropBox or SkyDrive were doing an okay job of.

 

Keep up the good work!

 

[GreatMarko]

I've spent about a week playing with Sync, and its almost perfect, there are 2 things that hold it back for me right now:

1) Memory usage grows with the number of files of being being tracked. This makes it a hog on my desktop machines, and makes it quit on my little ARM NAS (GoFlex)

Remember that Sync is currently in "Beta", and memory/cpu consumption will likely be improved before Sync leaves "beta". It has already been improved on a number of occasions with new Beta builds, but then subsequent Beta builds can have a higher memory/spu consumption. This is to be expected with "Beta" software, but will likely become more stable as the beta progresses.

 

2) Multiple locations on one machine for the same Secret, e.g. an internal drive, and a removable external drive, when you plug the external in, it syncs with the internal.

This has been discussed in a number of threads, but the general feeling from the developers seems to be that Sync is primarily a file synchronization tool to keep your files synchronized between devices - it's not primarily a "file copy" or "backup" utility. Presently, if you wish to sync between an internal drive and an "external" drive, your external drive must be a compatible NAS drive.

 

3) Subdirectory shares, being able to share part of larger sync, this is where DropBox is very useful, but a read only sub-share in Sync would be excellent.

"Selective Sync" (already possible in the mobile apps) is also planned for desktop editions of Sync

 

4) Encrypted stores - the ability to setup a share that doesn't decrypt the files.

This functionality - encrypted nodes - is already present in Sync - although currently only via the API

[lolcat]

Remember that Sync is currently in "Beta", and memory/cpu consumption will likely be improved before Sync leaves "beta". It has already been improved on a number of occasions with new Beta builds, but then subsequent Beta builds can have a higher memory/spu consumption. This is to be expected with "Beta" software, but will likely become more stable as the beta progresses.

 

I am a bit uncertain if BTSync uses a modified or stock version of the Bittorrent protocol, if it is the sock version that would require a computer to encrypt all the files to create the torrent file, in this case creating a share would always be limited to the speed your CPU can encrypt something with AES. It seems unlikely the developers can somehow revolutionize how AES is done. The process of creating the share would be something like this: Create bootstrap torrent (contains a sqlite database of all the files and their respective torrents) -> encrypt a file inside the share -> make the torrent hash -> update the bootstrap torrent -> rinse and repeat until all folders and files are added.

If they use some sort of modified version they would probably get away with something like: Create bootstrap torrent -> create torrent for each file  -> update bootstrap torrent.

It is also possible they never update the bootstrap torrent but rather add new torrents for new files, exactly how this is done is unclear. If it is the latter method you won't hit the AES overhead until someone requests the files, if it is the first you would have to encrypt it first, and then again every time someone requests it (this can be avoided by having to sets and using the encrypted read-only secrets).

 

 

This has been discussed in a number of threads, but the general feeling from the developers seems to be that Sync is primarily a file synchronization tool to keep your files synchronized between devices - it's not primarily a "file copy" or "backup" utility. Presently, if you wish to sync between an internal drive and an "external" drive, your external drive must be a compatible NAS drive.

 

In the beginning I'd disagree with this, but since the encrypted read-only keys exists I feel adding backup to BTSync would be pointless. This doesn't mean using it along with a backup program couldn't be done in a good way. Simply use any incremental backup tool to make a backup to a computer on the network, and then use BTSync to replicate that backup to several off-site locations. This solves the biggest problem with backups today, the lack of bandwidth. With CrashPlan I get terrible upload speeds, and to upload to several locations I'd have to use my slow home connection to upload my backup several times. With BTSync my off-site backups could simply sync with each other, thus reducing my bandwidth needs considerably. 

To use BTSync to copy files seems like a terrible idea.

 

This functionality - encrypted nodes - is already present in Sync - although currently only via the API

 

Or you could spend a little time and figure out how the secrets are generated and then make your own.

[Bonsai]

Thanks for the reply Marko, and thanks for all your efforts at community management on these boards, whenever I was stuck it was generally your posts that got me going.

 

The memory usage on the NAS, causing btsync to quit, is currently stopping me from going any further with this, I will take another look when btsync exits Alpha.

 

Happy new year, excited to see this product evolve in 2014.

[lolcat]

How much ram is it using? My freeBSD box has a share containing somewhere between 2 and 3 million files. It stopped at 1.44m files, and has been using something like 6-700M ram ever since. Have you considered adding some/more swap?

[skiguy]

It is a great tool, and I think many people want to use it as a back up tool.    

 

DropBox was made by someone who kept forgetting his flash drive at home, so he wanted his files everywhere. 

 

But most of us want our files backed up from a master computer to others.   We're replicatng the data in many places as safety nets.  

 

That's why you see people wanting to plug in a 2TB USB drive, test it, take it to another building (or home), plug it in, and have it work the same.   

 

I've got 500 gigs of pictures and music going back to 1998.   I replicate this data in multiple places.       I want to set up a USB drive at home, get it syncing, then take that drive to another house where it will sync when that computer is on.     That's why you see people wanting to sync with the same computer.  They're looking to set everything up with a master, then move a drive to another machine far away. 

 

This would be more like a Carbonite example, where they send you a drive, tell you to fill it, you send it back, then everything syncs.      

[lolcat]

I want to use my toaster under water in my bath tub, that doesn't make it a good idea. BTSync would be usefull to replicate a backup made by proper backup software, the software should have version control, the ability to check the integrity of the backup, and for compatibility with BTSync it should not change existing files as the version control is mediocre.

I would use a combination of for instance CrashPlan and BTSync, then you can create a backup to an external drive, and then seed it to ten people and then the backup uploading would be far smaller after the initial backup. Or you could send a drive with the backup to the person who has the most bandwidth and less him/her seed it.

A backup program that uses the P2P abilities of bittorrent and has the ability to share backups easily would be brilliant. CrashPlan lets you back up to friends, but you have to upload the backup once to each computer. This is obviously a terrible solution if you have limited upload bandwidth or a huge dataset to back up. Also being able to use several providers without wasting ram and bandwidth.