Can You Sync Among Local Folders?

[coewar]

Not sure if this would be new feature or not.

 

But situation is that on Amazon instances, to keep your data you use EBS volumes, but when you spin them up, they also come with instance volumes that disappear when you stop the server. However those instance volumes are "free" and much faster because they are local disks, and also sometimes are even SSD.

 

So my thought was to have BTsync run on the instance syncing with a folder on the instance volume and on the EBS volume (appearing as just different drive letters in Windows). When I run applications like coding and building stuff, I can do that on the fast instance store, and then BTSync can catch up when it can to update the EBS store.

 

 

When I add 2 local folders with the same secret, BTSync complains, so I guess this is not possible at this time. But sure would be nice for this situation.

 

Even for other users to be able to quickly backup among other locally attached drives (internal or external) and not deal with proprietary and maybe slower software for that.

 

It would also have to be able to create the folder if it does not exist upon start up. Because when the instance is stopped, that whole partition disappears. When it's started, it's a blank slate, so BTSync would have to create the folder and then start syncing. Some of that we can script of course... but the limitation here I think is not being able to add multiple folders with the same secret.

[lolcat]

Why would you use BTSync for this? rsync would do just fine, wouldn't it?

You could run two instances running on different ports, but why would you? There are better tools for the job.

BTSync is properitary, so it would be a bad option if avoiding that is a goal. Rsync is open-source.

[coewar]

See here's where the "why would you" comments start coming. If you bring up 'rsync' then heck, I could use that to completely replace BTSync also. But that's not what I want to do.

 

Of course I could use bunch of other tools. But imagine how quick and easy it would be to just use 1 tool.. and I wonder if it's an unnecessary limitation in BTSync created by someone on purpose who simply "didn't think anyone would do that".

 

Because it's a lot nicer to just have BTSync. Add another local folder with the same secret. Bam.. you're done. No need to mess with some other application... 

not to mention that I'm talking about a Windows system... rsync for Windows is not quite there from what I can see.

Besides the point anyway.

I guess I should add.. that in the scenario we also use BTSync to sync with other servers elsewhere... in and out of Amazon AWS. So hence.. since a good service like BTSync should not try to guess how it's going to be used, and just focus on doing its job well, superficial limitation like not allowing the same secret on the same server could be avoided.

 

But .. maybe the reason is that doing so adds some complexity and so it just can't handle this now.

[lolcat]

The point is using BTSync for replication of folders on one computer would waste resources. It uses more CPU cycles, more ram, and more I/O than rsync would. It is like asking how you best can use scissors to mow your lawn, any sane person would tell you scissors would not be the right tool for the job, and would waste time and give you a sub-par result.

 

I don't use Windows, but I am sure it has some sort of software to synchronize two folders that would have far less overhead than BTSync. The bitorrent protocoll is good for distributing files among many nodes, this is where it has advantadges. It allowes one person to share a file, or a set of files with an unlimited number of people, and in the best case scenario it won't cost more than the bandwidht to upload it once. To replicate files locally however it would just waste resources, if you share has encryption you have to encrypt each file, add it to a database, send the piece, decrypt the piece and then write to the drive. This is so incredibly wastefull.

Ofcourse you can argue BTSync should do everything, but that would be pointless, you need simple efficient tools that does the job they are designed.

The complexity wouldn't matter, all you would have to do is to use the built in copy command of the OS, hardly a complex operation. It would probably require less disk I/O, ram and cpu cycles to just delete the old folder and then copy the new folder.

[cod3monk3y]

I haven't dealt with EBS, but I understand wanting a program that mirrors folders *across systems* to possibly mirror them across *drives* for something like backup. That's not really a far fetched concept.

 

As a work-around what if you spun up some sort of VM that mapped the new drive and ran on a different IP, with the dedicated purpose of just running a second BT Sync instance? Then add that to the know hosts of the main server, or it may find it automatically with DHT/peer discovery. 

 

But in your scenario are you concerned about partial deployment? That is, some software trying to run on a partial built file system? Like an automated build that starts to run (like Jenkins) while only half of the code is synched? We have this scenario in our deployment. We're deploying media to 50 computers, and they can't run on partial sets of audio files, DLLs, animations, etc. So I've set up a sync folder and am using robocopy on windows to do a full copy when I know the system isn't in use (nightly). We also do log file backup from the 50 computers that needs to happen when the application is shut down.

 

Others have mentioned this problem on this forum in regards to deploying web servers with BT Sync, and that it's a bad idea (because of the partial nature of the sync). Rather you'd want to shut down the server for a minute, deploy, then restart the server. (or of course, deploy to a second instance and remap your virtual server to the new deployment to reduce downtime).

 

Using something like a script to copy (as suggested) would allow you to script synchronously, and I don't think you can get that functionality out of BT Sync.

That is, of course, if this is at all applicable to your situation. 

 

Cheers,

/cm

[coewar]

I have accomplished what I was trying to do using this tool: (http://freefilesync.sourceforge.net/)

It's very fast and can do realtime sync in a rather flexible way. It does not seem to do block file copying though.

 

My scenario is AWS instances when started spin up with a blank instance store. Those instance stores are faster than the EBS volumes and sometimes even SSD so they are really fast. When it's running, I want to use the instance store for things like development work or whatever. But of course I want it to keep the backup on the EBS volume which is permanent. 

Optionally, I also wanted to use BTSync so that from let's say the EBS volume, I can also sync with another computer (or two) I have elsewhere. This set up allows me to get up and continue working where I left off with or without a persistent internet connection. Sometimes I might be RDP'ed into an AWS server, sometimes my desktop, sometimes my laptop while sitting in a car or waiting room without internet.

 

So now when I start my AWS instance, by the time I RDP into it, my files are already copied to the blank-slate instance store, and changes to either location I have configured to sync about every 5 seconds. I even sync the .git folder and have tested several actions using Git in one folder and having it appear in the other.

 

The fact that BTSync had bunch of overhead of encryption, etc.. is something that I can imagine being bypassed when it knows it's doing a local sync. No reason why it can't choose the right path based on it being either local or remote.