Generate encrypted folder locally

[tbdtbd]

I have a folder on my laptop and would like to back it up to a remote computer using bittorrent sync. The remote computer is untrusted and, thus, I will only install the encrypted key on it (which will give no read or write rights). This works fine when the encrypted backup is done to a remote computer over the network. However, I would like to be able to do it locally the first time because the amount of data to be backed up is large. The problem is that the bittorrent sync client does not let me create two versions of the same folder on the same computer (one with read/write rights and the other with only the encrypted key).

I contacted bittorrent sync support and they answered the client simply does not support having two versions (one encrypted and one in the clear) of the same folder on the same computer. Does anyone know if there is a workaround to do that?

Thanks.

[GreatMarko]

There's no real "workaround" as Sync is fundamentally designed to synchronize folders between devices. It's not possible to sync two folders locally on the same device.

[arin]

3 hours ago, tbdtbd said:

I contacted bittorrent sync support and they answered the client simply does not support having two versions (one encrypted and one in the clear) of the same folder on the same computer. Does anyone know if there is a workaround to do that?

Thanks.

During one of my experiments with last version of sync I have created a virtual machine and recreated the encrypted folder over there. However I do not known what happens when you copy and paste the folder on another device.

[ivarson]

1 hour ago, arin said:

During one of my experiments with last version of sync I have created a virtual machine and recreated the encrypted folder over there. However I do not known what happens when you copy and paste the folder on another device.

Obviously that will work, it's a completely different machine and the data will stay encrypted if copied elsewhere. However it would probably be more adequate to encrypt the underlying disk itself if possible. In windows 8+ it's  very easy to create encrypted vhdx (which indeed aren't bound to VMs) and performance might as well be improved too (just guessing though), you can also choose between aes 128 (like in btsync) or aes-256. 

[iswrong]

5 minutes ago, ivarson said:

However it would probably be more adequate to encrypt the underlying disk itself if possible. In windows 8+ it's  very easy to create encrypted vhdx (which indeed aren't bound to VMs) and performance might as well be improved too (just guessing though), you can also choose between aes 128 (like in btsync) or aes-256. 

Not really (if I understand you correctly). Disk encryption is only encryption at rest. If someone hacks into a remote machine, they can still see the unencrypted data. If someone hacks a machine that uses a BTSync read-only encrypted secret, they still can't see a thing.

Encrypted disks or images are for another threat model - e.g. when someone steals your laptop or desktop machine. It's probably locked, sleeping, or off and they cannot access the data (unless the thief is the NSA and they use cryogenic RAM freezing ;)).

[ivarson]

2 hours ago, arin said:

During one of my experiments with last version of sync I have created a virtual machine and recreated the encrypted folder over there. However I do not known what happens when you copy and paste the folder on another device.

Obviously that will work, it's a completely different machine and the data will stay encrypted if copied elsewhere. However it would probably be more adequate to encrypt the underlying disk itself if possible. In windows 8+ it's  very easy to create encrypted vhdx (which indeed aren't bound to VMs) and performance might as well be improved too (just guessing though), you can also choose between aes 128 (like in btsync) or aes-256. 

 

That's right of course. Although creating an encrypted vhdx with virtual smartcard technology and the vm having only a private nic against the host it is pretty hard to "hack"  that isolated vm since it wouldn't even have lan nor Internet access in any direction.  But yeah, you're still right :-p

[tbdtbd]

3 hours ago, GreatMarko said:

There's no real "workaround" as Sync is fundamentally designed to synchronize folders between devices. It's not possible to sync two folders locally on the same device.

Right, that is exactly what I want to do: to back up the data on my computer into an untrusted remote device. To do that, I need to first bootstrap the system. That is, I have to do the initial backup. When the amount of data to be backed up is large (e.g., several TBs), it makes no sense to do the initial backup over the network. It would take a lifetime. Instead, I would like to do it locally and copy it to the remote machine using, for example, an external USD drive.

Note that this type of bootstrap functionality is supported by regular backup systems. For example, IDrive offers the IDrive Express service, which is described as follows: "Quick backup and retrieval of your data up to 3 TB in less than a week via physical storage shipment, ensuring no bandwidth usage". Having bittorrent sync support this type of offline bootstrap would be fantastic because this functionality is the only piece missing to fully support untrusted nodes.

So, why is having two versions of the same folder (one with read/write rights and the other encrypted) on the same device not supported? Is this a limitation in the bittorrent protocol (e.g., the way folders and devices are identified on the wire), or is it a limitation in the client (i.e., because it checks that a given folder can exist only once)? If is is the latter one, it would be easy to fix it, I guess.

Thanks!

[tbdtbd]

3 hours ago, arin said:

During one of my experiments with last version of sync I have created a virtual machine and recreated the encrypted folder over there. However I do not known what happens when you copy and paste the folder on another device.

Yes, as a workaround I had also thought of using a virtual machine... or a container (e.g., docker), which would be a lighter solution than a VM. Nevertheless, if this was supported natively without the need to virtualize things to cheat the client, that would be really great.

[GreatMarko]

1 hour ago, tbdtbd said:

So, why is having two versions of the same folder (one with read/write rights and the other encrypted) on the same device not supported? Is this a limitation in the bittorrent protocol

It's a limitation of the protocol.

[iswrong]

9 hours ago, tbdtbd said:

Yes, as a workaround I had also thought of using a virtual machine... or a container (e.g., docker), which would be a lighter solution than a VM. Nevertheless, if this was supported natively without the need to virtualize things to cheat the client, that would be really great.

On Linux, it also works just by running sync instances using two different users. On our RPi, we have two instances of Sync running with a partially overlapping set of folders.

[tbdtbd]

11 hours ago, GreatMarko said:

It's a limitation of the protocol.

Interesting. Thanks for your answer!

I guess changing that part of the protocol would be a relatively big deal and it will not happen even if somebody submits a feature request, right?

Out of curiosity, I had a look at the bittorrent protocol some years ago. Is the specification from October 2013 the one bittorrent sync uses? Into which fields I should be looking at in order to understand a bit better this limitation/design choice?

[tbdtbd]

3 hours ago, iswrong said:

On Linux, it also works just by running sync instances using two different users. On our RPi, we have two instances of Sync running with a partially overlapping set of folders.

Thanks for the tip. I do not have any Linux box around today but I have just tried with Windows7. I have two users on the same machine and I am able to create an encrypted folder on the same hard disk just by using the second user, as you suggested. It is an old machine, but I got a transfer speed of around 18 MB/s, which is reasonable even for a large amount of data. I also tried and generated the encrypted folder on a different machine, which I connected to the first one using a ethernet cable (directly between both machines). Interestingly, I got higher transfer speeds. So, it seems the bottleneck in the first experiment was the I/O of the hard disk.

I also tried creating the encrypted version of the folder locally and transferring it to a remote machine, which is what I was after from the beginning. It also works perfectly well. The bittorrent sync client on the remote machine spends some time indexing the new folder and notices it does not need to download anything over the network.

In summary, this "workaround" is trivial to implement on any system and gets the job done. Thanks!

[GreatMarko]

5 hours ago, tbdtbd said:

Out of curiosity, I had a look at the bittorrent protocol some years ago. Is the specification from October 2013 the one bittorrent sync uses? Into which fields I should be looking at in order to understand a bit better this limitation/design choice?

Sync uses a "variation" on the BitTorrent Protocol. Specific protocol details have not been published.

[iswrong]

8 hours ago, tbdtbd said:

Out of curiosity, I had a look at the bittorrent protocol some years ago. Is the specification from October 2013 the one bittorrent sync uses? Into which fields I should be looking at in order to understand a bit better this limitation/design choice?

The key structure is documented here:

http://sync-help.bittorrent.com/customer/portal/articles/1628254-key-structure-and-flow

There used to be an open source implementation of the 1.3 (?) platform. You can still find it when searching this forum. I think the protocol changed in 1.4 and then there is the new identity-based syncing. Of course, it still builds on Bittorrent infrastructure, such as the DHT, etc.

I would be surprised if it is actually a protocol limitation, given that syncing between two instances on the same machine works. I guess it's more likely that it's just not supported by the software (to do it within one instance). I am also not too sure what it would buy one (since you can use rsync as well to synchronize a set of files locally), except for in this particular case where you want to generate the encrypted files.

[Inuart]

Just being able to run some btsyncs in parallel (on different ports maybe?) and letting us users deal with the consequences of our actions would be awesome.