Btsync Content Delivery System Using Nested-Shares And The Api

[lolcat]

If we had a BTSync CDS (Content Delivery System), as a separate client with some extra abilities I can see endless possibilities. In this post I realized BTSync creates a new torrent for each file. This is a far more clever way to deal with updating torrents that what I would have imagined. Props to the devs who came up with it! If each file is its own torrent, then all we need (if it isn't so allready) to make a sofisticated CDS is to be able to construct a share out of the existing torrents, and distribute the appropriate keys to decrypt, or download the files.

The most important feature of BTSync is the ability to be used by someone who doesn't know left from right on a computer. You can make an installer that creates the folder, and sets up a share for your grandmother, all she has to do is download an executable, double click it (then click past the billion warnings if she uses windows) and voila, you have a shared folder(or folders depending on your needs). You could make two folders for instance, one for sending (where she has to have read-write access) and one for receiving (where read-only would be better).

Hoarders of data like me is likley to have several terrabytes of data, to sync all this to my grandmother would be extremly inconvinient, and undesirable as parts of the files are private, and parts of the files are uninterssting to her. But when I setup one or more nodes to mirror all my files, I wouldn't want to have to make a symlink on each and every node (I might even be unable to if they use read-only encrypted secrets), if I then were able to use the BTSync CDS to link my /massive-btsync-folder/picutres/respectable-christmas-photos/ to her /home/grandma/shared-folders/lolcat/incoming, she would be able to join the swarm and benefit from the aggregate bandwith of my servers, and whomever else I shared the file with.

 

The BTSync CDS could allow all kinds of interessting combinations of data, something that would be far out of the scope for the average Joe who just wants his girlfriend to get his phone pictures, but would be perfect for people who share a lot, and have a lot of files. This way we enable advanced users to fully utilize the opportunities the software provides, while still making the basic features extremly simple for the normal users who is just thrilled that someone after all these years invented a way to send files without hassle.

I am not sure if this is features that "Sync Enterprise Beta" is planning, and therfore Bittorrent inc is unwilling to allow API calls to enable the creation of this software. I do see how this would be usefull for managing massive shares for a big company. An intranet where access is ensured, and speeds are fast both internally and externally. The ease of adding all the encryption read-only keys to an existing share would allow seeding, backups (if the secrets are also backed up) and distribution without having to care about the platform, or implement additional encryption. Either deploy the btsync application to a server (virtual or dedicated), or find a provider that will accept your encrypted read-only key, and voila it is good to go. It is so simple, yet so brilliant.

I can see so incredibly many uses if you could cross refrence files between shares, add shares to other shares, and modify the share structures. Bill quits, no problem, make a new group share, sync the old one, attach it to the required users shares, and delete the old one. Downloading is slow in the building across campus with the slow uplink: just add a node there, install any OS, give it an encrypted share, and add the shares and files relevant for that group to it. Data is encrypted, and speed is ensured.

A few client features would also be benificial for this system, first it has to be able to get the secret for each file and store it in a database. In addition it should have some way to backup the entire list of secrets, think something like KeePassX, a master password, and you can see all your secrets, you could even sync it to your phone, then if you need to share a folder, you can simply generate the QR code, and they can add it. And the clients API would have to be greatly enhanced. It seems the features of the API is rather limited at this moment. The ability to add web-seeds and a share browser would be nice (almost like µTorrent, you can walk through folders, see what is there, and then download the files you would like), although a setting to not download anything, and ability to download one and one file through the API would make us able to produce this ourselves. Improved statistics would also be important to monitor preformance and discover bottlenecks. It would be nice if BTSync provided a way to easily identify if the storage media, or network, or the cpu, or ram is the bottleneck. This would make improving preformance much easier. The ability to bind to different interfaces is also important, maybe you have multiple networks, NICs or internet connections. Being able to bind to only the network interfaces you want makes setting the network up easier. Then you also need to be able to configure them differently, and preferably be able to setup rules for seeding. Being able to specify a blacklist or whitelist would enable this.

The BTSync CDS should be able to analyze the preformance of the nodes, I wonder if the easiest way to do this would be to add your own tracker, and then have all the BTSync instances report back their numbers (max speed, min speed, average speed, how much downloaded, how much uploaded, possibly CPU use, ram use, free disk space too). It should also be able to connect to nodes and specify behaviour, like when to sync, bandwith limits on different days, and times of day, limits on who to seed to, and prioritizing specific shares or users. A node only serving the LAN would be desirable in many setups, or a "master" node only used to sync the other "master" nodes. Then you could setup a master node to sync other master nodes and ignore users (preferably the ability to limit or ignore users or user groups based on the day and the time).

 

So now all we need is to fork BTSync, or hope for the developer to open up the API and add the few missing features to allow us to make this work. After the API and the features are implemented it is just a simple matter of programming.

I would love to hear more about your thoughs and ideas around this.

[sandymac]

you: more code, less talk

[lolcat]

I'm working on it, but they haven't given me my API key yet. So as a work around I decided I could just start modifying the SQLite databases that BTSync uses. It appears I could be able to make selective downloads enabled for any user by just modifying the BLOB in the "files" table of the database associated with the sync.

 

d8:fs_errori0e4:havei34e7:ignoredi0e11:invalidatdi0e11:invalidatedi0e4:maind4:

hash20:b146a4d8f331663dfd421b4b95ca38:mtimei1324556326e7:npiecesi34e5:

owner20:004bbf9596197eb9d07e981b2ddce194d1f1de46343a70:pathl5:Messy4:20112:122:2212:IMG_1399.J|PGe4:

permi493e4:sizei1113135e5:statei1e4:timei1388194516e4:typei1ee6:pvsizei-1e6:pvtimei-1e3:

sig64:cc7a970d2c61ee568c913789ce7881f21e65241996b8f36b964baa52def36e345e0ed5abb57f5786f34114d29a2cb49231b5b5a7395374c9488e520588f16d0f65

 

 

If ignoring a file is as easy as setting "ignored" to something else, then I could make a tool that goes through a database and ignores every file or ignores all the files in a folder.

I find the owner string to be somewhat interesting, and it is unclear what the sig part contains.

[pmow]

Couldn't some rudimentary CDS functionality work with the API as it is?  Using a read-write share for each user, and the "end user" removing the files/folders once sync is done?

[lolcat]

Couldn't some rudimentary CDS functionality work with the API as it is?  Using a read-write share for each user, and the "end user" removing the files/folders once sync is done?

 

The biggest obstacle is the lack of nested shares. P2P yields no advantadge if you can't actually do P2P. As it is now you could end up in a situation where you need one share per file, and that is obviously difficult to acheive effectivly. If I could make virtual shares I would be able to share with the users what they need, and they would provide bandwidth.

[pmow]

A CDS with a simple interface (like a download manager, just configured with a download location) should be possible though right? I'm not sure what your requirements are.

The API app would simply move the file out of the share, to the configured directory. Would that not accomplish what you need?

[lolcat]

Sure it could be done. But it would require far less to implement nested shares. With nested shares this becomes far easier.

If you move the files out of the folder, and create hardlinks to the btsync folder I guess it would work, but it would still be a sub-par solution.