Add A Light Transfer Protocol For Small Files?


haraldthi

Recommended Posts

Bittorrent is made for large files, and it's good for that. However the splitting up the file in pieces before transmitting them in semi-random order adds a significant overhead per file transfer, and this becomes a nuisance when transferring small files.

There was a project in the Debian community once, to transfer the package archive over Bittorrent. It would be ideal for offloading the package archive, but they gave up. In part because most of the files in the archive are too small to be transferred effectively with Bittorrent.

Now it shows up in btsync again.

Partially, when I use btsync to back up and share sound files and photos. Each file is rather small, but as a collection it becomes huge. Btsync isn't very good at it, with frequent stops in file transfer to catch up with its overhead.
I also use btsync on an Iphone 4, and the app crashes frequently when I use it to start playing a sound file I have synced. (I'm in a (rather minor) band, and I record what we do so we can learn from it). I guess it's due to btsync taking up a lot of memory, and I guess it's what cell phones and pads in general do to manage: Kill processes when memory becomes limited. If btsync could use less memory most of the time, it could be kept longer in memory and be better at running in background.

So to the point. I know it's kind of a "kill your darling" request, but could btsync not keep its full synchronization protocol in memory all the time?
Instead, it could detect smaller files (less than 10 MB could be a good default) and transfer them from one computer to the other the boring way it's been done since the beginning of file transfer? It would be lighter on resources, and since there are probably a lot more files in the queue anyway, btsync can choose to transfer one file from one computer and another file from another computer and share resources that way.
Then, when there's a larger file on the queue, the full file splitting tool can be loaded to memory (via a library, a subprocess or whatever) and then forgotten from memory when there are no more large files to transfer.
This way, btsync can work well for both small and large files, plus work better on memory constrained devices.

What do you think?

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.