kos13

Sync API wishlist

Recommended Posts

Please share your thoughts on Sync API, what you need and what you plan to do with Sync. If you have a use case in mind, or just another random thoughts about API, please share them with us.

This would help us in developing API that would fit your needs.

thanks,

kos

Share this post


Link to post
Share on other sites

1. Dedicates API keys that you can generate unlimited of & give certain permissions. For example, if site A wants to use my share, but, I'm not sure if I trust them, they say they just want to dump log files of my VPS onto it, I can select "Only allow writing of files, not reading".

2. Be able to retract API keys, say I cancel my VPS on the above site, I no longer want them to able to write to my share. I want to removed said API key from being used

3. Obviously, ability to read files, write files, make folders, delete folders, delete files, move files, rename files, all of the standard filesystem commands. Not sure if it's possible, but, maybe make symbolic links and the rest of the more 'advanced' filesystem options? Not sure how well that'll work though.

4. Limit IP (ranges) to API keys, if the above VPS site gets hacked, I want only their servers to be able to do stuff.

Although, I have a question, how will the API work? Will it be like a web-API where you connect to one of your servers and then that deals with asking the nodes? Or will you just contact any node in the system and ask them to talk to the other nodes? Or will you pretend to be a node? I've never dealt (or heard of) any P2P APIs, so, I'm having an issue trying to understand how it'll work. If you provide this, I'll probably be able to think up a few more, I'm just confused in which "in point" it'll come in from.

Share this post


Link to post
Share on other sites

Although, I have a question, how will the API work? Will it be like a web-API where you connect to one of your servers and then that deals with asking the nodes? Or will you just contact any node in the system and ask them to talk to the other nodes? Or will you pretend to be a node? I've never dealt (or heard of) any P2P APIs, so, I'm having an issue trying to understand how it'll work.

We have several options how to implement API. I created this thread to collect use cases and figure out the best way to implement API.

Share this post


Link to post
Share on other sites

Another few things:-

A. Ability to do basic tests on files without having to download them all (have the nodes in the system who do have it run the commands). Stuff like egrep/head/tail/sed/tr/wc, just all the basic linux I/O commands.

B. Ability to edit small amounts of the file without ever having the file, say I know I want to edit bytes 10,000 through 12,500 with "DO A BARREL ROLL!" 147.058823529 times, then, I don't really want to download the full file (Which may be insanely large). You'd know where to edit using the above linux I/O commands.

C. Ability to select "Only upload X many copies" and let the other nodes deal with sharing it within the network. Useful for third party companies who want to preserve bandwidth, once they upload one copy, they no longer want to seed it for their customers, the customer's PCs can then seed between themselves.

Share this post


Link to post
Share on other sites

If you have a TTM date for the API I would like to know it asap. My humble suggest is that it should be available in a cross-platform way, preferably in C++. If possible with java and dotnet extensions. But I think the best approach would be to have a javascript oriented API also. So, in combination with Socket.IO, Node.JS and Dynamic DNS we can build a fully operational clould shared storage service using our won computers and a browser!!!! Really cool!

Share this post


Link to post
Share on other sites

If you have a TTM date for the API I would like to know it asap. My humble suggest is that it should be available in a cross-platform way, preferably in C++. If possible with java and dotnet extensions. But I think the best approach would be to have a javascript oriented API also. So, in combination with Socket.IO, Node.JS and Dynamic DNS we can build a fully operational clould shared storage service using our won computers and a browser!!!! Really cool!

Or, you know, just release the API protocol specifications and then every programming language is supported? Excluding javascript because of the same origin policy (https://en.wikipedia.org/wiki/Same_origin_policy), however, you could use PHP & javascript to overcome that limit.

Share this post


Link to post
Share on other sites

Few more:-

A. Get file modification, access and creation date (Assuming that the filesystem supports this)

B. Get size (Real size, not size on disk as that'd change through nodes)

C. Get hash of file (MD5/SHA1/SHA2/CRC/etc...)

D. Get hash of certain 4MB parts of files (That bittorrentsync uses)

E. Access files from inside rar/7zip/tar/gzip/zip archives and treat them as if they were any other file.

However, I'm starting to think what real use an API would give, and, I can't think of much.

Share this post


Link to post
Share on other sites

Few more:-

A. Get file modification, access and creation date (Assuming that the filesystem supports this)

B. Get size (Real size, not size on disk as that'd change through nodes)

C. Get hash of file (MD5/SHA1/SHA2/CRC/etc...)

D. Get hash of certain 4MB parts of files (That bittorrentsync uses)

E. Access files from inside rar/7zip/tar/gzip/zip archives and treat them as if they were any other file.

How this would be different from regalar OS commands?

Share this post


Link to post
Share on other sites

In many cases just the command line should suffice. Running the btsync executable when there's already a process in action could trigger commands over IPC (named pipe probably being the easiest)

e.g.

btsync --list [*glob*]

List all (or matched by glob) synced directories, their status (including last sync date) and their secrets

Additional options:

--files

Also list files

--peers

Also list peers

--ignore

List ignored files

--script

Output tuned for scripting (csv? tsv?)

btsync --add <directory> [--secret <secret>] [--pause]

Add a directory. If a secret is specified use it, otherwise generate a random one and print. Optionally start paused to allow checking the list of files to ignore.

btsync --delete <secret | dir>

Stop syncing a specific

btsync --trash <secret| dir>

Stop syncing, destroy data

btsync --pause

btsync --resume

btsync --ignore <dir | secret> --files <*glob*>

btsync --ignore <dir | secret> --list

btsync --unignore <dir | secret> --files <*glob*>

Probably enough to be getting on with.

My use case is transferring genetic sequence data to our analysis cluster without having to give outside individuals logins. BT Sync can already do this, but a nice set of CLI options would allow me to put a somewhat friendlier UI on top for my users.

Share this post


Link to post
Share on other sites

Okay list of things to do to the server ...

  • Create a new secret share (perhaps with a new empty directory)
  • Edit any configuration on a share.
  • Delete a share
  • Pause a share; disconnect from clients pretend it's deleted but keep it so it can be resumed instantly.
  • Delete the files and directories that are in or were in a share
  • Fetch individual files from the server or upload them for sharing.
  • Turn the debugging on/off and view/download the debug log.
  • Local or remote call backs (run programs) from the server whenever files are created, removed or modified. Eg: anti virus; it'll need the ability to revert to old versions. eg: anti-delete; once uploaded write permission is removed from the file.
  • Call backs when a node (including this node) has finished syncing, callback when all nodes finish syncing (define "all nodes", eg: named list, all currently connected, 7 unique identifiers). Might want to delete the share when everyone or enough peers have it, might want to prevent us updating the share until everybody has a consistent copy. Might want to take a backup when it's consistent. Might want to shutdown the VPS when we're done.
  • Possibility of sync being suspended while the callback is running. (ie keep it consistent)
  • Trigger a scan of the filesystem for a share
  • Suspend/resume scanning the filesystem for a share (nothing will be changed locally)
  • Scheduled stops and starts of shares, perhaps just the ability to upload a 'crontab' but windows too.
  • Alter the .SyncIgnore file and similar
  • Server callback programs should be able to safely stop and restart the btsync program.

Okay, that's enough for now ... :)

Share this post


Link to post
Share on other sites

How this would be different from regalar OS commands?

I just assumed that the API would be for remote modification of shares without having the full files downloaded (E.G. for my service to modify your shares without having your shares, so, I can't run my MD5 hashing software on a file I don't have), if I'm incorrect, then, it'd help to know how the API would be used (Which, I asked before but apparently you don't know yourself).

EDIT:- Unless you're talking about launch arguments, which, I wouldn't really call an API. I'd simply call it a command-line interface.

EDIT2:- Example:-

Youtube API provides the length of the video, if I were to download said video I'd know the length, but, at that point I'd stop calling it an API and start simply calling it a downloader & metadata reader.

EDIT3:- Or maybe you mean plugins for BTSync? I can understand why I wouldn't need the commands if it was simply a plugin, something that's run along side BTSync.

Share this post


Link to post
Share on other sites
  • Local or remote call backs (run programs) from the server whenever files are created, removed or modified. Eg: anti virus; it'll need the ability to revert to old versions. eg: anti-delete; once uploaded write permission is removed from the file.

This is an excellent idea. You could use these callbacks to trigger filesystem level snapshots (VSS on Win32, zfs snapshot, xfs_freeze + lvcreate -s etc) which would also mean that btsync wouldn't have to worry about keeping deleted/change items anywhere, giving version control...

Share this post


Link to post
Share on other sites

I just assumed that the API would be for remote modification of shares without having the full files downloaded (E.G. for my service to modify your shares without having your shares, so, I can't run my MD5 hashing software on a file I don't have), if I'm incorrect, then, it'd help to know how the API would be used (Which, I asked before but apparently you don't know yourself).

Please, it would be very interesting. We never thought about such use case.

EDIT:- Unless you're talking about launch arguments, which, I wouldn't really call an API. I'd simply call it a command-line interface.

EDIT2:- Example:-

Youtube API provides the length of the video, if I were to download said video I'd know the length, but, at that point I'd stop calling it an API and start simply calling it a downloader & metadata reader.

EDIT3:- Or maybe you mean plugins for BTSync? I can understand why I wouldn't need the commands if it was simply a plugin, something that's run along side BTSync.

API is so generic terms that different people might use in different context. Therefore we asked you what you want to see. This will give us better chance to develop something really usable.

Share this post


Link to post
Share on other sites
Please, it would be very interesting. We never thought about such use case.

Not sure if that's a question as in "Please continue" or just a statement, but, for the sake of not replying for no reason, I'll reply as if it's a question.

No idea the exact terms of how to do it (As thinking that far ahead would require me to be coding the thing), but, you'd connect to the tracker and ask for a node that is listening on your API key, the same way you do with secrets, just, a marker saying it's an API key and not a secret. Once it connects to a node, it's pretty much the same as if it were a node (Downloading & uploading), however, it asks the node to do everything instead of doing it itself. Instead of downloading a file, modifying it, and then reuploading it, it sends a request to the node to modify it, and then the node just uploads it. Same with if you want to create a file, you ask the node to create it and then the node distributes it. Obviously because of this you can't hash files yourself, or search the file for regex terms/etc yourself, so, you'd have to get nodes to do it for you and reply back with the results in a simple TCP connection.

You could also do callbacks and the rest by having an option to keep the stream open and apply 'ids' (chosen by the client) to certain requests. I believe (Not done any NNTP in awhile, so, I may be wrong) that the NNTP protocol does the marker thing, and, I believe the routerOS API (Although I haven't looked too far into it) also does the kind of stuff I'm talking about.

NNTP specification:-

https://tools.ietf.org/html/rfc3977

RouterOS API wiki:-

http://wiki.mikrotik...wiki/Manual:API

Share this post


Link to post
Share on other sites

No idea the exact terms of how to do it (As thinking that far ahead would require me to be coding the thing), but, you'd connect to the tracker and ask for a node that is listening on your API key, the same way you do with secrets, just, a marker saying it's an API key and not a secret. Once it connects to a node, it's pretty much the same same as if it were a node (Downloading & uploading), however, it asks the node to do everything instead of it. Instead of downloading a file, modifying it, and then reuploading it, it sends a request to the node to modify it, and then the node just uploads it. Same with if you want to create a file, you ask the node to create it and then the node distributes it. Obviously because of this you can't hash files yourself, or search the file for regex terms/etc yourself, so, you'd have to get nodes to do it for you and reply back with the results in a simple TCP connection.

And if this is implemented, what will be a use case for such functionality?

Share this post


Link to post
Share on other sites

And if this is implemented, what will be a use case for such functionality?

For the use of allowing applications to automate file manipulation and modification without requiring the full bulk of a share (which may just be a couple KB, all the way up to a few TB).

I believe dropbox has an API like this (Although, I've never used it, so, I can't be 100% sure), you might want to look into some of the projects done with that API. I'll have a look myself after I post this.

EDIT:- Here's dropbox's documentation:-

https://www.dropbox....lopers/core/api

You'll have to scroll down quite a bit, or look at the side-page list.

EDIT2:- For an example list of projects (Although be it, not the best list of examples), here's a google search:-

https://www.google.com/search?q=https%3A%2F%2Fapi-content.dropbox.com%2F+site%3Apastebin.com

Share this post


Link to post
Share on other sites

The ability to set capacity caps for a given folder via the API would, I'm sure, a very highly sought after feature.

Use Case: (hypothetical)

I have a dedicated server which I use for backup storage -- a friend would like to use my online storage server as well. I agree, but I'd only like him/her to use a maximum of 1GB of storage space. Using the API, I should be able to set any given folder which is attached to his specific secret key, an (admin) set maximum storage value.

This is a highly useful feature when thought about at length.

Share this post


Link to post
Share on other sites

The ability to set capacity caps for a given folder via the API would, I'm sure, a very highly sought after feature.

Use Case: (hypothetical)

I have a dedicated server which I use for backup storage -- a friend would like to use my online storage server as well. I agree, but I'd only like him/her to use a maximum of 1GB of storage space. Using the API, I should be able to set any given folder which is attached to his specific secret key, an (admin) set maximum storage value.

This is a highly useful feature when thought about at length.

Wouldn't that be more a feature (Thread being "Wishlist") than an API kit? I'm not saying it wouldn't be nice to also have a hook for it in an API (so data storage companies can automatically apply data changes when you pay/etc without staff), however, first of all the item needs to be implemented.

Share this post


Link to post
Share on other sites

Without releasing the Sync protocl, making API feature requests is kind of like shooting aimlessly without a target. Anyway, my turn to shoot:

  1. RESTful API in the client app e.g. 127.0.0.1:8888/api/<resource names>/
  2. JSON support in API request / response
  3. API versioning via HTTP header
  4. API around getting metadata of a share e.g. /api/share/<secret key>/

    1. Getting metadata of a share without sync'ing the share. An analogy using BitTorrent-speak is to view the list of files in a .torrent without starting to download the torrent
    2. Query share size, list of files, metadata of individual files, list of clients of a share
    3. Ability to download individual files / directories of a share without being part of the sync
    4. Query change history of individual files / directories
      1. "Who changed this file at what time?"
      2. "What is the file's checksums before and after changes?"
      3. Would be great if the entire change log of the share is a query-able API resource

[*]Ability to reference individual files / directories in a share using a global unique ID / permalink

  • The unique ID of a file does not change after a file is added to the share
  • Moving, modifying or deleting the file does not alter the unique ID
  • New files added to the share does not overlap with previously deleted files

[*]Ability to query history of files / directories deleted from a share

  1. Support date range in all history query

[*]Ability to rollback changes of a files / direcotries (I know, this is sounding more like an SCM)

[*]Ability to perform simples filesystem operations (silimar to Dropbox's /fileops/ API)

[*]Ability to query all active shares on this btsync instance

[*]Query activity log of a share's SyncTrash

Share this post


Link to post
Share on other sites

Okay list of things to do to the server ...

  • Local or remote call backs (run programs) from the server whenever files are created, removed or modified. Eg: anti virus; it'll need the ability to revert to old versions. eg: anti-delete; once uploaded write permission is removed from the file.

This will be a tremendously useful feature! The BT Sync client should also be smart about limiting the no. of concurrent callbacks running, to prevent exhausting the computer's memory / disk bandwidth etc.

Share this post


Link to post
Share on other sites

BTTorrent Sync is an all-new-stype excellent transfer tool in the world, it combine both p2p and internet syncing utility, I think no one has fully thinking about how to use and help their working before...I am using Aria2 now, and it's help my work very much ,

How about checking "Aria2" API, I think it could be a good template for somepart...

although "Aria2" is one of P2P tools, and also providing good and completely API example , But I think if you can provide some basic functionality with first step , It's already enough for so much MIS help doing there work....

for most import API I think as above:

1. Add/Edit/Remove specific share folder

2. Resync specific folder

3. Add/Update/Remove predefine host

4. Enabled/Disable engine, just enabled/disable not close application (maybe someone need schedule p2p transfer)

5. Speed limit control

6. report each shared folder transfer state

7. all API call by XML style , for exp REST or XML-RPC, bittorrent sync build-in is better

Share this post


Link to post
Share on other sites

Something to add to my list.

  • A callout (trigger) that fires before a file is modified (created or deleted). This could be used to implement a shared permission scheme with digital signatures (or just bloodymindedness) If this node doesn't like the change (eg the new file has the wrong digital signature) it can abort it in some way eg:
    • Send a delete to the network. (perhaps by doing a rename of the old file out of the way, accept the change then rename the old one back)
    • Send a rename of either the before or the after file so they don't collide.
    • Sign the file and move it from the upload directory to the next available distribution area.
    • etc etc.

    [*]A callout (trigger) that fires as soon as it's known that a file will be changed but before any data is downloaded. So a part of a share can be made read only by us automatically reverting any changes before we make them.

The use cases are things like:

  1. distribution of software and accepting patches, reports, tests back.
  2. create a wiki data area with full versioning and rollback.
  3. Anything else were changes need to be audited or controlled.

EDIT: I think there would also need to be some sort of indication made to these two triggers so that a different change conflict management could be implemented. Instead of 'newest time wins' it could be 'oldest time', 'biggest file', 'rename old file', 'do a three way'.

Share this post


Link to post
Share on other sites

As an app developer, I have somewhat simple needs for an API.

Basically, the API should be enough to implement a Dropbox-like app for accessing BTSync shares:

  • Search for a share by key
  • List contents of share (without synching the entire share locally)
  • Upload/Download a specific file (again without a full sync)
  • Sync a share (probably on demand, to avoid killing the device's battery and bandwidth)

In my case the motivation isn't so much to build a Dropbox app replacement, but to implement sync functionality so the users can have their data replicated on all of their devices.

In an ideal world this would be possible through a library which could be added to any app.

A not-so-ideal possibility would be some sort of web-API which would essentially mean the app would be talking to a remote btsync client. This would still allow for all of the above functionality, but it would turn the app into a power-user thing: they would have to know how to setup and access their remote machine from anywhere.

Share this post


Link to post
Share on other sites

Any possibility to subscribe to synchronization events from an external application?

I can definitely see use cases in my everyday work where I could trigger some actions when a file is updated.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now