root-12

This is a really good idea for a NEXT GENERATION WEBBROWSER APPLICATION that uses DHT instead of DNS protocol. The application will have to manage the caching of pages with selective sync, and that is far from impossible. Q: What if it's a very large site. And you are not interested in watching everything on it? The same application can choose to view classic webserver pages without further ado and thus cover both types of usage, and at the same time avoid putting mental load on the user. We all know when a server is down and google chrome prompts: "Sorry the page is unavailable. Do you want to view a cached one?" Next, the problem regarding domain names is solveable as a DHT: If just a few DNS servers are set up as bittorrents (using bitcoin for authenticity), a query to bittorrents would be suitable for the DNS lookup. www.my.ultra.fancy.site.here would be DNS'ed as a torrent-file, so when you "search" from the application, you actually put a search for a bittorrent, and the torrent returns the read-only secret. So even watching updates on a large site (google) is easy if you can tell the service that you want to subscribe to a particular subject. Q: an intermediary search engine has to be able to catalog all files A: No, George Rzevski invented the broadcast-based search engine back in the 70'ies, which was way ahead of its time. It used a protocol where the local application broadcasted a keywords to peers on a scalefree network based on a "guess" of who would know. These then either returned their guess or echoed the query to those who they believed might know. Though the pagerank algorithm was not yet invented, Rzevski's model used an 8 bit "confidence" level, to provide a response. 255 was the same as "I don't know and don't know anyone who knows". Based on Jure Leskovec's research from 2008-2012 (Stanford) we now know that you can touch every application on the internet in this manner in 32 steps if we know just 2 devices. The trick is that the analysis of the query is distributed, whereby it is as if every device in the world had a keyword and a peer list. So when a query arrives, to a device it asks itself can I help by either providing an answer to the query or by echoing the query to a list of people I know? Bittorrent research that states that 8 peers is enough, provides a reasonable heuristic. When the broadcasting device is done "asking" and has assembled all responses, it will be capable to answer and point others to the results. So while you wait for your peers to respond and you are browsing through the search result pages < 1, 2, 3, ... next > you indicate search depth, looking for lower ranking results and extending search horizon. Hereby you "broadcast" to your peers: "Sorry mate, the last result was nice, but not what I wanted. Could you please ask your peers for alternatives?" They will then take the next step down the confidence rank and provide answers. (George Rzevski originally invented this for data-mining material that in the 70'ies could not fit onto a single data tape. So his solution was very pre-hadoop, but pretty much did the same if all servers were known. The problem however was that tapes wore out, cracked, etc. so from time to time there would be down-time. To overcome this, he added the confidence level of "who should know" and if all pointed to a machine that was not available it could provide a maintenance message. -But that is another story...) Let's re-invent the internet browser :-)

An interesting suggestion/comment. While it is correct that the description above is "centralized" given the presence of the NAS, I believe it is a small step to allow for multiple NAS's by extending the `/catalogue`, and possibly introducing the element of stochastic profiling to assure redundancy based on availability. I believe all that this will require, is: 1. A log of availability on each device. 2. A function to compute the local the set of files which a device may provide availability for in which time-window To create the distributed scheduling method with availability forecasting sounds like an interesting challenge.

Yes. I intend to put it on github for others to expand/improve.

Bittorrent hack - Virtual Private Network It always annoyed me a little, that sync providers have the mindset: “Either it all goes here, or it goes there…” Fact is I want remote control over my files from any device at any time, so this is the specification of how I have created my own VPN using BitTorrent Sync running as an OS agnostic background service… A Network Attached Storage. I have a 3Tb NAS with five folders: /BTsyncApp//BTsync/catalogue/BTsync/files/BTsync/shared/BTsync/devices/BTsyncApp contains the server application. It is not synchronized using BTsync. It is just the application. /BTsync/catalogue contains a html page (catalogue.html) with all files and is BTsync’ed to all AVAILABLE ON ALL DEVICES. This is about 100kb of simple html text. The webpage looks like this: +---------------------------+ | HTML5 -- simple app --- | +---------------------------- | files/folder/ | | files/folder/name.ext | | files/folder/name2.ext | | files/folder2 | | files/folder2/name.ext | | files/folder2/name2.ext | +---------------------------+ When I click on any hyperlink I get a box with the following options: [get][open][delete][destroy][move][share] / [revoke] [create torrent][create 1-time torrent]if it is a folder I get the following additional options: [create folder][share folder] / [revoke share] /BTsync/files contains approximately 2 Tb of data, and there is no way I want BTsync to synchronize this data across my devices. /BTsync/devices contains a folder for each device. For example: /BTsync/devices/HTCphone/BTsync/devices/win7Laptop/BTsync/devices/LinuxLaptopAnd finally /BTsync/shared is used in a special manner which I’ll explain later. The Devices All devices have BTsync installed and have two folders: catalogue and files. For example: My old HTCwildfire has the folders: /BTsync/catalogue/BTsync/filesMy windows laptop: D:/BTsync/catalogueD:/BTsync/filesMy linux laptop: /home/me/BTsync/catalogue/home/me/BTsync/filesI can now open the html file from ./catalogue as a webpage in any browser on any device. The attentive reader would have noticed that only the NAS has the folder /BTsyncApp. The devices do not. That is because the server generates the catalogue-webpage in /catalogue. The devices create a hidden appended worklist within the webpagefile which the server analyses. Thereby the only place where changes can be made is on the server, which prevents synchronization conflicts of the file catalogue. Other files may be updated asynchronously but that will influence the content of the file, not the location - which is what the BTsyncApp looks for. HTML view When a given file is not on the device, the link [get] is bold. If I click on the link [get], the htmlfile is saved, with a setting “get” for the filename. The htmlfile is then delta-synced effectively using BTsync, whereby a python script on the NAS will discover the delta and will trigger python to create a hard link from the NAS to the BTsync folder of the device. Very simple. For example: On HTCphone [get] “drive/folder/name.ext” will hard link /BTsync/files/drive/folder/name.ext to /BTsync/devices/HTCphone which is BTsync’ed with the phones folder /BTsync/files/. So after the hard-link is established, the file is automatically downloaded to the phone. The same occurs if I [get] a folder: The hard link implies that the rest of the tree should be synchronized. Warning: (because of fumbling I have learned to prompt a “confirm/cancel box” in the webpage with very large buttons). When the synchronization is completed, the python script on the NAS updates the webpage-file and synchronizes it across all devices. At this point in time the file should have been synced to my device, and the link [open] is changed in the htmlcode to bold, whilst the link [get] is normal, i.e. no longer bold. Of course, if it is a large volume of data, and I want to know how far my download is in the meantime, I can just open the BTsync application and see what it is doing. Now if I chose to delete a file in BTsync on a device, the local file is of course removed, but on the NAS only the hard-link is removed from the particular device folder. So for example: me@linux$ rm /home/me/BTsync/files/drive/folder2/name2.ext will delete name2.ext locally, and the NAS will follow by removing the hard link. The file itself is not deleted from files/drive/filder2/name2.ext only the hard link for the device. I will get back to how this may be achieved further down. When BTsync makes a change to the folder pynotify reports a change on the file system and triggers the python script to update the simple webpage. Hereby the field [open] is changed to [get]. I’m sure this could be done using some internal mechanism in BTsync, but I don’t know how. Share with others If I want to share a particular file with somebody else (this is the cool thing) I press the hyperlink [share]. This prompts the python script to create a folder in /BTsync/shared/ with the sha1hash of the file concatenate with the time, which it adds to BTsync’s list of synchronized folders using command line instructions. Example: /BTsync/shared/c191687881d9345c914186591ea850f7c1efb63c-2013-10-28-235959/ This is because it permits me to share the same file to multiple persons with different folders and will maintain the ability to revoke them individually. Next, the python script creates a hard link from the file to the shared folder, which is then ready for remote access using the read-only “secret”. And, keep in mind that the overhead for setting up the hard link is negligible. Example: /BTsync/shared/c191687881d9345c914186591ea850f7c1efb63c-2013-10-28-235959/name.ext Finally a can find the read-only “secret” in the BitTorrentSync application and give the code to the person I am sharing with, within seconds as hard linking is very fast. The menu again So my menu is limited for files to:[get] which makes the file available on a select remote device.[open] which indicates where the local file should be.[delete] which deletes the local file and unlinks the devices hard-line on the NAS’.[destroy] which deletes the local file and makes the NAS traverse all links to remove the file and all hard links permanently.[move] which sends a file on a listed device to another (set of) device(s). So I can push the hard link off my mobile phone and onto my laptop. When BTsync see’s that my phone is next to my laptop, it concludes that it is easier to move the file peer-2-peer than requesting it via the NAS.[share] which makes the file available in a separate folder as read-only. When shared, the option changes to [revoke]which FIRST removes the folder from BTsync’s list, THEN unlinks the file from the [share] and then deletes the folder.[create torrent] which makes it possible for people to download the file who cannot install BTsync. For some corporate users this is be helpful.[create 1-time torrent] which - surprise :-) - only can be used once.The following options appear in addition if it is a folder:[create folder] which prompts for name[share folder] which prompts for size in Mb and makes a shared folder on the NAS which any remote device can read the secret from an add to the BTsync. Any size limit greater than -1 will be translated into how much space is available for the user.[revoke share] which first creates folder [name-revoked-dd-mm-yy] then transfers the hard links of all files in the folder [name] to the newly created folder; and then removes the folder [name] from BTsync’s list of sync’ed folders. Finally it is up to me to decide whether I share the read-only or the full-access secret using my BTsync application.[destroy folder] which wipes the content of the folder on all sync’ed devices. The python script deletes any files in the folder and places a flat text note: “This folder was destroyed by the owner. Date/time”. The folder can obviously not be deleted as this would not synchronize across to other devices (doh!) but you get the point.This is simple, but it works. The key element is of course a machine of high on-line availability. Since we are not wanting to use public cloud systems, I wrote this for the most low-power device I could imagine: The Raspberry Pi with an external hard-drive. Python is as close to OS agnostic as it gets, but I leave the concepts described here for others to review, tune and criticize. My objective was to develop the required functionality and prove the concept, as a scripted transactional system which proved to work. Maybe the team at Bittorrent can supersede by integrating this/augmenting the existing UI’s? Todo:* Cleanup function, that - during idle/night time identifies duplicates in the /BTsync/files on the NAS and substitutes them with hard links. It saves space works faster. This must not work outside /BTsync/files as it could erase BackUps.New (4 Jan 2014). Function that load balances storage on devices. Each device will be added a config file (maybe BTsync config file) that states permitted volume. If a device requires files to be synced/download for which there is no space, then this option will permit the application to take the least used, oldest files and remove them from the device to make space for the newly required file. The removed files will still be available on the NAS. The objective is to save the user will from having to worry about local disk-space.New (6 Jan 2014). Function that maps the device IP address, for multiple NAS, so that all files are available in different netzones. The initial idea is to use the first three ip-address digits (ip://a.b.c.x) to assure availability in different zones. Hereby if one ip-provider cuts the network to the NAS in zone a.b.c, the files can be available from another zone (x.y.z). This will apply to devices holding a "/files" folder only.