Sync-Ing Over Http Restricted Firewall

[cod3monk3y]

I work for a small research company that makes educational software/games, and we've got a deployment of 50 computers in 10+ schools.  We're currently synchronizing our application and media content from our server to the schools, and our log files from the schools to our server using SyncBack (FTP, yes I know -- without the S, no need to lecture). 

 

The schools have varying and inconsistent firewall rules, so the sync isn't happening at some locations. I know the limitations of the networks and why the transfers aren't working, so I'm not asking how to get through the firewall limitations This question is more to determine if btsync can get through strict firewall rules.

 

Some locations allow FTP, SSH, SFTP, HTTP, and others are locked down to just HTTP. I've tried using WinSCP instead of SyncBack and tunneling SSH over HTTP using Aache mod_proxy, but even this is detected at some sites as a malicious proxy attack.

At some sites we do not have the ability to open firewall ports.

We need a synchronization solution that will work given these restrictions. Is btsync capable of handling this scenario? What ports does btsync use, what protocols for establishing the network (peer detection) and what protocol and ports for data transfer?

 

Basically in the worst case all data needs to be transferred over HTTP protocol, and *not* using HTTP connect (e.g., Apache's mod_proxy ). In the best case the school is "wide open."

 

I think btsync will work great for us, as it will limit the external bandwidth since not all 10 computers at each site will need to download the application updates. One can do the external download and the peers can sync from that "master" and from each other. In the worst case scenario that the school is completely shut off from the world, we could upgrade just by stopping by with an updated laptop and btsync would propagate all the changes to the peers on the LAN. I'd like to avoid that if possible.

 

 

Thanks in advance!

 

cm

 

p.s. One of my alternative deployment solutions is using git to deploy the application (like Heroku uses). This should work since git has an HTTP protocol for pulling. The downsides are that we have a *lot* of data in our application/media and don't need/want to store the version history/deltas (in my tests the initial repo was about 1.8x as large as the folder hierarchy); and all the clients would have to pull all the changes, hogging network bandwidth.

[lolcat]

 

We need a synchronization solution that will work given these restrictions. Is btsync capable of handling this scenario? What ports does btsync use, what protocols for establishing the network (peer detection) and what protocol and ports for data transfer?

BTSync has some cloud option to break through firewalls, no idea how it works.

The name Bittorrent Sync doesn't provide a clue? It uses the ports you ask it to, it detects peers through local peer discovery, DHT, and through the tracker. It uses a (possibly somewhat modified) Bittorrent protocoll. It uses the port you tell it to.

[cod3monk3y]

The networks at the schools are often extremely limited for obvious reasons. One school district blocks everything but HTTP, and is even blocking SSH tunneling over HTTP (using Apache's mod_proxy / HTTP connect method). FTP is shut down. STFP is shut down. Sites with the word "proxy" are blocked. 

 

I've read a few posts in this forum about IT blocking "bittorrent-like" traffic, and requiring ports to be opened. I'm pretty certain that UPnP is not going to work given the strength of the IT in the schools.

 

I see the option for adding known hosts in the folder settings, which lets me specify a host and port which would let me connect to port 80, and I'm going to test this out next time I get to the schools. But most inbound traffic is blocked. We had this problem when we tried using FTP, since it assigns random ports for data transfer. SFTP worked on one site through a tunnel (again, the sites are not consistent and IT doesn't always have the time to accomodate our needs). I know that any return traffic trying to connect to the laptops on site will need to have the router forward an external port to the port I specify in the settings. But we won't be able to get IT to do that. 

 

I couldn't find information on the Bittorrent Sync relay anywhere. More information on this will help me know whether it will work and what to talk to with the IT departments in the districts.

 

To get a common working solution for my problem for all the sites, I'll likely need sync option that communicates over HTTP exclusively, which means I'm probably stuck with WebDAV or Dropbox (which uses a RESTful web service API for transfers). I'd much prefer the Bittorrent Sync route because it reduces external bandwidth (our media files are pretty extensive) and is a great common solution for live mirroring back home on our servers with the collected data as well, and the files are not stored in the cloud (since they may contain sensitive student information).

 

So to focus my question, how does the relay work?

 

/cm

 

 

p.s. If the name had given me a clue to answer my question, I would have answered the question myself. I do a great amount of research before posting questions on forums (here, stackoverflow, etc.), and only do so after an extended period of experimentation and effort. I've read a few of your posts on the wishlist, and many of them come across as condescending and arrogant, and this response is no exception. I appreciate any information you can supply in helping in my scenario, but please drop the assumptive belittling tone or don't reply at all.

[cod3monk3y]

We ran our tests today, and BT Sync was not able to connect, up or down, through the firewall. "Relay" was enabled, and we even tried adding a "known host" to the synced folder on port 80 which I verified was working through another setup (PC1 -> Verizon Jetpack -> dynamic IP -> Linksys E4500 --> port forward 80:80XX --> PC2). 

 

It would be great to have more information on how the relay works so I can try to diagnose the problem, and work to properly get through the firewall. I'd be happy to work closely with the BT developers to provide diagnostics and a test case for an extremely tight firewall.

 

Meanwhile, Dropbox worked (as expected since it's all over REST). Cubby also worked in both cloud hosting and peer-to-peer mode (which was a real surprise). I could not find documentation on Cubby's protocol, but it just works right out of the box and we're okay paying $9.99/month for a year for something that works.

 

For my personal stuff (like my password safe) I'm really happy with Sync since it keeps everything out of the cloud, and I'm going to set it up to try out. I'll keep an eye on the relay situation and reconsider using it once that's resolved.

[lolcat]

Remember the encryption used by BTSync has not been independently reviewed, the data can still be captured by an adversary in transit, and Bittorrent inc is a US company, so they can be forced to implement backdoors and they would not be allowed to tell the users about it. It is NOT suited for confidential information. I would not transfer a password database using it without having it independently encrypted (but then you would be just as well off using DropBox).

Why the media belives it is "NSA safe" is a mystery to me, there is ample evidence they can just store what is sent over the internet, and encryption that isn't documented or independently reviewed is worthless. Closed source software from the US can also not be trusted, service providers has (and can) be forced to implement a backdoor.
 

[RomanZ]

It would be great to have more information on how the relay works so I can try to diagnose the problem, and work to properly get through the firewall. I'd be happy to work closely with the BT developers to provide diagnostics and a test case for an extremely tight firewall.

 

 

To transfer data thru relay server BTSync tries to contact r.usyncapp.com server with UDP over port 3000. Relay server identifies packet destination and redirects it to appropriate peer.

 

Keeping in mind that school network is strictly fire walled and does not let anything out except destination port 80 - relays will definitely won't work. Also note, that using a relay server reduces the strong side of BTSync significantly - the speed of synchronisation.

 

The only way in such configuration would be to use "known hosts" and try to use a single open port to establish connection to your office and try to track where packets are get lost.

 

Hope it helps.