BTSync over SSH tunnel

[coewar]

Opened new post to be more specific on this topic.

I have servers that are already interconnected via SSH tunnels and I can tunnel anything TCP without a problem and it works great. Since BTSync uses UDP, not TCP, I had to figure out how to tunnel UDP.

Well, I got the UDP tunneling working by using Zebedee which is wrapping the UDP inside TCP. I made some tests and verified the communication. I also am dumping the data and can see some BTSync data arriving. However, somehow BTSync just doesn't care. I configure to a specific port for BTSync to use, but wonder if it needs something besides that custom port.

I probably won't bother continuing this as it seems to be more complicated than it's worth. I'm forced to open FW UDP ports. I know that's better performance, but I was less concerned about that. This was for experimentation but needed to tunnel UDP for different purpose anyway so thought to include BTSync in that test.

Agonizing details:

To outline, here's what I have, making up port numbers:

ServerA and ClientA are the same box just either acting as a server or as a Client.

ServerB and ClientB are another box, same story.

ClientA BTSync -> 127.0.0.2 : 2001 ClientA Zebedee -> 127.0.0.1 2002 ClientA SSH tunnel

-> ServerB : 22 -> 127.0.0.1 : 2000 ServerB Zebedee -> 127.0.0.2 : 3000 ServerB BTSync

ClientB BTSync -> 127.0.0.2 : 2001 ClientB Zebedee -> 127.0.0.1 2002 ClientB SSH tunnel

-> ServerA : 22 -> 127.0.0.1 : 2000 ServerA Zebedee -> 127.0.0.2 : 3000 ServerA BTSync

ServerA has SSH service on port 22

ServerA has Zebedee service listening in TCP/UDP mode on port 2000.

ServerA has BTSync listening on UDP 3000.

ServerB has SSH service on port 22

ServerB has Zebedee service listening in TCP/UDP mode on port 2000.

ServerB has BTSync listening on UDP 3000.

Starting at the basics, both servers have SSH connections to each other.

Both SSH connections have client to server TCP tunnel where locally 127.0.0.1 : 2002 tunnels to the other server's 127.0.0.1 : 2000 which is the Zebedee service.

Then, each server's Zebedee service is configured to accept connections on port 2000, and allows TCP and UDP redirection to 127.0.0.2 : 3000 which is the BTSync service.

It seemed that if I used just 127.0.0.1 in BTSync, it almost fooled itself in thinking it's syncing with itself. But by using something like 127.0.0.2, or .3, etc it seemed to at least try to make the connections and data went.. but again somehow it did not care or maybe somehow it did not reach the other side all the way to BTSync.

[rdebath]

Hmmm, I didn't really mean to push Zebedee through an SSH tunnel. It's got a reasonable security all on it's own.

As for connecting servers through SSH tunnels, to me that seems like the wrong tool. I would normally use OpenVPN to transport LAN data to another private network. BTSync works seamlessly with OpenVPN in the same way as any other bridged or routed network.

OpenVPN can transport over UDP or TCP, though UDP is strongly recommended.

If you don't like OpenVPN's management of tunnels (it does simple ptp links or star networks) the tool gvpe is similar but does full mesh routing. It can also use, basically, any IP packet type as it's transport, not just TCP or UDP so you don't have to open up anything you don't want to.

(Of course there's nothing to stop you mixing it up however you like)

[mackrauss]

I had some problems with BTSync since I used a iptables based firewall on the Linux server I wanted to sync with. Although I opened the port that I set BTSync to, no syncing took place (discovery was possible).

Turns out that udp port 3838 seems important for the data transfer. After openening udp 3838 in the firewall BTSync started to work.

Hope this helps you. Would be great if you could post your configruation files should you succeed.

[coewar]

I did forget to mention that I was observing BTsync process which was listening on the custom port as well as UDP 3838. I will try to also forward this somehow... but that leads me to this question...

Which direction matters? If I am on a roaming laptop where I connect to the I-net at coffee shops, libraries, etc.. and have a common server that I connect to (in some way), the FW rules would basically only allow my roaming BTSync to initiate connection to the specific server I specify when I configure it to only use specific hosts. But that host obviously would not be connecting back to me.. so is that enough for BTSync to send to the Host my updates, and also receive updates that might have happened on the Host?

[rdebath]

You don't need to transport 3838 if you're using directly specified clients.

Port 3838 is used to send multicast packets across the 'administrative domain', so if you have multicast routers that know which networks attached belong to the same people these packets can travel across routed networks. When the client receives one of these packets it tries to connect to the source address of the packet using a normal unicast packet from it's "custom" port number.

In practice I doubt these packets will go outside the local and bridged networks very often.

They will NEVER go onto the internet. (In theory)

If port 3838 is blocked the "Search LAN" function will not work.

Note: IPTables can sometimes change the source address of a multicast packet to the firewall itself (MASQ and SNAT), in this case if a BTSync client later receives the packet it will attempt to connect to the firewall. This is unlikely to work.

[coewar]

Thanks for clearing that up.

Now, still I have my remaining question of a roaming BTSync instance... only the client would initiate the connection to the "server", is that enough to do a full sync, or does the "server" instance also have to initiate a connection to my roaming client? The roaming client of course, could not open any ports at all.

Regarding SSH vs OpenVPN.. the question is really what am I doing? And we are not tied to a data center with 4 walls; we have stuff all over in hosted sites. OpenVPN requires a router that can do it (or installing the software on Windows as a service? Haven't checked if that's possible other than maybe via Cygwin). But in my case, SSH is the perfect solution for everything else I'm doing. I typically utilize machines on hosted infrastructures and even develop from them. They have great i-net connection, whereas my roaming laptop may or may not. Plus, I never trust a laptop so if it dies or gets stolen, I literally lost nothing besides the hardware.. not even my passwords are stored on it.

So often I'm using SSH to RDP in and etc.. The performance of SSH tunnel is fantastic compared to OpenSSL-VPN.

Using SSH service also allows us to control our user authentication as we want it. Adding Zebedee in addition to that and not go through the existing authentication would just add headaches.

[coewar]

BTW, thanks for the tip about SSL-VPN. For some silly reason, our router had it defaulting to TCP. Changed to UDP and noticed a difference!

[rdebath]

There's no such thing as a BTSync 'server'. For two clients to be able to sync to each other (both directions) they must only be able to make a connection. This means that a UDP packet must be able to get from Client A all the way to Client B. The problem comes with NAT, if the 'server' (Client B.) is behind a NAT (or a normal firewall) it won't receive the packet from Client A unless a port is mapped to allow unsolicited packets from the outside world. This is the second function of the tracker; it tells both Clients about the other so they are both trying to connect and both poke a hole from inside their respective NATs (firewalls) and neither of the NATs thinks the packets are unsolicited.

...

OpenVPN doesn't need a router that can do it; There are native clients for Linux and Windows (i86 & x64).

But you're right, OpenVPN does need Administrator/root access so a simple Linux web host won't allow it. I like sshfs in that case. Though I have used OpenVPN with VM style hosters for both OSs. (gvpe is basically Linux, but can use OpenVPN's TAP devices with cygwin -- I'm not so keen on that)

...

As for stolen hardware; I normally set laptops up so it has one certificate that connects automatically. Had the laptop stolen; revoked the certificate. A couple of days later the laptop comes knocking on the firewall "connection refused, certificate revoked". The IP address went to the police. The perp spent Xmas in jail (and his sister got a night in lockup too, bet she was pleased with him ).

Of course, most of the users much prefer to work locally on their laptop, they get FDE so no laptop recovery I just hope BTSync will make for some good backups.

[lux]

So you added the respective other computer via "known_hosts"?

With lan_use_tcp you should not need to complicatedly route UDP traffic.

[rdebath]

No, but you do have to "complicatedly route TCP traffic".

You don't notice it in the case of a web browser because all you see is what the server feeds you and NAT is good at making you a simple consumer.

But if you want to connect TCP through back to back NATs you have to arrange for one of the NATs to be of a type that allows a SYN response to a SYN packet (rather than the normal SYN-ACK) and for BOTH the clients to attempt a connection at the same time with a very short timeout. It's rather unreliable to put it mildly.

Having said that, I don't know offhand what BTSync uses as it's "on the LAN" criteria; it might be "found this host with multicast" which wouldn't work at all.

But you're not wrong, it is easier to tunnel TCP traffic because there are more tools available to do it.

So I'll put something on the wishlist thread.

[lux]

From what I gather from other threads, putting IPs into the list of preset hosts defines them as "local". Might be getting this wrong though.

Don't know if 127.0.0.1:yourportofchoice works at all there. I've done SSH tunneling before with my server but I don't need it for BTSync.

[rdebath]

Okay, I just tried it; it looks like we get a mixture of UDP and TCP connections while it's got something to do then it settles down (!) to a continuous stream of UDP packets once the connection goes idle.

If I block UDP the connection times out, the peer disappears from the GUI and doesn't come back.

Even if I restart the clients. It's very persistent at hammering at the firewall rule though.

So, yes, predefined hosts appear to be treated as local, but no it doesn't help because UDP is still required.

As an aside I can also confirm that this config doesn't initiate any connections:

,"check_for_updates" : false

,"use_upnp" : false

//...

{

"secret" : "OO7AAAAAAAAAAAAAAAAAAAAAAAAAAAAA" // * required field

,"dir" : "/home/btsync/quiettest" // * required field

,"use_relay_server" : false

,"use_tracker" : false

,"use_dht" : false

,"search_lan" : false

,"use_sync_trash" : false

}

[coewar]

sifting through this again... going to get back into it after some of your follow-ups.

What does "lan_use_tcp" do? I tried that and it didn't help before. I can route TCP anywhere, but just wondering if there is some trick to it after glazing over what was written about SYN-ACK stuff.

[rdebath]

What does "lan_use_tcp" do?

It's a performance tweak only. When it's on the client will connect to other clients normally using UDP but when it has a lot of data to transfer (and it thinks it's peer is on the same lan) it'll attempt to open a TCP connection to the same IP and port number as it's peer's listening UDP port. With some network adaptors this connection will be offloaded, with others the TCP/IP network code is better optimised than BTSync's UDP data transfer. Most of the time it'll make little difference.