Latest Desktop Build 1.4.103

[cubik]

@cubik

First thing to try is to increase send / recv buffers and set "low_disk_priority" to "false". If it does not help - let me know, I'll instruct you to collect profiling data to find out where is the bottleneck.

Thanks for the suggestions - I followed all three at 25/25 and false.  It didn't appear to make a difference.  I did then add a 2nd peer on another server just to see if it would speed things up having another leg.  Still fluctuating, but at approximately 3x the speeds, and for longer duration of burst.  I'm happy enough for now.  I did check the logs and came across many "status:137 error:<NULL>" entries after a long run of "Piece 44... completes"

 

In-between this, I also completely removed the AppData folder, .sync files from the shared folders, and installed 1.3.109 and rebuilt the syncs.  The speed was much more stable, but approximately 1/10th what 1.4.103 could reach for a burst.

 

I control my PFSense firewall with no shaping/qos/snort etc. behind a 100/100mbps WiMax link.  I also have direct access to my datacenter techs that say no changes on their end with firewall, and no p2p throttling of any kind for my rack network.  1gbps on that end.

 

I love seeing your billboards around SF.  Amazing work.  Can't wait to try the Chat tools, and V2 of BTSync.  Cheers.

[RomanZ]

@cubik

The "error:<NULL>" line in log just indicates the absence of error. I suggest collecting profiler logs and sending to us for analysis - we'll try to see where the bottleneck is.

 

@pjmsullivan

There are a number of cases which can introduce "old files overwrite new ones" issue, please check if you are subject of any:

- Application crash / unexpected kill (like, kill -9 on Linux).

- Shutting down peer with enormous amount of files and forcefully killing the app (same as previous)

- Files with mtime from future (will always overwrite files on other peers)

- Files updated / moved on 2 or more peers while one peer has Sync off.

 

@b0rman

Use the link dropped by @GreatMarko as instructions and drop me a profiler data. We'll try to locate bottlenecks in your setup.

[pjmsullivan]

@RomanZ

From what I understand, some programs save an empty file first, then fill it with data.(?)  Based on the logs I sent in, the empty file syncs with the other peers, but then overwrites newer files with data with the old blank file.  The file below was a pdf saved with a network scanner to peer 44F1 (filename "20141126185459.pdf").  These debug log excerpts are from peer AD63.

[2014-11-26 17:50:56] FC[C5F4]: got file from remote (192.168.1.6:15670): 20141126185459.pdf state: 1 type: empty total:0 have:0 t:1417053056 mt:1417053050 ot:38233 o:peer ADCA h:0000000000000000000000000000000000000000

Then, this peer (AD63) overwrites the file on peer ODBA:

[2014-11-26 17:50:57] MC[C5F4] [0DBA]: Local file 20141126185459.pdf is newer than remote t:1417053056/1417053051 ot:839/55829 o:this peer (AD63)/peer 0DBA

Finally, peer 7402 overwrites the empty file on this peer (AD63) with another empty file.

[2014-11-26 17:51:02] MC[C5F4] [7402]: Local file 20141126185459.pdf is older than remote t:1417053056/1417053066 ot:839/79281 o:this peer (AD63)/peer 44F1

This behavior is very strange.  I can't figure out what is going on.  The clocks are all within a second of each other.  Edit: It appears that the time difference between the files is very small - though I don't know exactly how to read the time codes.  

 

There are no app crashes, or kills on linux (no linux box online).  Could it be that the clock offsets for the various machines are off?  What would cause a peer to write an mtime in the future?

 

There may have been a forceful kill in the past - would that cause these problems? 

 

How can I fix this? Should I uninstall and reinstall on every peer?

[RomanZ]

@pjmsullivan

 

Various apps use different ways to save data (some allocate disk space and write data there, some create empty placeholder and then replace it by renaming temp file) - actually Sync should track all these changes and apply'em accordingly to other peers.

 

The time is saved in unixtime format - there are a bunch of online converters, briefly - it is amount of seconds passed since certain point in the past (Jan 1 of 1970).

 

The app crashes and kills have only immediate effect, i.e. file overwrite happens after Sync (which crashed) starts again.

 

How can I fix this? Should I uninstall and reinstall on every peer?

I don't have immediate answer to that. I strongly suspect that DB cleanup (i.e. re-adding folder) will solve the issue for some time. I'd like to get full debug logs from both computers for this issue (if it is reproduced easily) and see if we can find the traces on what really happened there.

[pjmsullivan]

@RomanZ

Thanks for the answer.  I will try to get logs for the other peers and reproduce the problem.  Will that be useful to identify a bug, or is this unique to me?  

[RomanZ]

@pjmsullivan

It is not unique and we are getting reports from other users about same issue. Though - we don't have enough information yet to make a fix for the issue, so your help would be highly appreciated here. Turn on your debug logs, increase your debug log size just to make sure everything fits there - and send logs to me when issue reproduces.

[pjmsullivan]

@RomanZ - I will do so.  I'm working night and day until Tuesday, but I'll do so then.  

 

Best,

Patrick

[kratochviljan]

One cosmetic issue in webui:

(Menu is rolling under the upper bar)

[kratochviljan]

And one serious issue in "listening port" settings.

Port settings under shared folder in "Use predefined host" settings has absolutely no effect to communication between peers.

As you can see on this screenshot from my test environment, there is port 11111 set for both peers.

 

 

But all communication between peers going thru ports 4096 which is set under global preferences in "Listening port:" field.

I made few experiments with those settings and my result is, that only matters on global Listening port. You can type whatever you like at shared folder setting because this settings is ignored.

 

As you can see on this screenshot which is made on the same machine "Listening port:" is set to 4096.

I also made some basic network monitoring with netstat, output can be seen on upper part of the screenshot.

There is only port 4096 opened. Neither port 11111 which is set under folder settings was not opened, even so sync is running fine. 

 

 

 

Basically, is there any need for setting port under shared folder / predefined host settings ?

I think that this option is useless. There is no need to set other port under each synced folder.

Each peer has its own listening port and this should be enough.

 

Or I misunderstood your intention ?

[GreatMarko]

Basically, is there any need for setting port under shared folder / predefined host settings ?

I think that this option is useless. There is no need to set other port under each synced folder.

Each peer has its own listening port and this should be enough.

 

Or I misunderstood your intention ?

A couple of things to remember, firstly, the "Pre-Defined Host" settings essentially give Sync a "hint" at where other peers are located. Setting pre-defined hosts for a folder does not mean that Sync will use them "exclusively" to connect.

Secondly, after entering/changing pre-defined hosts you may need to restart Sync for changes to take affect.

[patoka]

>>Setting pre-defined hosts for a folder does not mean that Sync will use them "exclusively" to connect.

 

What if relay server, tracking server and search LAN are off? Wouldn't a pre definded host list be a way of making sure only a set number of PCs could sync?

[RomanZ]

@kratochviljan

The "Listening port" actually forces Sync to listen selected port and use it for all possible communications (it still will use different for outgoing TCP connections).

 

Setting predefined host just indicates Sync that particular IP will have certain port open. It doesn't force remote Sync to open this port, and it does not prevent local Sync to use other means (Tracker, Relay, multicast, cache) to find other peers.

 

@patoka

If you disable tracker, relay and LAN search - sync still will attempt predefined hosts as well as look into it's cache of addresses (hold for 7 days by default).

[patoka]

Is tracking server more secure than relay server? The info it the UI guide hints at that.

 

I'm assuming pre-definded hosts were designed for WAN IPs to avoid a 3rd party relay and don't have an advantage over search LAN in a local network.

 

[jmr01]

 

There are a number of cases which can introduce "old files overwrite new ones" issue, please check if you are subject of any:

....

- Files updated / moved on 2 or more peers while one peer has Sync off.

 

@RomanZ

Please explain this further. Does Sync always need to be "on" on all peers whilst editing files that are in the sync folder? I use a laptop that I normally only sync when in the office. If I'm out of the office on battery power I will turn off sync and other services to save battery broadband allowance.

[GreatMarko]

Does Sync always need to be "on" on all peers whilst editing files that are in the sync folder?

 

No, your devices don't all need to be online to edit a file.

You can edit a file with devices offline, and changes will sync the next time they are online

[jmr01]

Please explain the statement  "old files overwrite new ones"  when "Files updated / moved on 2 or more peers while one peer has Sync off."

[RomanZ]

@patoka

They both are very secure by one simple reason: they do not have access to your data.

When your Sync connects to tracker - it sends its share IDs (which is public and gives no hint about content or key) and local IP. Tracker keeps the list of connected shareIDs and lets other peers know a global and local IP:port for new peer. The rest is done without tracker.

Relay server simply proxies data packets from one peer to another when direct connection is not possible. As packets in global network are encrypted by default - relay has no idea what is passing by.

 

The predefined hosts can be used when either you don't want to use tracker server or when you have complex network setup and want explicily let Sync know which IP and port use to connect to remote peer.

 

@jmr01

There is a known case when Sync can overwrite new files with old ones. It is bound to the way how Sync monitors file changes. It reproduces when same file was changed on 2 peers and Sync was off on one of them. When Sync starts up, it will believe that file that was changed while it was off is the newest and might overwrite this file on second peer.