Ro Nodes Got Out Of Sync And Some Files Do Not Update Any Longer

[stanha]

I just encountered a pretty nasty problem and it would be great if someone who knows what it means suggest some solution.

I wanted to make sure that in my sync main folder on the master node (r/w) I have the latest version of the files (kept in a separate directory tree). So I did a cp -ru (instead of -rup) command via Cygrwin, which should not have touched the file update time stamps of files that are the same, but, unfortunately, it did modify the file update times.

After a few minutes, I noticed that all the r/o nodes that are sitting on this share started to show that they are completely not in sync. When I realized that file times were changed, I did "Pause syncing" and then, when everyone have disappeared from Devices list, I hand copied the entire tree via Windows Explorer, so that it restores the old file update stamps.

Then, after I reenabled syncing, all the r/o nodes show that they still need over 100 Mb of files to be updated, but they do not update for some reason. But the only thing that changed in those files was the file modification date, which was set to more recent date than they originally had and which was restored after I hand copied the files again restoring the original file mod dates.

Furthermore, I hand modified one text file on the master node in order to see if r/o nodes are going to get the updated version. But not a single r/o node is getting the updated version of that file, and time stamp on it obviously later than when things got out of whack. That means that they will no longer receive the updates on some files.

(Latest update on the above point: Well, I just checked now if I modify one file I do want propagated if it would update the other nodes. At the moment there was only one r/o node and that one did indeed updated with newer version. But there does not seem to be any r/w nodes on line at the moment. So... as they say "the jury is still out" on that one. We'll have to see more action in order to find out if this is true. May be someone knows this case for sure and could comment on it. That would be appreciated indeed.)

I thought that if the only thing changes with the file on r/o node is the file modification date, then that file should not be downloaded again from the r/w node, and instead its modification time should be set to that of the master node. But this is not what seems to be happening.

What is interesting is that since I edited one file for testing and it obviously had a newer modification date and still it did not propagate, then the issue has to do either with directory, the creation date of which was also modified, or with file status on r/o node and it was permanently marked as non-updatable and non-propagatable.

Well, it look even worse than that now. I just looked at the Folders tab on the master node, and it show an error message:

"Error: Bittorent Sync can not identify destination folder".

What?

So, it looks like that all the r/o nodes started to download the same exact files, just because their file mod date stamp has changed. Then, when I temporarily paused the syncing, they have distributed the "newer versions" of these files among all nodes, even though file were exactly the same, and then, when these files were restored back with their original date stamps, they refused to adjust their local date stamps to those of the master and permanently marked all those newly downloaded files as "bad" - non-updatable and non-propagatable.

Why so? What am I missing here?

So, does anyone know what it means and what can be done?

They are probably fully synced because files with more recent time stamp would have the same hash - the contents did not change.

Unfortunately, there is no way to tell those nodes to click on

"Restore modified files to original version" button, which would probably fix

this problem and they would show fully synced again.

That is why I added a message on a Wishlist thread to either extend the device name column in Devices tab, or, better yet, add another column for the device message, which should have probably 1k characters in size, and, even if you can not see the whole thing, you should still be able to copy/paste it to your editor.

I keep seeing the need to communicate a message from device again and again as I see some people are having some kind of problems and can not start syncing, or their clock is way too off, and even if I set a bigger range of time so that their error message disappears, they still do not start syncing.

So, I'd like to communicate to the some message and they could probably reply to it by editing their device message. But this is another matter.

Thanks in advance if you have a solution, and, better yet, if you fully explain what is exactly going on in this situation in a clear, step by step way, covering all the angles of it. Because this one definitely deserves to go into a detailed user manual I am working on right now.

[RomanZ]

stanha

I suggest that you've removed the .syncID file. BTSync shows error "Error: Bittorent Sync can not identify destination folder" when you delete / damage .syncID and it no longer can identify the folder. Of course, any sync to such folder is impossible.

[stanha]

stanha

I suggest that you've removed the .syncID file. BTSync shows error "Error: Bittorent Sync can not identify destination folder" when you delete / damage .syncID and it no longer can identify the folder. Of course, any sync to such folder is impossible.

Thanks for the info. I bet you are right.

I recall vaguely that I have deleted every single file from the main folder when I saw 100 megs of files being downloaded back to the reference master from one of r/w nodes that had the older picture of the state of affairs.

So, when I copied those files back from Win explorer from the original source, I probably did not copy the .syncID file, not even realizing that it is a critical file to have around.

I am not going to ask what is .syncID file for?

[RomanZ]

I am not going to ask what is .syncID file for?

 

 

Well, you did . It contains unique identifier of the folder. The folder and it's content is identified by .syncID, not by folder name or path. 

[stanha]

Well, you did . It contains unique identifier of the folder. The folder and it's content is identified by .syncID, not by folder name or path. 

Well, but, just as I said, I was not GOING to...

And I am certainly not going to ask this one:

But why do you NEED it in the first place?

[RomanZ]

1) By this file we identify the very folder existence (if, say, sync folder is on a flash drive the fact that it disappears means that device was unmounted, not that all files were deleted)

2) By this file we identify that the folder belongs to current instance of BTSync.

[stanha]

1) By this file we identify the very folder existence (if, say, sync folder is on a flash drive the fact that it disappears means that device was unmounted, not that all files were deleted)

2) By this file we identify that the folder belongs to current instance of BTSync.

I see. Thanks. You are pretty helpful... I'd say not a typical in many support forums for other s/w products.

Is it also used in the case when key (hash for share) is changed?

And, if so, can I have 2 copies of it, - one for the old key, so I can continue and/or update some things on the old key, and the 2nd one - for the new key. So, even though the key has changed, I can switch back and forth between the versions, assuming I also change the key in GUI?

[RomanZ]

I see. Thanks. You are pretty helpful... I'd say not a typical in many support forums for other s/w products.

 

BitTorrent takes care of its users.

 

And, if so, can I have 2 copies of it, - one for the old key, so I can continue and/or update some things on the old key, and the 2nd one - for the new key. So, even though the key has changed, I can switch back and forth between the versions, assuming I also change the key in GUI?

 

Not sure that I fully understand your question. SyncID is a hash of secret. So it changes automatically as soon as you change the secret.

[stanha]

BitTorrent takes care of its users.

I LOVE to hear things like these!

Not sure that I fully understand your question. SyncID is a hash of secret. So it changes automatically as soon as you change the secret.

Well, what I mean is this:

I have given a r/w key to some guy who either went "rogue", or started doing something crazy without realizing the consequences.

So, basically, he can screw up the whole share pretty bad, especially if he has some not so noble intentions.

In this case, what are my options to prevent him from screwing up the share pretty bad?

It seems to me that about the only thing I have control of is to somehow change the key. But then I have an issue: how do I inform other nodes about key change?

So, I thought I could change the key and then I could go back to the old key by simply replacing the SyncID file, so that it corresponds to the old key and add or modify some file to put the message informing them of a new key.

Basically, ALL I want is to find the solution for the r/w keys going rogue and starting to screw up the share. Not sure it makes technical sense to you, but I do not know how to ask the question that makes sense technically to you.

Does it make sense NOW?

(Otherwise, I am going to cry...)

[dms2013]

I will answer this one.

You should only give the R/W key to trusted people.

 

give the Read-Only key to people who you can't trust or are like wild cats on the keyboard.

 

There is no way to change the R/W code by sending a command. If it were, it would be useless, as the rouge person could do that too, probably before you.

 

So now you need to generate a new R/W key and change it manually on all the R/W nodes.

Then you need to give the new RO key to the RO nodes, as the old one won't work anymore.

 

You can look in .SyncHistory for recently deleted(30days) files up to 1GB per default on all systems except for iOS and Android (I think).

[stanha]

File does not fully sync

I have noticed that some r/o nodes were at one point fully synced but then started to show various amounts to be still uploaded. Those amounts are usually relatively small, between 30 to 500 k.

Since installing 1.3.67 with this new tool tip showing which files are queued to be transferred, now I can see exactly what is the problem with those nodes.

Just now I saw one node that was showing 31k still to be uploaded. I turns out that one particular file was still to be uploaded. Its file size is 500k or so.

So, I modified that file to see what happens. At that time there were 7 nodes on line. All of them got updated normally, but that particular node would also show that the size to be uploaded was 500k, but when it was done syncing, it would again show the same 31k still to be uploaded.

Question: is it possible for the file to resync, but only sync those blocks that had a hash indicating they were updated?

If I recall correctly, I remember that someone said that files of less than 4 megs (do not remember the exact number) were being synced as a whole, but files larger in size would only sync those blocks that were changed.

[GreatMarko]

Question: is it possible for the file to resync, but only sync those blocks that had a hash indicating they were updated?

If I recall correctly, I remember that someone said that files of less than 4 megs (do not remember the exact number) were being synced as a whole, but files larger in size would only sync those blocks that were changed.

 

That's correct. Please see "When a file changes, does BitTorrent Sync transfer the entire file again, or just the part that's changed?" in the Unofficial FAQ