necula@cs.berkeley.edu

Sync-Ing The .syncignore File

Recommended Posts

I have done some experimentation (on Mac OS X) and it seems that the .SyncIgnore file is not being sync-ed. I think that this is counter-intuitive and I am not sure what the full implications are. If machine A has a .syncignore, and I setup a sync with machine B, the latter will not get the .syncignore but will get all the files except those mentioned by the .syncignore. If the ignored file is created on B (say it is an automatically generated IDE configuration file that I do not want to sync), will it try to send it to A ? What if I now share the same share with a machine C. Will B and C send to eachother the files that A would ignore?

 

I think that the semantics of .syncignore should be global, just like the share has a global semantics. All replicas have to be identical module the files in .syncignore.  

 

Thanks, George.

Share this post


Link to post
Share on other sites

Hi necula@cs.berkeley.edu,

 

Currently .syncignore is not synced automatically, and as you noticed - either BTSync need to be adjusted to understand cross-platform semantics of .syncignore, or it should sync it in a smart way to adjust semantics according to local OS.

 

Thank you for feedback, we'll consider it in future releases. I suggest also posting to wishlist.

Share this post


Link to post
Share on other sites

Where is the latest documentation for how .SyncIgnore works ? I have discovered (the hard way) that this file can only be at the root of the share. I have tried to make it a symlink to a SyncIgnore file that is synchronized, but the symlink is rewritten. And I am having trouble figuring out how to name a file in a given subdirectory that is not root. 

 

I also do not understand what is meant by the sentence in the FAQ: ".SyncIgnore ... will not work with the files that have already been synced." Does this mean that if I add a file to .SyncIgnore after it has been synced the file will continue to be synced? 

 

I do think that this file (or something like it) should be synchronized between replicas. You want to be able to declare globally what is the intended content of the share. 

 

Thanks, George.

Share this post


Link to post
Share on other sites

Currently .syncignore is not synced automatically, and as you noticed - either BTSync need to be adjusted to understand cross-platform semantics of .syncignore, or it should sync it in a smart way to adjust semantics according to local OS.

I think it is a bad idea to automatically sync .syncignore.

Because it forces the r/o nodes to behave in the way the r/w nodes dictate.

But any r/o node may wish to ignore some files, that are not in the master's vesion of .syncignore, like .avi or .mp3/4, etc.

In that case, even if he modifies the .syncignore, but, in the future has some kind of problem and decides to "Restore modified files to original version",

then his .syncignore will be overwritten from the master, and he may not even realize it, but he'll start downloading the humongous video or audio files that he would not want.

I think the cleaner solution could be co create a .syncignore.master file where master proposes to ignore some files, which r/o node may or may not decide to copy to his main .syncignore. This way, the user has control of his situation.

Furthermore, since master has some files in its .syncignore, that means that they will not even propagate to r/o nodes, in which case, what is the point of even having the same files in syncignore if you never receive them from the master in the first place? Do these files even exist in the database of the r/o nodes?

So, either "Restore modified files to original version" has to allow the selection of files and, possibly, "All" checkbox, or .syncignore should not be overwritten by the master nodes. Another alternative could be to automatically copy the .syncignore.custom file, which user has to accommodate his interests, to the main .syncignore file after user clicks "Restore modified files to original version".

It seems to me that the very idea of .syncignore is logically not quite consistent. Intuitively, each r/o node should have full fredom what it decides to ignore, regardless of other r/o nodes, that could have totally different syncignore files.

Secondly, if master wants to exclude some files from propagation and puts them in its syncignore, the r/o nodes should not even know about those files. It seems that they should not even be in their databases.

Thirdly, the user can override the master version of syncignore, at least the way it works right now, and master is not going to resync that file in the future because it will be marked as user modified file, not to be stepped upon.

Basically, the less rigidity you have in your design and the less you impose things on user from the master - the better.

Share this post


Link to post
Share on other sites

I am a new user of BTSync, and I have not considered the use cases you mention (r/o copies and "Restore Modified Files"). Is there somewhere an explanation of what can the user expect in the various situations that may come up? 

 

I think that having separate ignore files with different semantics might be fine. I think there should be a way to declare globally that certain files are not included in the share, for both r/w and r/o nodes. Those are local files. I am willing to help formulate a solution that works for all or almost all use cases, but I feel that I should start by reading about the uses cases. 

 

Thanks, George.

Share this post


Link to post
Share on other sites

I am a new user of BTSync, and I have not considered the use cases you mention (r/o copies and "Restore Modified Files"). Is there somewhere an explanation of what can the user expect in the various situations that may come up?

Well, basically, the propagation logic gets extremely complex if you realize the myriad of things that may happen, like user modifies, deletes or renames some files (on r/o nodes), and then changes his mind and wants it all back in the original state, or file time stamps have been changes accidentally, different r/o nodes are in different state of completion, and on and on and on, the net effect may be not what user wants.

So, to "fix" all these situations there is a "Restore Modified Files" feature, which simply means: go to the database and reset the "non-updatable" and "non-propagatable" bits of all the files, or whatever actually happens in btsync.

That, in turn, causes the r/o node to "get in line" with the master node and any funk he might have created is simply nulled out and he gets the latest version of the master files, regardless of anything. At least that is the way I understand it at the moment.

In terms of proposing any solutions or improvements, the product is not mature enough at the moment and this selective sync issue is quite a significant issue and plenty of people are talking about its effects.

But I think the issue needs to be addressed in a more direct way. That is, provide a GUI based dialog box of Win Exploer type where user can click on any files or folders and exclude them from the sync, like it is done in may systems that insert the custom behavior to the Win Explorer and allow the user to make changes to defaults in terms of file selection.

Yes, ultimately, it may be linked to automatic generation of .syncignore and its contents may be read to enable/disable the file/folder selection checkboxes in GUI.

But, any way you cut it, I think the assumption that all the r/o nodes have to have the same .syncignore files is invalid, unenforceable and not quite logical, at least as it stands right now. The common paradigm is a file/folder viewer where you can click on + to expand the folders and click on a checkmark to enable/disable it syncing. This is not something revolutionary, but pretty common nowadays.

Share this post


Link to post
Share on other sites

Where is the latest documentation for how .SyncIgnore works

The most precise information could be found in this topic:

 

http://forum.bittorrent.com/topic/28148-syncignore-behavior-not-as-documented/?p=79700

 

I also do not understand what is meant by the sentence in the FAQ: ".SyncIgnore ... will not work with the files that have already been synced." Does this mean that if I add a file to .SyncIgnore after it has been synced the file will continue to be synced? 

 

 

It means that the files that were already synced won't disappear from another peer after you add them to ignore list.

Share this post


Link to post
Share on other sites

The most precise information could be found in this topic:

http://forum.bittorrent.com/topic/28148-syncignore-behavior-not-as-documented/?p=79700

It means that the files that were already synced won't disappear from another peer after you add them to ignore list.

I assume you are talking about both, the r/o and r/w nodes.

Correct me if I am wrong.

In that case, can I assume that those nodes (r/o and r/w) are still going to propagate it to other nodes unless they are added to their .SyncIgnore?

And, once we are at it, do you have any objections to to the idea of using the .SyncIgnore.Master file instead of automatically propagating the .SyncIgnore, which will step on the local copies of it, at least on r/o nodes?

It seems that it at least addresses your concern about O/S dependencies and does not automatically override the individual preferences on the r/o nodes.

Secondly, if it is automatically updated on the r/w nodes as well, then, the original r/w node that created this share, which I call the "reference" r/w node, might get a mild attack by other rogue r/w nodes, if those, for example, exclude ALL the subfolders and files from propagation by simply modifying their .SyncIgnore file, in which case, ALL the nodes would stop being updated without even knowing it.

Or, the other r/w nodes might simply make a mistake or get the wrong idea in their heads about the original intent for this collection by the original reference r/w node, and he, being the author, might not like this idea because it may be damaging the the very intent behind this collection, which they may or may not know.

I am more interested in general public applications of BTSync and not in well controlled predefined private networks or well cooperating nodes in some virtual network. And in those situations, all kinds of things may happen and, even if you change the key to this collection in order to exclude some other r/w node, that have gone "rogue", it seems to me that the collection with the original key is still accessible, or isn't it?

Secondly, in general public applications how would you inform all the nodes that the key to this collection has changed, especially considering the case that some of them may not come back on line in days, or may be event weeks or months, and when they come, how do they know that the key was changed?

It seems to me that when you change the key, the only thing that changes is the access to the collection, but not the collection itself, and the databases on all nodes remain exactly the same as they were before the key change.

I'd like to understand more fully what happens if you change the key.

In particular, is it true that this collection is still accessible via old key except it is not going to be updated via NEW key and the only nodes that are going to get the updates are the nodes that have the NEW key?

Adding an additional message mechanism

or extending the size of Device name field

in order to provide more informative message to the nodes

In this context, in some other posts a while back I proposed to add some mechanism for the nodes to display an additional message from their node, where they, for example, may post some problems they are having, or post their requests for collection extension by the master or reference nodes, or the r/w nodes informing all other nodes of some changes to the collection, or give recommendations to those nodes that they notice having some problem syncing.

For example, I am having a few nodes that keep displaying the error message about their clock being different by more than 10 minutes. And they are sitting on a share for days, but their nodes are not updated for some unknown reason.

Also, if I recall correctly, I saw at least one node that simply would not start syncing and I have no idea why and, basically, have no way of asking him or recommending something to be done.

Right now, the Device name field is too short to display a meaningful message. So, I am kind of concerned with this issue because I would not like to see some people simply giving up on this share just because they could not even start syncing and would simply leave without me even knowing what was the cause of their problem.

I hope this issue is going to be looked at, and to me it seems like a priority item and not some "luxury" feature. Plus, considering the fact that this mechanism could be implemented within hours, if not minutes, at least in its most basic form, leaves me with hope that you guys will indeed look at this issue - the sooner, the better.

Share this post


Link to post
Share on other sites
In that case, can I assume that those nodes (r/o and r/w) are still going to propagate it to other nodes unless they are added to their .SyncIgnore?

 

Right.

 

And, once we are at it, do you have any objections to to the idea of using the .SyncIgnore.Master file instead of automatically propagating the .SyncIgnore, which will step on the local copies of it, at least on r/o nodes?

 

Complexity. Having 2 sync ignores will drive users mad. 1 should be enough.

Share this post


Link to post
Share on other sites
Guest proactiveservices

To stick with a single .SyncIgnore file, it could have mandatory section headers:

 

{{Sync with peers}}

desktop.ini

.temp

.part

 

{{Do not sync}}

Temp\*.doc

Share this post


Link to post
Share on other sites

To stick with a single .SyncIgnore file, it could have mandatory section headers:

 

{{Sync with peers}}

desktop.ini

.temp

.part

 

{{Do not sync}}

Temp\*.doc

 

Have a quick search of the forum for "syncinclude" - it's been proposed a number of times to have the ability to define "inclusion" rules rather than, or as well as, just "exclusion" rules (which is what we have currently with .SyncIgnore). It makes a lot of sense, and hopefully some sort of ability to define "inclusion rules" will emerge at some point!

Share this post


Link to post
Share on other sites

Have a quick search of the forum for "syncinclude" - it's been proposed a number of times to have the ability to define "inclusion" rules rather than, or as well as, just "exclusion" rules (which is what we have currently with .SyncIgnore). It makes a lot of sense, and hopefully some sort of ability to define "inclusion rules" will emerge at some point!

I see a logical conflict with the idea of syncinclude file.

Current approach uses "positive" logic. The default is "include all". From that set you can exclude whatever you want, which is logically consistent operation.

If you are proposing syncinclude, that is the "negative" logic (depending on how you look at it). In that case, what would happen if some file was in both, the .SyncIgnore and .SyncInclude file? How would you resolve that logic? And mistakes like these happen all the time.

So, it seems either you completely get away with .SyncIgnore if you propose the .SyncInclude, and you simply deal with the opposite logic, or the very semantic meaning of .SyncIgnore becomes unclear. Why do you need to ignore anything if your default logic is to ignore EVERYTHING?

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.