.syncignore Behavior Not As Documented?

[aslamK]

Added a folder with video files to be synched across a LAN only. I wanted to sync files in subfolders and exclude any in the parent folder. The user guide has the following note...

 

Note that .SyncIgnore is applied only to the folder where it is contained and will not work with the files that have already been synced.

...so I figured that adding the following lines would accomplish what I wanted:

*.mp4*.avi

I put a .SyncIgnore file with these lines in the directory to be synched on both machines before adding the folder in BTSync. Upon adding the folders I find that everything in the subfolders gets synched except the media files with the above extensions. The web console (Linux) reflects this, showing about 60kB in the folder instead of a few GB.

 

Have I misunderstood how .SyncIgnore works?

 

My .SyncIgnore file is attached: SyncIgnore.txt

 

I'm using v. 1.2.82.

 

[dms2013]

You are doing the opposite. Add the name/extension of files you DON'T want to sync.

[aslamK]

You are doing the opposite. Add the name/extension of files you DON'T want to sync.

 

I _don't_ want to sync *.avi, *.mp4 files in the top-level/parent folder. I _do_ want to sync *.avi, *.mp4 files in the subfolders. Only the top-level/parent folder has the .SyncIgnore file, so why are the files in the subfolders not getting synched?

[dms2013]

It is per share and not folder. That's is my experience at least.

[aslamK]

It is per share and not folder. That's is my experience at least.

 

That contradicts the documentation which says, "Note that .SyncIgnore is applied only to the folder where it is contained..."

[RomanZ]

aslamK,

 

Actually .syncignore is applied to a sync folder and all subfolders inside. We'll reformulate user guide to be more precise.

[aslamK]

Actually .syncignore is applied to a sync folder and all subfolders inside. We'll reformulate user guide to be more precise.

 

Thanks, RomanZ.

 

Is there a way to have btsync ignore the files at the top-level as I described?

[RomanZ]

aslamK,

 

Yes. Please try to add to your .syncIngore next line:

/*.mp4

/*.avi

 

the leading "/" means "in this root folder only".

[stanha]

aslamK,

Yes. Please try to add to your .syncIngore next line:

/*.mp4

/*.avi

the leading "/" means "in this root folder only".

OK, I'd like to clarify the .syncIngore behavior once and for all and to add this info to the detailed user manual I am working on. So, please be patient with "stupid questions".

I have seen a few posts on .syncIngore, but they look either scatchy

or simply like guesses.

So, what happens if leading slash is not present? (*.mp4)?

Does it mean that it would be applicable to ALL subfolders on ALL the levels below the root folder of the share, which means that the folder traversal behavior is a default?

Does leading slash mean "override folder traversal"?

Because on the O/S level the spec *.mp4 would be interpreted as the immediate or current directory (as expanded by the shell), or top level folder for the share, [unless -R or -r command line flag is specified for many standard utilities].

So, is it correct to interpret the leading slash as the top folder for the share, and not the absolute path (root folder of a device on the O/S level)?

Can we assume that on Windows the above specs would be:

\*.mp4

\*.avi

and the only difference on Linux and Win is that you use backslashes instead of forward slashes?

Because I have seen some post somewhere, where one guy tried to use the following spec on Win:

E:\folder\share_folder\*.mp3

where share_folder is the top folder for the node, which means the absolute path on the O/S level. This seems to be incorrect as that would imply that you can access the files outside of the root folder for the share, which is absurd.

Or, would this syntax be correct as well?

Qutestions:

1) When you want to ignore things on certain level under top folder, say, for example, files in the immediate subfolder, is the following syntax correct?

/*/*.avi (ignores *.avi files in all the immediate subfolers of the root folder of the share)

/*/*/*.avi (the same, but on the 2nd deeper level)

3) Is syntax of *.avi equivalent to ignore in the subfolders of ANY depth?

4) I have seen spec like

/**/*.avi

What does double star mean, if this is a valid spec?

[aslamK]

aslamK,

 

Yes. Please try to add to your .syncIngore next line:

/*.mp4

/*.avi

 

the leading "/" means "in this root folder only".

This works, thanks.

 

It would help to have a doc that lists the complete .SyncIgnore syntax from the horse's mouth, as it were. This is particularly true for any syntax that varies from OS conventions like what @stanha points out.

[RomanZ]

So, what happens if leading slash is not present? (*.mp4)?

Does it mean that it would be applicable to ALL subfolders on ALL the levels below the root folder of the share, which means that the folder traversal behavior is a default?

 

Yes. Means - applies to current folder and all subfolders.

 

\*.mp4

\*.avi

and the only difference on Linux and Win is that you use backslashes instead of forward slashes?

 

Right. Slashes should be OS-specific. "\" for Windows, "/" for nix-based OSes.

 

E:\folder\share_folder\*.mp3

where share_folder is the top folder for the node, which means the absolute path on the O/S level. This seems to be incorrect as that would imply that you can access the files outside of the root folder for the share, which is absurd.

Or, would this syntax be correct as well?

 

One should use relative paths in syncignore.

 

/*/*.avi (ignores *.avi files in all the immediate subfolers of the root folder of the share)

/*/*/*.avi (the same, but on the 2nd deeper level)

3) Is syntax of *.avi equivalent to ignore in the subfolders of ANY depth?

 

Correct.

 

4) I have seen spec like

/**/*.avi

 

Did not encountered specs like this one. Double star does not make much sense, should be working same as single star.

[stanha]

RomanZ,

Thanks for your answers, appreciated indeed.

We'll add it to our version of the detailed user manual.

I hope you don't mind me being may be way too detailed

and may be ask some questions that might be obvious.

I just want to get every single thing we attempt to describe

in as detailed way in in every conceivable permutation imaginable.

So the whole thing should be clear even to the people with most

basic knowledge about computers.

[Mr_Moonlight]

stanha - looking forward to reading your user manual regarding this point.

 

I'm reasonably computer literate, but find this part of BTS rather cumbersome and am hoping that longer term when the product matures it will be better incorporated into the UI.

I quite like the way this can be achieved with FreeFileSync, by running a compare and then being able to right click on files to add to the ignore list.

[stanha]

stanha - looking forward to reading your user manual regarding this point.

Well, thanks to your request, here comes the first version:

Using .SyncIgnore file to exclude some files from syncing

From the manual:

.SyncIgnore is a regular text file (UTF-8 encoding) containing one file or folder per line.

You can use "*" and "?" wildcards in .SyncIgnore files?

"?" is a substitute for a single character instance and matches any character,

"*" is a substitute for multiple character instances.

Examples:

*.dat would exclude from syncing all .dat files types regardless of name i.e. myfile.dat, anotherfile.dat, etc.

abc???.dat would exclude from syncing files named "abc123.dat", "abcdef.dat", etc... but wouldn't exclude files named "abc1.dat" or "abcdefg.dat", etc

*.mp? would exclude from syncing file types such as .mp3, .mp4, .mpa, etc, but wouldn't exclude file types such as .mpeg

Video - exclude all the files in the directory Video on any level (depth) of the path.

Video/*.avi - exclude all the .avi files in the directory Video on any level (depth) of the path.

/Video - exclude the entire Video subfolder, but only if it is the TOP level subfolder.

The effects of .SyncIgnore are more subtle that it might look at the first glance and file specifications are not intuitively obvious and behave slightly differently than they do in the O/S.

In the following rules we will use the directory level notion, which is the same thing as directory depth in the file path. The 1st level is the root folder of the share. The 2nd level is a subfolder of the main folder, and so on.

Rules

File names are specified relative to the root directory (main sync folder), and not the absolute path in the file system.

So, the absolute path

E:\folder\share_folder\*.mp3 is incorrect specification and will not match anything.

To specify the subfolders, file names are specified according to the same exact rules as on your O/S in respect to directory separator characters, ie to specify the subfolders, use the backslash character ("\") on Windows, and forward slach character on Linux ("/").

For example: /Audio/*.mp3 or \Audio\*.mp3 will exclude all the .mp3 files in subdirectory Audio of the main folder, but not in any other directory or subdirectory of any other level.

If you specify some file name with or without wildcards, and your file name does NOT start with O/S dependent subfolder separator character ("/" or "\"), then btsync will check all the subfolders of all depths, from the root folder, down to the deepest level, and if it finds a file that matches the name(s) you specified, including the use of the wildcards, than those files will be considered a match and will be excluded.

For example *.mp? will match any file of .mp3, .mp4, .mpa types in all the subfolders regardless of their depth in the path.

If you want to exclude only files in the top level folder, then begin the file name with a directory separator character ("\" or "/").

For example: /*.mp3 will exclude the files of .mp3 type ONLY in the top level directory.

If you want to exclude only files on a certain subfolder level, then you begin the file name with directory separator character, specify the mach specification for that level, possibly using the wildcard characters, then separator again, followed by the file name spec.

For example, you have the following top level subfolders, Audio, Video, Books, and in the Books, you may have another subfolder, PDF, and you want to exclude only .txt files in any of top level subfolders. You would specify it as follows:

/*/*.txt - exclude all the .txt files in the top level subfolders only (immediate subfolers of the root folder).

/*/*/*.pdf - exclude all the .pdf files on the 2nd folder level deep.

/*/PDF/*.pdf - exclude all the .pdf files on the 2nd folder level folder called PDF. The leading /* will match all the subfolders in the top level subfolders (Audio, Video, Books). Then, on the 2nd level deep subfolder, it will match the PDF subdirectory, and in that subdirectory it will match any .pdf file.

Question about .SyncIgnore general behavior:

Does it mean "do not update this file from the master (r/w) node

(or any other r/o node that has the latest version of the file)?

Or does it mean "do not propagate this file to other r/o nodes? Or both?

Answer:

Both. BTSync checks ignore list before requesting files from remote

peers and never tells other peers that it has such file if it is in

ignore list.

[stanha]

Question:

What will be shown in the Devices tab if some files are excluded

via .SyncIgnore? Will it still show that the node is fully synced?

Or will it show some size still remaining to be uploaded?

Related question:

What does it mean if some node used to show that it is fully synced

and then it just keeps showing that there is still some size to be

uploaded, and never shows that it is fully synced since then?

Does it mean that the user on r/o node deleted, modified or moved

some files out of the sync path?

[RomanZ]

I hope you don't mind me being may be way too detailed

and may be ask some questions that might be obvious.

 

I'm here to answer questions.

 

What will be shown in the Devices tab if some files are excluded

via .SyncIgnore? Will it still show that the node is fully synced?

Or will it show some size still remaining to be uploaded?

 

FAQ recommends to keep .syncignore files same on all peers.

 

If this is true:

If some files are excluded, BTSync won't calculate them in files amount value and summary data size in UI. Also, it will show peer as fully synced when all non-excluded files are synced. 

 

If syncignore is different, next scenario applies:

Say, we have PC A and PC B with some folder synced.

 

PC A has *.abc files ignored. It also has bunch of other files as well as several .abc. After sync done, PC A will show that it is synced with PC B, amount of files and volume includes all files except .abc.

 

PC B has bunch of files as well as several .abc. After sync is done, PC B will show that it is yet to upload a number of files to PC A, their amount and volume is a sum of all .abc files that are on PC B.

 

What does it mean if some node used to show that it is fully synced

and then it just keeps showing that there is still some size to be

uploaded, and never shows that it is fully synced since then?

 

It might mean:

1) remote node is RO and some files are deleted / modified

2) remote node has some files ignored and simple won't accept them

3) remote node is mobile device and synced only part of files due to selective sync.

4) some issue occurs that prevents files from syncing completely (permissions, letters case bug, locked files, out of free space, etc.)

 

Does it mean that the user on r/o node deleted, modified or moved

some files out of the sync path?

 

Yes. One of the cases is when user changes / deletes the file on RO peer and settings prohibit from re-syncing it.

[Mr_Moonlight]

stanha - very good, many thanks!

[stanha]

stanha - very good, many thanks!

I am glad you are happy.

Can we assume that every single imaginable aspect of .SyncIgnore is covered now and not a single question remains?

To RomanZ:

Thanks a lot, appreciated indeed. It really helped to clarify some things.

Your latest comments have been added to the master version of the manual.

Looks like we've made quite some progress on this and related issues.

If you think that this issue is fully covered, then we will consider its

status: closed and fully covered (in our manual).

If not, it would be great if you shed some light on missing points

and/or some possible misunderstanding in my messages on this topic.

[stanha]

"Fighting" r/w nodes and magical appearance of duplicate files/folders

There is an interesting case of r/w nodes "fighting" with each other (in case you have multiple r/w nodes on some share). If you delete some files or folders on one r/w node, it won't help anything if you have multiple r/w nodes.

Even deleting it locally, will not prevent it from being re-downloaded again in few seconds from other master nodes if they are on line, and even worse, if you are the only r/w node on line and delete some file or subfolder and verify that it all worked fine, you may not realize that it was nothing more than illusion.

The same thing happens if you rename some file or subfolder, with even more amusing effects.

Because it still exists on other master (r/w) nodes. As soon as they come on line, you will get your deleted or old versions of renamed files or folders downloaded from them again.

Remember, all the fully synced r/w nodes contain the aggregate collection that is created as a result of UNION operation. Those files or folders that are absent on some nodes will be sent to them by the nodes that have them.

And even more interesting behavior is observed if you rename some subfolder. What happens if it was already uploaded to other r/w nodes is that now you have two different folders with the same contents, the one with old directory name on other r/w nodes, and the other, with new file name on your node, and, since your new folder contains the same exact files as the old one, then it will immediately start uploading that folder to other r/w nodes as a new folder, and the other r/w nodes will send you a copy of the folder with OLD file name.

Once the sync process for these folders is complete, ALL the r/w nodes will have two copies of the same folder with the exact same contents, but with two different folder names.

And, even more interesting, once you notice that you have 2 copies of the same folder contents but with different name on your share, you may try to delete the folder with the name you do not like, not even suspecting that it came from other r/w nodes and wondering about where did it come from. But it will be all in vain if other nodes have that folder. They will simply resend it to you again.

So, you MUST use .SyncIgnore if you decide to delete or rename some files/folders in case you have multiple r/w nodes, and some of them have already synced with those folders/files that you have deleted or renamed.

Furthermore, all other nodes must also have them in their own .SyncIgnore file, and they may not even suspect that they have some duplicate files/folders as a result of actions on other r/w nodes. So, before deleting or renaming some files or folders, first, enter their names in the .SyncIgnore file and save it, and only then you can delete or rename them.

Using .SyncIgnore Master file

In order to prevent some unusual or strange effects arising as a result of deleting or renaming some files or folders (see: "Fighting" r/w nodes and magical appearance of duplicate files/folders), what you can do is to create the .SyncIgnore.Master file on one of r/w nodes.

Also, in order for all the nodes to be in a consistent state and contain the same exact information as all other nodes, in case some files or subfolders are excluded from syncing for all sorts of reason, such as containing the private information, or scratch pad files, not meant to be propagated to other nodes, ALL the nodes need to have the same .SyncIgnore file contents. Otherwise, some of them will be updated with unwanted files and will propagate them to other nodes.

Note: .SyncIgnore file does not propagate between the nodes automatically. Because it is an O/S dependent file and Window's version of it won't work with Linux and vice versa because of difference in file naming conventions.

But the .SyncIgnore.Master WILL be automatically updated and propagated, just like any other file, if ANY r/w node modified it or extended it.

The beauty of this approach is that nobody can dictate to your node which files you do or do not want to update and propagate, as it would be the case if .SyncIgnore was automatically propagated. Because that would be imposing someone else's opinion on what you do or do not want to carry on your node, which does not make sense and is not a "democratic" approach.

The only thing to be kept in mind is that it is an O/S dependent file and so on Linux, all the backslashes ("\") in the file names have to be replaced by the forward slashes ("/") and all the blanks in file or directory names have to be replaces with "\ " to follow the Linux convention.

What it means is that all the nodes can be automatically informed about the master nodes' "latest and greatest" .SyncIgnore contents via .SyncIgnore.Master file. So, they can simply copy/paste its contents into .SyncIgnore file and do some editing for the Linux version of it in order to comply with the O/S file naming convention.

This should work great and prevent all sorts of files or subfolders being distributed unnecessarily.

Furthermore, all the discussions and arguments about whether to automatically propagate the .SyncIgnore simply resolve without a single effort of the part of BTSync development efforts and everything "works as is".

And even more important for general public distribution, where one node has no idea about the other nodes and where the information may be updated dynamically, at any point, there might be some changes desired to the .SyncIgnore file.

But how do people know about the "latest and greatest" unless the .SyncIgnore is automatically propagated from the master nodes? And if it IS automatically propagated, then the latest master version of it may "step" on your local changes that correspond to your personal view on what to carry and what not on your node.