mcgroarty

Extended Attributes Now Supported - Details?

Recommended Posts

The release notes say that the Mac now supports extended attribute syncing. Wonderful!

 

Has anyone spent time with these? I'm curious if you can answer any of...

 

* Do extended attributes survive in a mixed environment? (Mac and non-Mac)

 

* If extended attributes are stored on non-Mac machines, where are they kept?

 

* Are extended attributes synced separately from files? For example, if I set Finder colors on 50 1G files, does it have to re-checksum 50G of data? Just the metadata?

 

Any catches or nice surprises?

Share this post


Link to post
Share on other sites

* Do extended attributes survive in a mixed environment? (Mac and non-Mac)

They survive mixed environments, but can't be synced to Linux machines (there is a limitation on Linux PCs on how xattrs should be named, and Mac xattrs does not satisfy this naming convention). On Windows - it depends on file system as they are stored in alternate streams.

 

* If extended attributes are stored on non-Mac machines, where are they kept?

Alt streams on windows, xattrs on ioses.

 

* Are extended attributes synced separately from files? For example, if I set Finder colors on 50 1G files, does it have to re-checksum 50G of data? Just the metadata?

Separately. They are treated as files inside files. In your sample xattrs should be synced separately without syncing or re-hashing of your 1Gb files.

 

Any catches or nice surprises?

xattrs / alt streams are synced on a whitelist (you can find the list where you saw the announcement). So if something falls behind the list - it won't be synced.

Share this post


Link to post
Share on other sites

I have just started using Sync between a Mac and a Windows machine (HFS+ on the Windows machine via Paragon Software's implementation) and my extended attributes are ending up in the same directory as the source file, but with an extension  ':com.apple.FinderInfo' for instance.

 

It's definitely not being put into the alt streams, or back into the file on the HFS filesystem.

Share this post


Link to post
Share on other sites

AzaK

 

The issue happens due to peculiarities of Windows API working with alt streams and implementation of HFS+ for Windows.

 

BTSync has to store xattrs received from Mac machines in alt streams.

 

Alternate streams on Windows are accessed with the same API as the rest of files, but application also adds ":<streamname>" line to access some particular name. So, to access to AltStream "ZoneIdentifier" of freshly downloaded installer, application opens a file "C:\Users\<username>\Downloads\MyInstaller.msi:ZoneIdentifier".

 

Actually, HFS+ (in contrast to NTFS, FAT and other MS file systems) supports ":" in the file names and paths. So, any developer who wants to implement HFS+ for windows has to choose: either support WinAPI to access xattrs in the way as Windows does, OR - support colon symbol in filename. Paragon Software decided to support colon in the name of a file.

 

To summarize - this is an issue of a HFS+ implementation for windows, rather then BTSync bug.

Share this post


Link to post
Share on other sites

Hi Romanz, thanks very much for the explanation. Paragon implementing it that way makes sense so it's a properly compliant implementation. What would be good then, is for BTSync to know about the filesystems that are in play and utilise the correct behaviour.

 

I assume that if the destination was running on OSX, then BTSync would use an appropriate method for alternate streams and things would work?

 

Maybe there's a feature idea there. To handle accessing streams differently if the destination filesystem does not adhere to the ":stream" format. That would require some other way to access alternate streams on Windows and I'm not sure there is one.

 

I really, really wanted to use BTSync but this is sort of buggering me up a bit. If I went and made sure my filenames were NTFS compatible, would all the extended attributes sync and restore correctly?

 

Thanks

Share this post


Link to post
Share on other sites
I assume that if the destination was running on OSX, then BTSync would use an appropriate method for alternate streams and things would work?

Of course. OSX has another API to access xattrs.

 

Maybe there's a feature idea there. To handle accessing streams differently if the destination filesystem does not adhere to the ":stream" format. That would require some other way to access alternate streams on Windows and I'm not sure there is one.

We'll consider it in future.

 

If I went and made sure my filenames were NTFS compatible, would all the extended attributes sync and restore correctly?

Not sure that I understand what you ask. If file system would be NTFS - BTSync will put synced xattrs to alt streams as was designed.

Share this post


Link to post
Share on other sites

xattrs / alt streams are synced on a whitelist (you can find the list where you saw the announcement). So if something falls behind the list - it won't be synced.

I've looked for this whitelist but couldn't find it. Any chance you could post a link to it please?

Alec

Share this post


Link to post
Share on other sites

@Alec,

 

Here you are:

 

ExtAttr & streams are synced according to the whitelist. It can be adjusted in .SyncStreams file, with same templates as .SyncIgnore. Streams / xattrs that are being synced by default:

com.apple.FinderInfo
com.apple.metadata:_kMDItemUserTags
com.apple.ResourceFork
com.apple.metadata:kMDItemFinderComment
com.apple.metadata:kMDItemStarRating
com.apple.metadata:kMDItemOMUserTagTime
com.apple.metadata:kMDItemOMUserTags
com.apple.metadata:kMDItemOMUserTagTime

Share this post


Link to post
Share on other sites

I'd like to know more about the .SyncStreams files, but I can't find any mention of it elsewhere.  Can you supply more information about how to use it?  In particular, since I'm syncing from OSX to Linux I'd like to turn syncing of extended attributes off until it works.  Thanks!

Share this post


Link to post
Share on other sites

In particular, since I'm syncing from OSX to Linux I'd like to turn syncing of extended attributes off until it works. 

I have disabled them in each share by excluding the streams listed by @RomanZ in my .SyncIgnore. The reason for this was the incompatibility of streams on cross platform shares, e.g. OS X xattr cannot be written on FreeBSD fs.

Share this post


Link to post
Share on other sites

@milleri,

 

.SyncStreams is a white-list file which can contain records about streams \ xattrs to be synced. It supports the same format as .SyncIgnore file, so wildcards can be used.

 

However, to get rid of the xattrs being synced you need a .SyncIgnore, not .SyncStreams.

.SyncStreams just complements defauls xattr whitelist with any user's xattrs.

Share this post


Link to post
Share on other sites

I honor the design decision to accommodate people who need to replicate contents, including those xattr, but with this age of the internet (just like you wouldn't expect this to be ending up in the ZIP file, for example), I really think those attributes should reconsidered to be designed opt-in model, perhaps though .SyncStreams you have mentioned.

 

1. They don't seem to cooperate really well with encrypted nodes (since it can't know the file name, it can't effectively block those out from .Sync?)

2. Possible name collisions -- what if someone's archiving, say the iOS source code, wouldn't there be possible that come. apple.foobar would be inadvertently excluded?

3. And it can also inadvertently bloat the file

 

I'm seeing right now 96 bytes of data alternating between completed sync and "↑ 96 bytes" to two of Linux encrypted nodes on my Mac right now...

Share this post


Link to post
Share on other sites

I honor the design decision to accommodate people who need to replicate contents, including those xattr, but with this age of the internet (just like you wouldn't expect this to be ending up in the ZIP file, for example), I really think those attributes should reconsidered to be designed opt-in model, perhaps though .SyncStreams you have mentioned.

 

The answer should be to get it working well in the default mode, not to turn it off by default if someone isn't confident about the implementation. Extended attributes are necessary for some apps to function properly or efficiently. Maybe not your apps, but there's a reason this nut got cracked early with the major cross-platform sync services.

 

From my past experience, offering up specific, simple steps to reproduce problems that bother you and putting them in a support ticket as a bug report gets fast results. I had five or six issues fixed within 1-2 releases... sometimes they offered a developer build within the week! This team moves quickly.

Share this post


Link to post
Share on other sites

I'm also experiencing the issue with the finderinfo files showing up in the same directory as the original file, the file only shows up on my mac.

 

Currently I am syncing between one mac and 2 windows machines.

 

For now I have disabled all the stream in the ignore file on the mac.

Share this post


Link to post
Share on other sites

Guys,

 

Adding the stream files to the .syncignore did not prevent them being created... I've not got hundreds of these all over my drives, is there a way to disable creation of these files? Is the appearance of them in the source folder normal?

 

Any help would be greatly appreciated.

Share this post


Link to post
Share on other sites

@markswift:

I spent most of last week figuring this out.  It turns out you need to add the streams to .SyncIgnore on the LINUX machines, not just the source Mac.  Specifically, adding:

*com.apple.FinderInfo

*com.apple.ResourceFork

 

to the .SyncIgnore on every system is what fixed it.  Oh, and you need to restart BTSync on every system as well.


@BTSync team:

At the very least, this issue needs an entry in the FAQ.  I get that xattrs need to be synced, but the current implementation completely breaks Mac<->Linux syncing, with no explanation for why it doesn't work.  I don't see why the functionality can't be selectively disabled on the Linux client until it's working properly; it literally just needs to add the appropriate lines to .SyncIgnore in the same way that .thumbs.db is added.

In my judgment, that seems like a bigger bug than the one you are fixing.  No sync at all is worse than incomplete sync for certain specific apps.

Share this post


Link to post
Share on other sites

@Devonavar

 

Thank you for the feedback. We are implementing rather big changes and the issue with xattrs is going to be fixed in 1.4 release. Mac <-> Linux sync is going to be smooth and quick.

Share this post


Link to post
Share on other sites

@ Sync Support

 
Good to know about this work-around.
 
Devonavar is very right here: This behavior should be added to the FAQ or get fixed, ASAP.
 

The behavior of an affected system is quite weird, since it is hard to see a logic:

 

- I have 10 folders synced between two macs

- one folder is synced to my linux box

- failure to sync one file with linux stops the whole synchronisation!

 

This makes this system pretty unreliable: problems with one shared folder should

not in any way affect the behavior of other shared folders.

Share this post


Link to post
Share on other sites

@markswift:

I spent most of last week figuring this out.  It turns out you need to add the streams to .SyncIgnore on the LINUX machines, not just the source Mac.  Specifically, adding:

*com.apple.FinderInfo

*com.apple.ResourceFork

 

to the .SyncIgnore on every system is what fixed it.  Oh, and you need to restart BTSync on every system as well.

 

Suppose my source directory is on my Mac.  Do you mean the same .SyncIgnore file needs to be copied to every synced folder and subfolder  on the Mac before syncing?  And after syncing to the Linux system--will I need to copy the same .SyncIgnore to all the cloned folders there too?  I may not be up for all that.

Share this post


Link to post
Share on other sites

RomanZ reported that the issue with xattrs would be fixed in 1.4 release, and that Mac <-> Linux sync would be smooth and quick.  Now that 1.4 is out, has anyone verified that this is working?

Share this post


Link to post
Share on other sites

 

RomanZ reported that the issue with xattrs would be fixed in 1.4 release, and that Mac <-> Linux sync would be smooth and quick.  Now that 1.4 is out, has anyone verified that this is working?

 

 

Improvements to the handling of xattrs were indeed made for 1.4.

 

Sync 1.4 introduces a new separate "StreamsList" file in addition to .syncignore (which for 1.4 has become "IgnoreList") - you'll find both these files within a folder's hidden ".sync" folder.

 

StreamsList is file allowing you to specify alternate streams, xattrs and resource forks to "white list". Like IgnoreList (formally, .syncignore) It also supports "?" and "*" wildcard symbols.

Share this post


Link to post
Share on other sites

 

RomanZ reported that the issue with xattrs would be fixed in 1.4 release, and that Mac <-> Linux sync would be smooth and quick.  Now that 1.4 is out, has anyone verified that this is working?

 

 

It seems to work so far.  I'll be watching...

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.