mcgroarty Posted April 4, 2014 Report Share Posted April 4, 2014 The release notes say that the Mac now supports extended attribute syncing. Wonderful! Has anyone spent time with these? I'm curious if you can answer any of... * Do extended attributes survive in a mixed environment? (Mac and non-Mac) * If extended attributes are stored on non-Mac machines, where are they kept? * Are extended attributes synced separately from files? For example, if I set Finder colors on 50 1G files, does it have to re-checksum 50G of data? Just the metadata? Any catches or nice surprises? Quote Link to comment Share on other sites More sharing options...
RomanZ Posted April 4, 2014 Report Share Posted April 4, 2014 * Do extended attributes survive in a mixed environment? (Mac and non-Mac)They survive mixed environments, but can't be synced to Linux machines (there is a limitation on Linux PCs on how xattrs should be named, and Mac xattrs does not satisfy this naming convention). On Windows - it depends on file system as they are stored in alternate streams. * If extended attributes are stored on non-Mac machines, where are they kept?Alt streams on windows, xattrs on ioses. * Are extended attributes synced separately from files? For example, if I set Finder colors on 50 1G files, does it have to re-checksum 50G of data? Just the metadata?Separately. They are treated as files inside files. In your sample xattrs should be synced separately without syncing or re-hashing of your 1Gb files. Any catches or nice surprises?xattrs / alt streams are synced on a whitelist (you can find the list where you saw the announcement). So if something falls behind the list - it won't be synced. Quote Link to comment Share on other sites More sharing options...
mcgroarty Posted April 4, 2014 Author Report Share Posted April 4, 2014 Thorough answer. Thanks a ton, RomanZ. It sounds like a great implementation. I don't know about others, but for me this was one of the last things tying me to Dropbox. Quote Link to comment Share on other sites More sharing options...
AzaK Posted April 9, 2014 Report Share Posted April 9, 2014 I have just started using Sync between a Mac and a Windows machine (HFS+ on the Windows machine via Paragon Software's implementation) and my extended attributes are ending up in the same directory as the source file, but with an extension ':com.apple.FinderInfo' for instance. It's definitely not being put into the alt streams, or back into the file on the HFS filesystem. Quote Link to comment Share on other sites More sharing options...
RomanZ Posted April 10, 2014 Report Share Posted April 10, 2014 AzaK The issue happens due to peculiarities of Windows API working with alt streams and implementation of HFS+ for Windows. BTSync has to store xattrs received from Mac machines in alt streams. Alternate streams on Windows are accessed with the same API as the rest of files, but application also adds ":<streamname>" line to access some particular name. So, to access to AltStream "ZoneIdentifier" of freshly downloaded installer, application opens a file "C:\Users\<username>\Downloads\MyInstaller.msi:ZoneIdentifier". Actually, HFS+ (in contrast to NTFS, FAT and other MS file systems) supports ":" in the file names and paths. So, any developer who wants to implement HFS+ for windows has to choose: either support WinAPI to access xattrs in the way as Windows does, OR - support colon symbol in filename. Paragon Software decided to support colon in the name of a file. To summarize - this is an issue of a HFS+ implementation for windows, rather then BTSync bug. Quote Link to comment Share on other sites More sharing options...
AzaK Posted April 10, 2014 Report Share Posted April 10, 2014 Hi Romanz, thanks very much for the explanation. Paragon implementing it that way makes sense so it's a properly compliant implementation. What would be good then, is for BTSync to know about the filesystems that are in play and utilise the correct behaviour. I assume that if the destination was running on OSX, then BTSync would use an appropriate method for alternate streams and things would work? Maybe there's a feature idea there. To handle accessing streams differently if the destination filesystem does not adhere to the ":stream" format. That would require some other way to access alternate streams on Windows and I'm not sure there is one. I really, really wanted to use BTSync but this is sort of buggering me up a bit. If I went and made sure my filenames were NTFS compatible, would all the extended attributes sync and restore correctly? Thanks Quote Link to comment Share on other sites More sharing options...
RomanZ Posted April 11, 2014 Report Share Posted April 11, 2014 I assume that if the destination was running on OSX, then BTSync would use an appropriate method for alternate streams and things would work?Of course. OSX has another API to access xattrs. Maybe there's a feature idea there. To handle accessing streams differently if the destination filesystem does not adhere to the ":stream" format. That would require some other way to access alternate streams on Windows and I'm not sure there is one.We'll consider it in future. If I went and made sure my filenames were NTFS compatible, would all the extended attributes sync and restore correctly?Not sure that I understand what you ask. If file system would be NTFS - BTSync will put synced xattrs to alt streams as was designed. Quote Link to comment Share on other sites More sharing options...
AlecEdworthy Posted April 17, 2014 Report Share Posted April 17, 2014 xattrs / alt streams are synced on a whitelist (you can find the list where you saw the announcement). So if something falls behind the list - it won't be synced.I've looked for this whitelist but couldn't find it. Any chance you could post a link to it please?Alec Quote Link to comment Share on other sites More sharing options...
RomanZ Posted April 18, 2014 Report Share Posted April 18, 2014 @Alec, Here you are: ExtAttr & streams are synced according to the whitelist. It can be adjusted in .SyncStreams file, with same templates as .SyncIgnore. Streams / xattrs that are being synced by default:com.apple.FinderInfocom.apple.metadata:_kMDItemUserTagscom.apple.ResourceForkcom.apple.metadata:kMDItemFinderCommentcom.apple.metadata:kMDItemStarRatingcom.apple.metadata:kMDItemOMUserTagTimecom.apple.metadata:kMDItemOMUserTagscom.apple.metadata:kMDItemOMUserTagTime Quote Link to comment Share on other sites More sharing options...
milleri Posted April 20, 2014 Report Share Posted April 20, 2014 I'd like to know more about the .SyncStreams files, but I can't find any mention of it elsewhere. Can you supply more information about how to use it? In particular, since I'm syncing from OSX to Linux I'd like to turn syncing of extended attributes off until it works. Thanks! Quote Link to comment Share on other sites More sharing options...
nils Posted April 20, 2014 Report Share Posted April 20, 2014 In particular, since I'm syncing from OSX to Linux I'd like to turn syncing of extended attributes off until it works. I have disabled them in each share by excluding the streams listed by @RomanZ in my .SyncIgnore. The reason for this was the incompatibility of streams on cross platform shares, e.g. OS X xattr cannot be written on FreeBSD fs. Quote Link to comment Share on other sites More sharing options...
RomanZ Posted April 22, 2014 Report Share Posted April 22, 2014 @milleri, .SyncStreams is a white-list file which can contain records about streams \ xattrs to be synced. It supports the same format as .SyncIgnore file, so wildcards can be used. However, to get rid of the xattrs being synced you need a .SyncIgnore, not .SyncStreams..SyncStreams just complements defauls xattr whitelist with any user's xattrs. Quote Link to comment Share on other sites More sharing options...
unsignedint Posted May 9, 2014 Report Share Posted May 9, 2014 I honor the design decision to accommodate people who need to replicate contents, including those xattr, but with this age of the internet (just like you wouldn't expect this to be ending up in the ZIP file, for example), I really think those attributes should reconsidered to be designed opt-in model, perhaps though .SyncStreams you have mentioned. 1. They don't seem to cooperate really well with encrypted nodes (since it can't know the file name, it can't effectively block those out from .Sync?)2. Possible name collisions -- what if someone's archiving, say the iOS source code, wouldn't there be possible that come. apple.foobar would be inadvertently excluded?3. And it can also inadvertently bloat the file I'm seeing right now 96 bytes of data alternating between completed sync and "↑ 96 bytes" to two of Linux encrypted nodes on my Mac right now... Quote Link to comment Share on other sites More sharing options...
benguild Posted May 9, 2014 Report Share Posted May 9, 2014 So will OSes that don't support these still "stick" on syncing them, or can we remove these exclusions on Mac OS X from the .SyncIgnore now? Quote Link to comment Share on other sites More sharing options...
mcgroarty Posted May 9, 2014 Author Report Share Posted May 9, 2014 I honor the design decision to accommodate people who need to replicate contents, including those xattr, but with this age of the internet (just like you wouldn't expect this to be ending up in the ZIP file, for example), I really think those attributes should reconsidered to be designed opt-in model, perhaps though .SyncStreams you have mentioned. The answer should be to get it working well in the default mode, not to turn it off by default if someone isn't confident about the implementation. Extended attributes are necessary for some apps to function properly or efficiently. Maybe not your apps, but there's a reason this nut got cracked early with the major cross-platform sync services. From my past experience, offering up specific, simple steps to reproduce problems that bother you and putting them in a support ticket as a bug report gets fast results. I had five or six issues fixed within 1-2 releases... sometimes they offered a developer build within the week! This team moves quickly. Quote Link to comment Share on other sites More sharing options...
markswift Posted May 16, 2014 Report Share Posted May 16, 2014 I'm also experiencing the issue with the finderinfo files showing up in the same directory as the original file, the file only shows up on my mac. Currently I am syncing between one mac and 2 windows machines. For now I have disabled all the stream in the ignore file on the mac. Quote Link to comment Share on other sites More sharing options...
markswift Posted May 20, 2014 Report Share Posted May 20, 2014 Guys, Adding the stream files to the .syncignore did not prevent them being created... I've not got hundreds of these all over my drives, is there a way to disable creation of these files? Is the appearance of them in the source folder normal? Any help would be greatly appreciated. Quote Link to comment Share on other sites More sharing options...
Devonavar Posted May 20, 2014 Report Share Posted May 20, 2014 @markswift:I spent most of last week figuring this out. It turns out you need to add the streams to .SyncIgnore on the LINUX machines, not just the source Mac. Specifically, adding:*com.apple.FinderInfo*com.apple.ResourceFork to the .SyncIgnore on every system is what fixed it. Oh, and you need to restart BTSync on every system as well.@BTSync team:At the very least, this issue needs an entry in the FAQ. I get that xattrs need to be synced, but the current implementation completely breaks Mac<->Linux syncing, with no explanation for why it doesn't work. I don't see why the functionality can't be selectively disabled on the Linux client until it's working properly; it literally just needs to add the appropriate lines to .SyncIgnore in the same way that .thumbs.db is added.In my judgment, that seems like a bigger bug than the one you are fixing. No sync at all is worse than incomplete sync for certain specific apps. Quote Link to comment Share on other sites More sharing options...
RomanZ Posted June 6, 2014 Report Share Posted June 6, 2014 @Devonavar Thank you for the feedback. We are implementing rather big changes and the issue with xattrs is going to be fixed in 1.4 release. Mac <-> Linux sync is going to be smooth and quick. Quote Link to comment Share on other sites More sharing options...
ctismer Posted July 8, 2014 Report Share Posted July 8, 2014 @ Sync Support Good to know about this work-around. Devonavar is very right here: This behavior should be added to the FAQ or get fixed, ASAP. The behavior of an affected system is quite weird, since it is hard to see a logic: - I have 10 folders synced between two macs- one folder is synced to my linux box- failure to sync one file with linux stops the whole synchronisation! This makes this system pretty unreliable: problems with one shared folder shouldnot in any way affect the behavior of other shared folders. Quote Link to comment Share on other sites More sharing options...
RomanZ Posted July 8, 2014 Report Share Posted July 8, 2014 @ctismer Updated FAQ. Also, the issue with syncing of one file should not prevent syncing others! If Sync behaves in that way - I'd advise filing a ticket to our support! Quote Link to comment Share on other sites More sharing options...
rjbailey Posted July 19, 2014 Report Share Posted July 19, 2014 @markswift:I spent most of last week figuring this out. It turns out you need to add the streams to .SyncIgnore on the LINUX machines, not just the source Mac. Specifically, adding:*com.apple.FinderInfo*com.apple.ResourceFork to the .SyncIgnore on every system is what fixed it. Oh, and you need to restart BTSync on every system as well. Suppose my source directory is on my Mac. Do you mean the same .SyncIgnore file needs to be copied to every synced folder and subfolder on the Mac before syncing? And after syncing to the Linux system--will I need to copy the same .SyncIgnore to all the cloned folders there too? I may not be up for all that. Quote Link to comment Share on other sites More sharing options...
milleri Posted August 28, 2014 Report Share Posted August 28, 2014 RomanZ reported that the issue with xattrs would be fixed in 1.4 release, and that Mac <-> Linux sync would be smooth and quick. Now that 1.4 is out, has anyone verified that this is working? Quote Link to comment Share on other sites More sharing options...
GreatMarko Posted August 28, 2014 Report Share Posted August 28, 2014 RomanZ reported that the issue with xattrs would be fixed in 1.4 release, and that Mac <-> Linux sync would be smooth and quick. Now that 1.4 is out, has anyone verified that this is working? Improvements to the handling of xattrs were indeed made for 1.4. Sync 1.4 introduces a new separate "StreamsList" file in addition to .syncignore (which for 1.4 has become "IgnoreList") - you'll find both these files within a folder's hidden ".sync" folder. StreamsList is file allowing you to specify alternate streams, xattrs and resource forks to "white list". Like IgnoreList (formally, .syncignore) It also supports "?" and "*" wildcard symbols. Quote Link to comment Share on other sites More sharing options...
rjbailey Posted August 28, 2014 Report Share Posted August 28, 2014 RomanZ reported that the issue with xattrs would be fixed in 1.4 release, and that Mac <-> Linux sync would be smooth and quick. Now that 1.4 is out, has anyone verified that this is working? It seems to work so far. I'll be watching... Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.