Gerhard Uhlhorn

Importend extended attributes are missing

Recommended Posts

There is a big problem:

Not all data will be synced on a Mac: the extended attributes (metadata) are not synced. But they are extremely important!

Some examples:

com.apple.quarantine shows that the file is downloaded from the internet and may be dangerous.

com.apple.diskimages.recentcksum contains the very important checksum of disk images. Without this attribute it is not possible to verify the disk image!

Some Editors store information about text encoding. But without the extended attributes this important information is lost.

com.apple.FinderInfo stores also the very important Spotlight tags. I have tagged all my files, and without syncing this very important information the files will get unusable for me!

Files without meta data are virtually corrupt files, because important informations are gone!

PLEASE SYNC THE IMPORTANT META DATA!!!

Sent from my iPhone using Tapatalk

Share this post


Link to post
Share on other sites

Hey there.

 

As far as I'm concerned, I'm really happy that btsync does not try to cover every single unimportant file system feature.

 

Since that's definitively a non-ussue for me but happens to come up every four weeks here, I suggest you to post to the wishlist thread.

 

Have a look at this discussion, I elaborated this a lot:
http://forum.bittorrent.com/topic/22882-destructive-sync-metadata-on-macs/?p=64350

Kind regards,

Stephan.

Share this post


Link to post
Share on other sites

Oh, these are not unimportant file system features!

 

E.g. in our company we mark our documents with color tags and give them a status: Red means stopped, orange is on work, yellow means, that the document is on hold für clearance, green is ready, blue is a successfully closed job and so on.

 

What do You think will happen if this important information is gone on sync? Our company has to stop working. The company will fall in chaos!

 

A sync without extended attributes is an unusable sync!

Share this post


Link to post
Share on other sites

Considering how easily extended attributes are lost, relying on them for important information seems insane. Sounds like what you need for your company is some actual project management tools.

 

Anyway, the issue with supporting EAs is that they're very filesystem/OS specific. Sync is supposed to be a platform-independent tool for syncing files. Haven't you noticed that it runs on a ton of non-Mac platforms?

Share this post


Link to post
Share on other sites

Hey there.

 

I used the expression "unimportant" for quite a purpose: I tried to make you think about if it's wise to use such a highly platform specific thing, rely on it and expect others to support it, too.

This obviously didn't happen. Something different happened instead: You feel offended. That was not intended. So sorry about that.

 

Did you read the thread I gave you? Especially: Did you read the very post of mine I gave you?

I think I explained quite a lot reasons why I expect btsync to cover only the very common file system features -- only those features that can be found on all file systems.

 

You run your business on those extended file system features? Well, that's a realy brave one, imho.

 

What happens if one of your colleges joins with a Windows computer or Linux? What happens when sharing network files via SMB or NFS? What about access to those files via smartphones, such as Android or Windows Phone? I expect even iOS to have difficulties to access those.

 

Don't get me wrong. If you want to run your business like that, that's up to you. Even though it binds you to a very small group of capable environments. If it fits your needs, do so. If you have utilities for all kinds of jobs you need to do, I'm glad about it.

But you definitively do know about those limitations. And you definitively do face this particular problem every couple of months with any new nice common tool that doesn't work with your restricted setup.

 

A synchroniziation tool not covering extended attributes is not unusable in general. See the thread I linked. There are lots of people having needs of individual features. And calling your particular feature more important than others (even though they are quite compatible) is close to being presumptuous.

 

And that's what and why I suggested in my first post: If you feel it's that important, add it to the wishlist thread.

 

But alling it a bug and ciritical show stopper is just not true.

 

Regards,

Stephan.

Share this post


Link to post
Share on other sites

Hello,

I have to agree with Firon and to some extend with Goli. I do not believe that extended attributes or any other metadata added to files through specific file system features should be synced at all. There are too many features on different platforms. The core capabilities of Sync should be enhanced more and the whole solution should focus on more stability and reliability. Whats the big reason to support special features of a given filesystem like HFS+ while there are other filesystems with other nice capabilities. The main reason for using Sync is its decentralized synchronisation of files and folders. If you would want to redistribute complete filesystems you should think of more centralized solutions. Don't get me wrong I see your need for these features, but I think you should really consider something different. For example what do you plan to do if another OS gets involved like Windows or Linux with its great variety of filesystems? What happens if one of your machines corrupts its filesystem and starts to sync it all to the rest? There are a lot of other solutions to implement extra metadata like full document management systems which take care of stuff like that.

So to me it is a very bad idea to implement special features of a single filesystem, not just for compatibility reasons more for your own data safety. 

So long

Doc Green

Share this post


Link to post
Share on other sites

Sounds like what you need for your company is some actual project management tools.

No, we are working like this since more than ten years, an it was always working fine for us. So, why should we spend money and time into a project managing tool?

 

 

Sync is supposed to be a platform-independent tool for syncing files. Haven't you noticed that it runs on a ton of non-Mac platforms?

Yes. And what is the problem?!? Dropbox is syncing this data without any problem. But if this is to heavy for BitTorrent to make so too, then this software is nothing for me.

 

 

goli

I don’t want to have a sync of EAs to other platforms. I only like to have a sync of EAs between Macs, because on a Mac these EAs are very important to most of their users.

 

All companies I’m working for as a consultant like to have a software like BTSync. But they are using Macs only and need the EAs. Without EAs this is a no go for them.

 

In one company I was able to install BTSync (due to the NSA problem). But they are back on Dropbox, because the problem with missing EAs.

Share this post


Link to post
Share on other sites

Lets look at this again from a different angle:

Every filesystem is designed to handle access to data on it. You want the filesystem to find and manage your data. There is a commonly known method to organize data by metadata which is basic and can be found in mostly every filesystem: names for files and maybe folders. This is pretty basic metadata attached to mostly every bit on a harddrive or whatever technology you use to store data. There can be additional data attached to it, also very commonly used is date of creation, last access/change and so on. So this is pretty basic stuff nowadays and they are handled differently by different filesystems but they are common. And usually there are similar handlers given by the OS to a program to gain access to the actual data. 

What is not common: what characters are available for this metadata, upper and lower case is already a difficult problem and is still unsolved. Furthermore this is handled differently by different filesystems it is even handled differently by the same filesystems on different architectures or implementations/versions, the journaling functions of HFS+ are not completely available on all Linux systems or its implementations if available at all. This is also true for an Apple scenario as you have pointed out. 

Why is that a problem at all if you use a Apple/Linux/Windows environment only? You have to make sure every Apple/Linux/Windows machine has the same set of capabilities, if you dont you will lose metadata at some point for sure. That renders such a capability useless if you strongly rely on it. 

Of course you can check for these special setups before starting a sync. Then you have to decide what you want to sync to another node if yours is not capable of that metadata. Should Sync fill all the missing metadata fields with zeros leave them completely blank? Ask the other node if it has any clue from prior versions of the file and so on...

That is absolutely solvable for homogeneous environments, but gets clearly very complex in herogeneous environments. At least you need some basic level of agreement on all participating machines what information can be exchanged without any kind of misunderstanding/corruption of data. At this point Sync or any other similar software development needs to decide what kind of filesystem specifics are worth to implement. And you have to specify a policy what to do with unsupported data. How difficult such a policy is, becomes obvious if you look at the update policy of Sync. A wrongly set timestamp on one node can sync old data and overwrite the legitimate newer data on another one. What happens if you use an old version of OSX where larger than 128kb extended attributes are not available and cannot be interpreted?

It seems obvious to cover the broadest and most common use of filesystems/data (name, date, size), any specific extended attribute or whatever you call it makes it harder to create a solution for a greater range of devices/systems. Any additional information provided by a filesystem is outside of the file itself and needs an additional piece of software to compute it. It has absolutely nothing to do with the stored data in a file and is specific to a filesystem.

Because of these considerations I believe one should not hurry with the implementation of special features of a single filesystem especially if they are not even a fixed featureset in a given OS environment. One should better have an eye on all use case scenarios and look into a solution which is most fitting for them all.

So even if Dropbox is capable of solving these very special use cases, it does not cover them all for sure. Extended attributes or file permissions are not transferred completely but maybe someone really needs this, I really wish I could use sync for that too. On the other hand there are solutions for such use cases already available AFP/NFS/SMB you name it and there are reasons why they are more centralized approaches: you need someone to enforce security policies, enrich data with additional metadata and manage all that. There is a reason why document management systems like Alfresco rely on central servers and special programs. They were specially designed to manage documents/files, Sync at the moment takes care of the storage of files not how you wish them to be handled by your users. And there is still enough work to be done to cover that function for a non-beta level. So why not focus on that or a while?

So long

Doc Green

Share this post


Link to post
Share on other sites

Egads.  All this hate for apple extended attributes.  Reliability for AE is really only a consideration in special cases like BTsync or with admins who are too lazy to properly handle the macs using services on their network.

 

Also; anyone who has used a project management system that integrates with apple labels for automatic status updating, or reads out status via labels would be sympathetic to their usefulness.  It is fantastic.

 

I am totally unsurprised by btsync not handling AE given the way it works.  It would be nice in the future, but *shrug* I have this feeling that coming up with a sensible way to sync ae is going to be a pain.

Share this post


Link to post
Share on other sites

It is not necessary to sync all AEs, but the tags are very important! And it should be no factor to sync them.

@ nils: Thank You. But if You have an 200 GByte disk image, and You change only the name of an file, BTsync have to sync the hole file again. This is not very useful. ;-)

Even if this method will work, it's much easier to use Dropbox instead of BTsync.

Share this post


Link to post
Share on other sites

That's the beauty of sparse bundle disk images, changes will only affect a few bands, which are 8 MiB each, see http://en.wikipedia.org/wiki/Sparse_image

 

Eyah. I did test sparse bundle images.  They do work as expected; etc only new/changed bands being synced.  I personally never tested this with dropbox due to the lack of possible need for it.

 

Kinda makes me consider that simultaneous mounting of the image file is quite possibly going to possibly shred it though.  The simple act of just mounting one triggers the updates of a few bands / modification date changes.

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.