Destructive Sync? Metadata on Macs

[metafork]

Hi,

basically BitTorrent Sync is an absolutely great thing.

But: I just made an experiment trying to sync a folder between two macs, and filesystem metadata and resource forks are not being synced, meaning that BitTorrent Sync behaves destructive (I've installed 1.1.48 on both computers).

Is this intended behaviour or am I missing something?

I also think that this is an important question and it should be part of the main FAQ.

Thanks

[rusl]

Please refer to the section in the FAQ about ".SyncIgnore".

The Macintosh system uses invisible files/folders such as .DS_Store, .fseventsd, and .Spotlight-V100 to remember the preferences for your Spotlight searches, metadata about the viewing preferences for windows, etc..

These are ignored by default by BitTorrent Sync. You can turn on synching for these files and folders by changing the text files within .SyncIgnore. See the FAQ and the unofficial FAQ which are stickied in the forum for details about that procedure.

Also, version 1.1.69/1.1.70 has now been released and has many significant improvements in speed and efficiency.

Edit: What do you mean by destructive behaviour? I synch between two Macs using the default .SyncIgnore settings and haven't noticed anything destructive. Please expand on that description. Cheers.

[metafork]

Rusl, thanks a lot for your answer.

As far as I'm informed about the implementation of certain features of Mac OS X both resource forks and Finder labels (implemented as Extended Attributes AFAIK) are implemented as HFS+ metadata features. They are stored in .DS_Store files and ._ressource files only on non-HFS+ filesystems.

Both information is lost in the synchronisation process. While loosing finder labels could be acceptable, loosing resource forks isn't, as for certain file types (so called Text Clips or URL dropped to a Finder window e.g.) they contain the data, while the file itself is empty (this is what I mean with destructive).

A Mac OS X-compatible synchronisation system needs to support those advanced filesystem features and if not should IMHO display a strong warning that it doesn't.

In the meantime I found another thread answering my question negatively (http://forum.bittorrent.com/topic/18036-extended-attributes/). That's a pity.

Thanks again for trying to help. Bye.

[sagdusmir]

Don't mix up the meanings of resource fork and metadata/extended attributes. The mentioned ".DS_Store", ".fseventsd", and ".Spotlight-V100" files/folders and "extended attributes" and "metadata" are optional on only a nice to have for most users. Missing resource forks in contrast will effectively destroy most types of files on a Mac. On unsupported file systems you will eventually get ".AppleDouble" files to carry this information - but i guess this does not play nice with syncing on file level.

resource fork ≠ metadata

..and if not should IMHO display a strong warning that it doesn't.

A sync tool that does not get along with resource forks should never ever overwrite/replace/move/touch a file with resource fork on a Mac. (IMHO) Instead it should quit with a warning and ask the user to exclude all types of files with a resource fork before doing anything in terms of syncing.

[metafork]

Sagdusmir: Yes, you're right, but to my knowledge both extended attributes and forks are implemented with the same filesystem features in current HFS+. .AppleDouble files are history, current tools (like rsync and cp) use different suffixes.

Are you experiencing the same, i.e. that resource forks are not synced and therefore the files destroyed with BitTorrent Sync?

[sagdusmir]

.AppleDouble files are history, current tools (like rsync and cp) use different suffixes.

Are you experiencing the same, i.e. that resource forks are not synced and therefore the files destroyed with BitTorrent Sync?

I have btsync installed on some Macs and a NAS with HFS+ format. For me using btsync results in a mess of generated and removed (.SyncArchive) .AppleDouble files and unfinished syncs. I have not tried syncing files that I know of that depend on a resource fork.

So far I tried btsync on camera raw files and mp3 files.

[goli]

Hey there.

You're talking about things that are way more than just a couple of more options. One clearly can discuss this over and over and will always find something else to be added that's file system related.

Just to give some notes:

• Currently there is now privilege support. Not chown, not chmod. You want to introduce that? Sure, do su.
  
• Think about ACLs. That's a little more than just this and clearly is a valid use case. So add this, too.
  
• Switch to Windows, where you have way different access priviligation. Would be just fair to have those, too.
  
• Now think about Windows in AD environments. At the first sight, that's just like usual Windows priviliges. But it shows that you should not rely on local computer or account names (human readable stuff) but on UUIDs in that case instead. So it's different from synchronizing those privileges to ADless environments, unfortunately.
  
• You talked about Resource Forks. Ok, got that.
  
• NTFS has "Alternative Data Streams" here, which can be compared to solve a similar problem, but it's cleary implemented differently. And luckily it isn't widely used.
  
• Now think about FAT, which happens to be the default file system on e.g. androids. There's non of the mentioned previliges available, access rights are enforced by mount parameters only.
  
• And shares being placed on SMB or NFS. The privilege configuration doesn't necessarily match the underlying file systems possibilities, does it? I find it quite likely to have a cheap 2TB NAS mounted through NFS wich doesn't support btsync itselfe. So I would run the btsync on my computer and do the synchronization remote.
  

I woud suggest to not share those kind of data. Not a single one. No Resource Fork, no Alternative Data Stream, no Privilege. Just plain file name and binary content.

On the second step, one could define "comparable classes". Those could be

• no privilege support
  
• simple privilege support (chmod/chown)
  
• extended privilege support (Linux ACLs)
  
• NTFS
  
• HFS+
  

And this list is to be continued, of course.

This should be a basic configuration parameter for each share, and as soon as a node has one of those options (exclusively selectable, so no combination of those possible) selected, it should simply refuse to share data with nodes that are configured in a different way.

The thing is: As soon as you do other magic, it's likely to connect a non-capable node and corrupt your data.

So I really would like to see none of those features being activated by default.

Kind regards,

Stephan.

[sagdusmir]

One clearly can discuss this over and over and will always find something else to be added that's file system related.

Full ack. Then btsync should stop creating .AppleDouble files in the first place.

[metafork]

I can totally accept the situation that Btsync won't support full filesystem features, but it's not an option for me then.

But it should be still noted on a prominent place, at least the FAQ, that those features are ignored (and files synchronized incompletely).