Serious problem with cases in mixed environment

[hawibtsync]

I had some problems in the past so I tried to reproduce it. Here we go:

 

Windows 8.1 with 1.1.82

Linux i386 with 1.1.82

Android with 1.1.33

 

Please have a look at the screenshot.

 

* I searched on the Linux box (Tower) for lower cases in front of a word.

* I changed that on the Windows PC for 6 files. For example: "Die Liebe in Zeiten Der Cholera.epub" to "Die Liebe In Zeiten Der Cholera.epub".

* --> All six files were added (not renamed). IMHO this is the first error.

* On the Linux i386 box I ended up with two files for all 6 examples ("Die Liebe in Zeiten Der Cholera.epub" and "Die Liebe In Zeiten Der Cholera.epub")

* On the Linux box I did delete the files with small capital. The other file was left on the device.

* --> Now all six files were deleted on the WIndows boxes (but the one with the big capital, not the ones I did delete on the Linux box). IMHO, next error.

* --> As a result they were deleted on all other devices.

 

The result: Renaming a file from uppercase to lowercase or vice versa destroys files!

 

Regards

[GreatMarko]

This is a limitation of the Windows operating system, and therefore, not actually a problem with Sync itself!

 

Let me explain - the files "TEST.txt", "Test.txt" and "test.txt", for example, can all co-exist in the same directory in Linux, and Linux will see them as three different files.

 

On Windows, however, it's not possible for the files "TEST.txt", "Test.txt" and "test.txt" to all co-exist in the same directory - Windows won't allow it/will assume its the same files.

 

Because Windows is unable to distinguish between these files, as a result, neither can Sync running on Windows enviroments - again, this is a limitation of Windows, NOT of Sync.

 

For this reason you should take extra care with your filenames when syncing between Windows and Linux, and ensure that you don't have files with essentially the same name, but letters in different cases, existing in the same folder.

[hawibtsync]

Windows limitation? Who cares? The product needs to handle this.

 

The main audience of BTSync _IS_ mixed environment where nobody knows what Operating System partners will use. So I would suggest a HUGE warning in the documentation before going GA. Something like "When using Bittorrent Sync in mixed environment don't rename files - uppercase to lowercase or vice versa - you will loose your data.". Leaving it as is will generate lots of problems - guaranteed ...

 

BTW, I'm no BTSync developer but wouldn't it be easy to find out that somebody tries to rename a file (case change) on the Windows platform? A proper warning could pop up then. It's just a suggestion...

[GreatMarko]

Windows limitation? Who cares? The product needs to handle this.

How can Sync handle this, if the underlying operating system itself sees "TEST.txt", "Test.txt" and "test.txt" as IDENTICAL!!?? - Short answer: It can't!!

 

Yes, ok Sync could potentially show an alert on Linux whenever you rename a file in any folder that Sync is monitoring, but this still won't resolve the underlying problem that's due to the physical limitations of the Windows operating system!

 

Whether such an "alert" is shown or not; you should still take extra care with your filenames when syncing between Windows and Linux, and ensure that you don't have files with essentially the same name, but letters in different cases, existing in the same Sync folder.

 

The same is true of certain non-alpha-numeric "special characters". For example, the following individual characters are not allowed in filenames on Windows: \ / : * ? " < > |

 

..again, a limitation imposed by the operating system, not by Sync!

 

Now, potentially this could/should perhaps be listed as a "Known Issue" within the Sync documentation, for those Linux users who aren't aware of this limitation of the Windows OS, but this is not a "bug" with Sync itself, it's a limitation of the operating system that has no easy solution, other than the workaround to be careful over the naming of "identical" files on Linux

[hawibtsync]

You don't get it.

 

Users of Bittorrent Sync will loose data if they ever rename a file from "A very important file.xls" to "A very Important file.xls". They will lose it silently. No warning, no error, no popup. In, say, 2 month you will notice that this file is gone for all partners, on all attached devices, on all platforms - forever. And SyncArchive is empty at this date...

 

BTW, the character limitations as mentioned are well known by the average user.

[nils]

You don't get it.

 

I am not sure you get the nature of BTsync at its current state: a synchronisation tool (with minimalist backup .SyncArchive implementation). BTsync is not here to obliterate a good working backup solution, which would be able to prevent data loss as described in your case. 

The first case of renaming files, which are added again, could maybe prevented by BTsync, if an md5/sha1 would be saved  for each file, but that might blow up the database for large shares as well. In short, it is not an easy solution and should go into a Known issue part of the documentation for the time being.

 

FYI: dropbox known issues

https://www.dropbox.com/help/145/en

[hawibtsync]

 

FYI: dropbox known issues

https://www.dropbox.com/help/145/en

 

 

Please read what Dropbox writes:

 

"[...] but one will appear as a copy of the original file and appended with case conflict."

 

It was always easy to find any problems with Dropbox. They append a hint to the files/folders that produce problems. With Dropbox you can always search for any problems that happened during sync. BTSync gives no hint at all. This is why I wrote "silently" - that's the real problem...

[Doc Green]

I did some testing, the behaviour is there and I tend to agree, that this might be a problem. But not a serious one to my mind. First of all SyncArchive prevents complete deletion of files. Second it is not true that Sync is simply silent. It does give you a hint via the notifcation messages and it shows changes in the history. It would be nicer if it sets off some kind of alarm, but for sure your data is not simply lost.

This is as mentioned an issue of the OS and its there for a ridiculously long time, not just causing problems with Sync.

 

So long

Doc Green

[goli]

Well, this isn't actually new:

http://forum.bittorrent.com/topic/23236-upper-lowercase-issue-in-heterogenous-environments/

 

If you think this a little bit further, you will reach this one:

http://forum.bittorrent.com/topic/22882-destructive-sync-metadata-on-macs/

 

The thing is: Where to stop and where to end.

• You expect lower-case/upper-case to be handled
• The Linux guys expect to have chown/chmod handled properly
• The NTFS guys increase this thought by expecting to have the NTFS privileges handled properly
• The HFS guys expect meta data to be handled properly, since on iDevices it's daily business to have valuable information in HFS meta data.
• Then the NTFS guys start thinking "hey, we have ADS, which is something like mac meta data" and expect them to be handled properly (although ADS on NTS is rarely used), too.
• The next step might be hardlinks to reduce file size, which is posible on various Linux file systems, HFS and NTFS.
• The next step might be junctions (the hardlink mechanism of NTFS), which alllows imho "hardlinks of directories" too, which is kind of a dynamic mount point but not an actual hardlink in Linux terms.

Do you get my point?

The thing is: btsync will never be able to handle every etch case.

And guess what: I call mixed environments etch-cases, since I expect closely connected groups (in terms of a social group, frendiship and family) to have equal environments. If you widen your group and start using btsync not as file synchronization tool but as an alternate file sharing plattform, then having heterogeneous environments might become more likely. But that's not what btsync intends to be made for.

 

But as long as the main scenario is "each single share is (in average) shared between three to fife devices of one or two individual humans", it's really unlikely to have the mixed file system capabilities on one side and people that are completely unaware of that fact on the other.

 

So, feel free to add your thoughts to the wishlist thread and add some explanation to it. I will not refuse calling it a valuable improvement. The btsync staff every once in a while announces that the wishlist thread is getting crawled and summarized in order to influence the products roadmap. So if so much people are bothered about this, the wishlist thread would be really the place to go.

 

But stop calling the current situation a major bug or serious problem, since it's expected behavior and has been discussed lots of times.

 

Regards,

Stephan.

[BeTheSync]

Hi All,

very interesting thread, thank you all for clarification,

but Data Loss (although the files are moved to the .SyncArchive) is not quite an option.

 

I'll add myself later to the wishlist as mentioned above, because renaming a file (adding some text which explain shortly the sync-problem) would be a good compromise here.

 

Just to add my scenario, because I have a question which is not covered here.

So please excuse my reopening:

 

3 Machines: Mac, Win, Linux (Synology DS). All running 1.2.73

 

1 full access shared folder. (origin is win) about 5328 files.

 

renaming a file on Linux from "Funky Is On (Funky Family).m4r" to "Funky is on (Funky Family).m4r"

will delete the file on Windows; throwing the file Funky Is On (Funky Family).m4r into .SyncArchive.

 

It will be renamed correctly on Mac.

 

So the file "Funky is on (Funky Family).m4r" now exists on Mac and Linux with the same case, but it does not exist on windows anymore.

 

 

This leaves the question for me, why, after deleting the file on windows and renaming it correctly on the mac, there is an inconsistency between the 3 synced folders?

 

The file "Funky is on (Funky Family).m4r" is missing on windows, but exists on Mac and Linux, AND it has the same case on those two systems. Why isn't it written to windows again?

 

 

 

 

 

[destiny117]

Hello,

 

I have just signed up to push this thread / issue, our scenario:

- We used Dropbox before but migrated to BitTorrent Sync because we have a (linux) server which is cheaper and always on

- Some users have Windows, some have Mac, inbetween is the linux server.

- Unfortunately, some Mac-users decided to rename directories / files like mentioned

- This seemed to be no issue at Dropbox, however messed everything up for us with btsync: both upper and lower case directories and files were synced to the Linux server, but failed syncing to clients (at least on windows). The client was stuck with Downloading xx GB but was obviously unable to do so.

 

This was with version 1.1.x of BTSync and we consider this a serious issue which should be handled and cant be compared to file-permissions or meta-attributes, or why do you think dropbox handles it?

We are now testing other solutions and even consider moving back to dropbox, because you just cant teach users enough in reality.

 

I'm not a linux-pro but i have not yet seen productive usage of mixed upper and lower case files / directories on linux systems; I guess its bad practice there too, however a solution might be implemented by offering Linux (and Mac?) users an option like 'Sync Case-mismatching files and directories seperatly?' which implies no windows peers will be connected.

Edited December 10, 2013 by destiny117

[tuxpoldo]

This is a limitation of the Windows operating system, and therefore, not actually a problem with Sync itself!

...

On Windows, however, it's not possible for the files "TEST.txt", "Test.txt" and "test.txt" to all co-exist in the same directory - Windows won't allow it/will assume its the same files.

...

Because Windows is unable to distinguish between these files, as a result, neither can Sync running on Windows enviroments - again, this is a limitation of Windows, NOT of Sync.

 

Sorry GreatMarko, this is wrong and not only now but since 1994 when I started writing system services for Windows NT. Look at the original WIN32 documentation of the function CreateFile : http://msdn.microsoft.com/en-us/library/windows/desktop/aa363858(v=vs.85).aspx

HANDLE WINAPI CreateFile(  _In_      LPCTSTR lpFileName,  _In_      DWORD dwDesiredAccess,  _In_      DWORD dwShareMode,  _In_opt_  LPSECURITY_ATTRIBUTES lpSecurityAttributes,  _In_      DWORD dwCreationDisposition,  _In_      DWORD dwFlagsAndAttributes,  _In_opt_  HANDLE hTemplateFile);

The magick happens if you merge in dwFlagsAndAttributes the bits FILE_FLAG_POSIX_SEMANTICS. The documentation says:

 

"Access will occur according to POSIX rules. This includes allowing multiple files with names, differing only in case, for file systems that support that naming. Use care when using this option, because files created with this flag may not be accessible by applications that are written for MS-DOS or 16-bit Windows."

 

Apart of that there are several good ways to handle such problems.