How Conflict Works?

[SamD]

Hi

I need help on this one, as I don't know what could be my next attempt.

Scenario: two computers, one-way synchronized with rsync for years (archive and delete option, that make that date-time is same on both side). More than 1,2To that I obviously don't want to upload again.

On both side I use a secret starting with D (RW-secret). The main folder is added to btsync, and indexing works fine (same number of files as it should be).

I verified using 'date' on both side (over ssh) that it answers the same time CEST.

Although I used rsync with checksum control in both direction, when synchronization starts, catastrophy... All files are marked as conflict and sent in both direction. Pure mess...

I have no idea what to do?

Any advice is gladly welcome.

Cheers, S.

--

BitTorrent Sync v. 1.3.105 (latest)

[SamD]

To what I undestrand, there is conflict when a file is different on 2 machines, that is, size is different and-or date is different.

Is checksum verified?

 

I tried to look at the logs to see if there is something written about conflict detection: it seems it is recording connection and nothing about indexing or conflict detection.

 

When a file is modified, just the portion of the file affected is transmitted, right?

Why, when there is a conflict, both files are fully transmitted?

[SamD]

Same problem here, but not solved it seems...

http://forum.bittorrent.com/topic/29508-sync-folder-on-two-computers-after-copying-files-to-each/

[SamD]

To BitTorrent Sync contributors:

Here is an idea. I could try to rsync back the btsync i386 application and associated files on my Mac so that files are hashed the same way. I could try to make a diff to see what's different in the database. Or I could just edit the downloaded db (but you'd have to tell me how) so that I can specify the path to the "equivalent" folder, thus using a synchronized db to avoid this initial conflict mess? What do you think? Does it have any chance to go anywhere?

[RomanZ]

@SamD

 

The .conflict files appear when one file on a peer is mapped to another file with different name on another peer. The most common use case is when files are recorded in different case between 2 Windows machines.

 

So the first thing I suggest to check if files producing conflicts has different letters case.

[SamD]

thx for reply @RomanZ

 

One side is Linux Ubuntu, other side is Mac Os X. So it should not be the problem.

As I said, intinial sync was made with RSYNC which preserves naming, case, and date (with the option t or equivalently a). Rsync is a command I apply from time to time, it is not running in the background.

And conflicts are not affecting some files, they are concerning most files (if any, I did not let corrupt the whole folder) and some subfolders.

 

OsX is setup in French, Ubuntu most probably in English or eventually in German. I don't see why this could affect anything (at least it is not an issue with rsync), but I mention this because of what you told in an other post.

[RomanZ]

@SamD

 

Do you have any files / folders containing some special French characters? If yes - could you please check if conflicted files are limited with files containing non-latin characters?

 

If this is not your case - then we'll need debug logs from your both computers to start finding out what happens in your case. Send them to syncapp@bittorrent.com, refer to this forum topic.

[SamD]

@RomanZ

 

Yes, most files affected have French or German or Swedish special characters like éèùàëîïöëçåßæœ and upper case.

It seems that in total it is 25% of the files that are affected: I did not remember it was so many

But some files don't contain special character.

So I am sending the logs.

 

Now what about it?

Why is there conflict produced by special characters?

 

I guess most of your European users would love that you don't limit yourself to US charaters

Do BTSync developpers plan to correct this soon?

[SamD]

I found a beginning of explanation here: https://rsync.samba.org/FAQ.html#2

It seems that synchronization is more an art than I thought. At least rsync people are giving a great hint here.

My best piece:

Something else that can trip up rsync is a filesystem changeing the filename behind the scenes. This can happen when a filesystem changes an all-uppercase name into lowercase, or when it decomposes UTF-8 behind your back.

An example of the latter can occur with HFS+ on Mac OS X: if you copy a directory with a file that has a UTF-8 character sequence in it, say a 2-byte umlaut-u (\0303\0274), the file will get that character stored by the filesystem using 3 bytes (\0165\0314\0210), and rsync will not know that these differing filenames are the same file (it will, in fact, remove a prior copy of the file if --delete is enabled, and then recreate it).

You can avoid a charset problem by passing an appropriate --iconv option to rsync that tells it what character-set the source files are, and what character-set the destination files get stored in. For instance, the above Mac OS X problem would be dealt with by using --iconv=UTF-8,UTF8-MAC (UTF8-MAC is a pseudo-charset recognized by Mac OS X iconv in which all characters are decomposed).

I would do more test about it as I indeed use HFS+ on my Mac.

 

Could BitTorrent Sync people tell us more about dealing with filesystem?

[RomanZ]

@SamD

 

Non-latin characters are somewhat complex to handle when you are dealing between different operating systems. BTSync should handle them well, but from time to time we encounter some glitches.

 

Please follow the Support guidance to resolve your issue.

[SamD]

HI,

Sorry it took me a while to give some feedback on this. I found the glitch.

 

I found out the BTSync was doing the right thing. I mean, I did not find that "conflicts" in BTSync is straightforward. It really is messing up files, and BTSync team might want to rethink the way conflicts are handled (for instance, letting the user to solve conflicts by some batch processing with simple actions). But the conflict reason was right.

 

Indeed, I discovered that I have very many files with French-German-Spanish-Portuguese-... accents in the filename. After quite some research, I found out that RSYNC was not used properly. If both Linux ext and MacOS HFS are using UTF8 encoding, MacOS is using a special UTF8 (of course why are the standards for?). So that I had to tell RSYNC to convert from Mac UTF8 to Linux UTF8, otherwise, as properly detected by BTSync, file naming is not the same on both side. in addition, file metadata (xattr) are not passed properly : I have no idea whether I shall bother, nor if BTSync bothers... I abandoned this issue.

 

So here is the correct way to use RSYNC for me so that no conflicts are detected by BTSync afterwards:

rsync -asz --progress -e 'ssh -p 22' --iconv=UTF8-MAC,UTF-8 --delete --backup --backup-dir=/home/me/RSyncTrash /Volumes/myDisk/Users/me/myFolder2Sync/ url2myServer:/home/me/myFolder2Sync

Sources: here is an explanation on how rsync deals with spaces in the filename (https://rsync.samba.org/FAQ.html#9) and here is a blog explaining that rsync on OS X is buggy (version 2.6) and should be upgraded to version 3.0.9 (http://elliotli.blogspot.se/2011/10/using-rsync-to-migrate-data-between.html).

 

Hope it will help.