List Of Unsynced Files

[vogon]

How can I get the list of files which are not synced?  

 

The "Devices" tab shows the amount of data which should be pushed to a node but I cannot find a way of pinning down the missing files. This is useful in case of stuck syncs.

 

BTSync should know which files are missing, since it reports the total amount.

 

BTW It would be important to notify a sync problem via the menubar icon (e.g. OSX).

 

Thanks for help

--sergio

[GreatMarko]

How can I get the list of files which are not synced?

The "Devices" tab shows the amount of data which should be pushed to a node but I cannot find a way of pinning down the missing files. This is useful in case of stuck syncs.

Sync doesn't currently alert the user if certain files fail to sync, however, you can find this information in the debug log, or a simpler method is just to compare timestamps of the files in your operating system.

 

BTSync should know which files are missing, since it reports the total amount.

It does, and if you enable Debug Logging, files that are presently unable to sync will be logged there.

 

BTW It would be important to notify a sync problem via the menubar icon (e.g. OSX).

Agreed - as long as this is customizable! For example, if you've got a file open/locked/in use in another application (because you're actually working on it), you wouldn't want a balloon notification from Sync every few seconds telling you the file can't Sync!

[GlenBaars]

sync.log on linux systems gives you this info.

[vogon]

sync.log on linux systems gives you this info.

 

Unfortunately it doesn't seem so, I have two kind of messages including filenames:

 

SyncFileEntry: Failed to write attributes for file <path> - 38

ReadFile error: <path>.!sync:753664:32768:32768:3

 

for several files which are actually present in all clients, regardless the error message. These messages are limited to the first sync attempt, then the log doesn't contain any reference to any one among the trouble files.

 

I checked the file sizes to understand whether there is a correlation between the error and the missing amount. The "ReadFile" amounts to 10KB, while the "SyncFileEntry" files to around 4GB. There's no relation with the reported missing 60MB. This amount is consistent among different OSX clients that sync without problems (they both reports that the rpi misses that amount).

 

Bottom line, the sync.log doesn't help to pinpoint the troubled files and among 13GB of images is like looking for the proverbial needle in a haystack

 

Any suggestion?

 

BTW I'm looking at the standard log, not the debug one. 

[stanha]

Sync doesn't currently alert the user if certain files fail to sync, however, you can find this information in the debug log, or a simpler method is just to compare timestamps of the files in your operating system.

It does, and if you enable Debug Logging, files that are presently unable to sync will be logged there.

Question: what exactly is the string I need to be looking for in the log file to see which files do not sync and why?

Q2: What do I do to make sure ALL the files do sync if some of them do not?

[GreatMarko]

Question: what exactly is the string I need to be looking for in the log file to see which files do not sync and why?

Well look for strings like "error", "failed", "bad", or "skip"

 

Q2: What do I do to make sure ALL the files do sync if some of them do not?

Sync will re-attempt to Sync files that have previously failed at recurring intervals until the files have fully synced. Remember, Sync can't sync files whilst they are open/locked/in use by other applications.

If you're sure no files are open/locked/in user by other applications, then the next step is to exit Sync on all your devices, locate and delete any .!Sync, .SyncPart, or .SyncTemp files on your devices, then restart Sync.

If this still doesn't resolve your issue, make a note of the secret for the folder in questions, then remove it and re-add it back to Sync using the same secret (this will force a complete re-index of the folder)

[stanha]

Well look for strings like "error", "failed", "bad", or "skip"

This is a waste of time.

[RomanZ]

Hi Sergio,

 

There are numerous reasons why files might be not synced. The most common reasons are:

- wrong permissions issue

- file being locked by some other app

- file name case was changed while renaming file (known bug).

 

The most precisely it can be identified via debug log. Could you please collect debug log on both your peers and send me for analysis? Thanks.

[stanha]

Hi Sergio,

There are numerous reasons why files might be not synced. The most common reasons are:

- wrong permissions issue

Ditto.

[stanha]

Well, it seems the problem of some files not syncing is resolved in our case and those files that did not sync before on r/o nodes on Linux, now do sync, and the master node (r/w) shows that the r/o node is fully synced.

See the following post:

http://forum.bittorrent.com/topic/27781-some-files-do-not-sync/?p=79032#post_id_79032

What remains to see if ALL the files do sync when updated on a r/w node.

[RomanZ]

stanha,

 

Glad to hear that issue has been resolved. Could it happen that some apps / users changed files on R/O node so BTSync considered them "changed" and stopped syncing? Even simple "touch"-ing would be enough for BTSync to consider them changed.

[stanha]

stanha,

Glad to hear that issue has been resolved. Could it happen that some apps / users changed files on R/O node so BTSync considered them "changed" and stopped syncing? Even simple "touch"-ing would be enough for BTSync to consider them changed.

Yep.

[RomanZ]

stanha,

 

I apologize for imprecise information. Touching file on a peer won't force BTSync to think it is changed. Actually, if BTSync realises that the file actually did not change (file hash is the same), it will bring MTime back to it's original value.

 

So, to make a file actually non-propagating on a read-only node you need to change content or delete (move out of the folder). When content is changed or file is removed, the BTSync "forgets" about the file unless "Restore file if changed" setting is set in folder properties. You can copy it back then, BTSync will ignore.

 

You can also use .SyncIgnore file to make sure some particular files / directories / file types are ignored and not synced.

 

-------------------

Back to the issue of file not synced. Removing sync folder removes DB, completely, including all data about files that was changed and marked to be non-propagating. The fact that allowing file replacement if they changed on RO peer resolves your issue makes me think that  files were marked in DB as changed. I don't know why, this might be revealed by full debug logs. Please share logs for analysis if you have them.

 

-------------------

We'll consider your chat and descriptive logs proposals for future releases.

[stanha]

stanha,

I apologize for imprecise information. Touching file on a peer won't force BTSync to think it is changed. Actually, if BTSync realises that the file actually did not change (file hash is the same), it will bring MTime back to it's original value.

No point in long posts.

[RomanZ]

Do you mean "Restore modified files to original version" on Linux r/o node

Folder Preferences -> Advanced tab?

 

 

Yes.

Actually, once we are at it, could you please tell me the exact operation(s) that is performed when you click on that check box? What does btsync do exactly? Does it go through every file in the database and reset its "non-updatable" and "non-propagatable" flags?

 

 

 

BTSync goes over all the files in DB, removes the “Don’t propagate” flag and requests the torrent file (list of hashes for file pieces) from other peers. File pieces that are changes are re-downloaded.

 

I guess the only "right way" of doing it would be for you to write a shell script to bulk move the files and folders that you do not want to propagate, and, once you restored the original state of all the files in the database, you execute another script - "restore the non-updatable and non-propagatable state", which is a bit funky.

 

 

 

Currently BTSync has no UI which allows change “Don’t propagate” flag on file granularity level.

First of all, its exact semantic meaning is not clear to me. Does it mean "do not update this file from the master (r/w) node? Or does it mean "do not propagate this file to other r/o nodes? Or both?

 

 

 

Both. BTSync checks ignore list before requesting files from remote peers and never tells other peers that it has such file if it is in ignore list. Please see more in http://forum.bittorrent.com/topic/17782-bittorrent-sync-faq-unofficial/, specifically section “How do I use "*" and "?" wildcards in .SyncIgnore files?”

So, what happens next, I assume, is that btsync starts indexing the whole folder from fresh slate. Meanwhile, after folder deletion, you have added some files or folders in this folder's hierarchy.

 

So, when the local end of r/o node is fully reindexed, it then have to query the master (r/w) node to see if its own view of the database reconciles with the master's view, and, if those new files do not exist on the master, then they are marked as "non-updatable" and "non-propagatable" on this r/o node.

 

And, if master has MORE files then the r/o node has locally, then those files are propagated to this r/o node and its view on the hierarchy in the database is reconciled with the master's view with the exception of files that were added before this folder was added back to btsync.

 

Is this the correct assumption?

 

 

 

Not precisely. Files that are not on RW peer won’t get into database. Once they appear on RW peers, RO node will add them to database and IF they are different from RW peer - will mark them as “Changed, do not propagate".

Well, not sure how useful those logs would be to you. Because I do not have the logs when the problem started. I only have logs just before it was fixed, which means that the "original sin" is probably not in the logs.

 

 

 

These files won’t reveal us the root cause. 

[stanha]

I think BTSync team is more authoritive source of user documentation.