Sign in to follow this  

Confused About 'size' Of Files Synced.

Recommended Posts

Running a large sync of an archive of files to make them available to some friends online, partly because I wanted to see how BTSync actually works for a large fileset.


Getting differing account of the size of the files synced.
My Originating machine (Ubuntu) tells me that there are 97,251 items, totalling 863.6 MB in the folder.


BTSyncs interface, despite later removing and readding the folder to kick off syncs again, is quite determined that I am syncing 820.86 MB in 89291 files. My Amazon cloud server I'm syncing this too says it has received the same number of files.


Removing the originating folder from BTSync and creating a receiving RO folder, BTsync says *IT* has received 820.86 MB in 89291 files - but when I check *it* it from the OS interface, it say it has 97,236 items, totalling 863.3 MB.

(Edit: the 15 missing files are 14 'Thumbs.db' files and a 'desktop.ini' file, all covered by .SyncIgnore.
So, I have at this point every reason to believe that every file is, in reality, being synced, except for the discrepency in the interface. So That's good.)


My initial theory was that BTSync was counting, not files, but file hashes - which would indicate that duplicate files were synced internally as 'same file different names' (and I can certainly see that as a feature not a bug), and the use of fslint in my original folder does in fact indicate the folder has "34,781,696 bytes wasted in 26993 files (in 16990 groups)" . . . which would support this theory if it actually indicated a difference of 7,060 files, but as *I* count it that would imply there should be a 10,003 file discrepency.

So at this point I really don't know *what* is going on, except the sync count returned in the interface definitely does *not*  match the number of files synced, and the *way* it doesn't match doesn't seem consistent, despite all three interfaces agreeing inside BTsyc regarding how many files are synced.

Edit: Intuitively I think this seems like it *has* to be internally counting by filehashes, despite not matching fslint's count of duplicates. If so it may be worthwhile it to adjust the interface to reflect this - something of the order "89,291 Unique hashes synced 97,236 named files."

Share this post

Link to post
Share on other sites

I think you've answered your own question in your edits - Sync's count of files/file sizes won't take into account files it's not syncing (including .SyncIgnore and .SyncID files and anything in your hidden .SyncArchive folder). Your OS on the other hand will take into account these.


There isn't much need really for Sync to display info on files/folders it's not syncing (although, there's a case to be made for showing the "size" of the .SyncArchive in the UI, so you can keep an eye on how big it's getting - but that's more a request for the Feature Request forum)

Share this post

Link to post
Share on other sites

The problem I'm seeing (If I've got this reverse engineered in my head right) isn't files that don't sync, it's that if you have two (or more) files that are exact duplicates (same files and given the discrepency between BTSync and FSLint, I *THINK* it has to have the same timestamp or something else that FSlint doesn't consider important.), it *syncs* both files, but only counts them once.


The immediate problem is that this mean BTSync actually syncs 97,236 files, but the interface claims you have only synced 89,291 - it *LOOKS* like about 8,000 files got lost somewhere.

A more esoteric problem would be the potential for hash collisions - two files that have the same 4MB sequences of hash signatures but aren't actually identical - does BTsync notice or is one file overwitten in favor of the other for recipients? But although I know Hash collisions *can* be engineered, and can think of ways to abuse that, I don't know enough about it in practice to do more than note it as something someone smarter than me should be aware of.

Share this post

Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Sign in to follow this