himselfv Posted July 29, 2014 Report Share Posted July 29, 2014 Some of my shares have tens of thousands files, and I've noticed they're loading very slow. When starting Sync, even before it starts "(Indexing...)" the folder, it is stuck for a whole minute simply adding it to "My Sync" folder list. I fired up SysInternals Process Monitor, and it seems there's nothing but this set of requests again and again: 12678438 11:36:24,3138382 BTSync.exe 3476 RegOpenKey HKLM\System\CurrentControlSet\Control\Cryptography\Providers REPARSE Desired Access: Read12678439 11:36:24,3138513 BTSync.exe 3476 RegOpenKey HKLM\System\CurrentControlSet\Control\Cryptography\Providers SUCCESS Desired Access: Read12678440 11:36:24,3138673 BTSync.exe 3476 RegCloseKey HKLM\System\CurrentControlSet\Control\Cryptography\Providers SUCCESS 12678441 11:36:24,3138791 BTSync.exe 3476 RegOpenKey HKLM\System\CurrentControlSet\Control\Cryptography\Configuration REPARSE Desired Access: Read12678442 11:36:24,3138915 BTSync.exe 3476 RegOpenKey HKLM\System\CurrentControlSet\Control\Cryptography\Configuration SUCCESS Desired Access: Read12678443 11:36:24,3139053 BTSync.exe 3476 RegCloseKey HKLM\System\CurrentControlSet\Control\Cryptography\Configuration SUCCESS These 6 requests are repeated a godless number of times, probably the same tens of thousands times, once for each file. There's no reading apart from rare block reads from btsync ".db" files, no network activity, justs these requests. This takes minutes. I'm guessing here, so pardon me if I'm wrong, but perhaps it's something that can be easily optimized? Maybe you're creating some sort of crypto primitive which looks into these keys on initialization, and in fact you could use the same primitive for all files? Because the actual "(Indexing...)" step runs much, much faster than this step. Quote Link to comment Share on other sites More sharing options...
RomanZ Posted July 29, 2014 Report Share Posted July 29, 2014 @himselfv I suspect that the things you see in the ProcMon is rather the symptom than an issue. We do not read \ write the keys you mention directly, but we use Microsoft CryptoAPI which does. Couple of things to clarify and to try:- Are you using Encrypted folders in your setup?- If you are syncing in LAN - you may try to disable encryption in LAN.- How many files and folders in general do you sync?- Does your hardware match the amount of files and folders (compare your RAM size to the amount of RAM consumed by Sync). Quote Link to comment Share on other sites More sharing options...
himselfv Posted July 30, 2014 Author Report Share Posted July 30, 2014 > Are you using Encrypted folders in your setup? Yes, and the folder in question is of encryptable kind (decrypted on this PC though). > If you are syncing in LAN - you may try to disable encryption in LAN. No, syncing over the internet. > How many files and folders in general do you sync? 5 sync nodes: 2300 files, 50 files, 8000 files (the problematic one), 2000 files, 1300 files. Only the problematic one is encryption-enabled, others appear instanteniously and start "(Indexing...)". > Does your hardware match the amount of files and folders (compare your RAM size to the amount of RAM consumed by Sync). Probably. 8 Gb RAM, virtual memory allocated by BTSync when fully loaded: 226 Mb. > We do not read \ write the keys you mention directly, but we use Microsoft CryptoAPI which does. Yeah, I thought it's something like that, but maybe you're instantiating some kind of CryptoAPI object for each file where one would have been enough? I'm just guessing, it just looks like this slowness happens when BTSync is reading *.db files (probably SQLite databases?), and the largest of those on my PC is 94 Mb. So disk-wise it could be read in an instant. I've run the API Monitor, and it seems BTSync does this for every file (e.g. 8000 times): BCryptOpenAlgorithmProvider(... 'AES')BCryptGetProperty('ObjectLength')BCryptSetProperty('ChainingMode')BCryptImportKey(...'KeyDataBlob')BCryptEncrypt(...)BCryptEncrypt(...)BCryptCloseAlgortihmProvider()BCryptDestroyKey() MSDN says: > Because of the number and type of operations that are required to find, load, and initialize an algorithm provider, the BCryptOpenAlgorithmProvider function is a relatively time intensive function. Because of this, we recommend that you cache any algorithm provider handles that you will use more than once, rather than opening and closing the algorithm providers over and over. Maybe this is the reason? Quote Link to comment Share on other sites More sharing options...
RomanZ Posted August 6, 2014 Report Share Posted August 6, 2014 @himselfv Thanks for the comprehensive analysis. We'll check if this is true indeed and if it can be optimized. Quote Link to comment Share on other sites More sharing options...
RomanZ Posted August 11, 2014 Report Share Posted August 11, 2014 @himselfv Indeed, CryptoAPI calls could be optimized for encrypted peers. We'll implement it for upcoming 1.4 release. Thanks a ton for very technical issue analysis, we really appreciate that! Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.