Search the Community

Showing results for tags 'indexing'.



More search options

  • Search By Tags

    Type tags separated by commas.
  • Search By Author

Content Type


Forums

  • Resilio Sync
    • Sync General Discussion
    • Sync Troubleshooting
    • Sync for NAS (Network Attached Storage)
    • Sync Stories
    • Developers
    • Feature Requests

Find results in...

Find results that contain...


Date Created

  • Start

    End


Last Updated

  • Start

    End


Filter by number of...

Joined

  • Start

    End


Group


AIM


MSN


Website URL


ICQ


Yahoo


Jabber


Skype


Location


Interests

Found 55 results

  1. I'm attempting to setup a sync between two mac servers. The folder i'm trying to sync is very large - 2.3 TB (yes, TB, not GB). It contains 215,000 items. The two macs are running current OS (today it's 10.11.6), and current Mac Server. They each have full 1gbps fiber internet. Honestly, is this just too much for Sync to deal with? It's been "indexing" for 3 days. If i allow the machines to communicate, then the sync does start to work... i've left it for 24 hours and several gigs has synced, but it's also Indexing at the same time. I imagine these processes compete for the CPU, so i've paused syncing in hopes of Indexing being able to complete. I wish there was a progress bar, % compete, ETA for the Indexing process as there is for the Sync process.....
  2. folder_rescan_interval can be edited in advanced preferences as a global setting. But, it is useful to allow individual folder specific rescan settings too. Rescan indexing can take a long time if you have lots of files. Example, I have 70K+ files in a lightroom folder, it takes 4 minutes to rescan with a fast SSD. The default rescan interval is 10minutes (600s) which means my laptop spends 40% of the time rescanning eating away battery. I don't need my lightroom folder to rescan every 10minutes, but other smaller document folders I would like to.
  3. Hey Sync Team! I have been using Resilio Sync on my Raspberry Pi for quite a long time now. Everything worked fine until I decided to synchronise my photo storage (roughly 60 GB). Resilio Sync is indexing since 3 days (and nights) straight and there is no end in sight. It is also using 70% of the CPU available on my PI. My two other folders still sync without any problems (they are only 5 GB each), only the big one seems to be stuck at the indexing progress (even though everything was transferred just fine). I hope you can help me! Sincerly Alexander.
  4. I've got what's admittedly a pretty large share, ~38GB, syncing between several Macs and a NAS. 2.3 has been great on the NAS and it works fine on the Macs except the thing is indexing several times an hour and chewing huge amounts of CPU and battery when it does. CPU will peg a core at 100% for 15 minutes at a time and the Energy Impact on my laptops puts BitTorrent Sync as consuming about twice as much power as everything else running on those machines combined. This was a complete non-issue with previous v2.x versions. Is there anything I can provide to help debug? [Here is a possible solution build - RomanZ]
  5. adoteq

    Indexing stuck

    Dear ... I am having issues with resilio sync on Ubuntu 17.10 64 bit. I installed sync, added a folder with the majority of the files already on my drive (pre-seed). I use a 8TB exFAT external HDD, but the indexing is already taking 3 days for only 2.96TB, and overnight nothing happened. What is the problem, and how do I fix this? Syncing happens, but very slow. Kind regards, Joannes Wyckmans
  6. Hi I was syncing some large files from a host down to my external USB drive and It has stuck at indexing, It currently says: 98% - indexing It has been doing this for hours now I tried pausing and resuming did nothing then I tried disconnecting and removing the sync then deleting the sync folder on the HDD and then I set up the sync again to the same destination and stuck at ( 98% - indexing ) again. Files sizes vary but total 30GB. I looks like it is stuck on the last 2 files that have not started to transfer. Thanks for the reply.
  7. I need to synchronize some Virtual Machine (Vmware) , that are splitted in sever files. Basically synchronization have to happen when machine is turned off, so i need to stop synching whenever i open a Virtual machine. Is there a way to stop indexing on folder and restarting only when i need it? If a put the folder in pause i see indexing happening in background (lot of i/o on disk) Apparently unique solutio is to turn off the sync process and restart when i shutdown the machine Any idea? Thank you
  8. Dear Sync community, I've posted the issue previously in https://help.getsync.com/hc/en-us/requests/18696 but im afraid, the following problem still persists: Resilio Sync 2.4.4 (732) is shown as occupying 100% CPU in 'Activity Monitor'. A folder containing ~114.000 of admittedly small files (45 GB) is constantly shown as indexing, even when setting folder_rescan_interval to e.g. 3600. I noticed the constant indexing issue also on a Synology NAS, however setting folder_rescan_interval to 3600 seems to have helped to alleviate the issue a bit here. On the Mac, running the command sudo fs_usage -w -f filesys > fs_usage_resilio.log will produce a ~750 MB log file within 30 seconds when resilio is running (as opposed to 1.6 MB within 30 seconds with resilio turned off). within the logfile the following sequences are repeated seemingly for every synced file: 19:52:03.015920 lstat64 <path-to-some-synced-file> 0.000005 Resilio Sync.1138808 19:52:03.015930 listxattr <path-to-some-synced-file> 0.000009 Resilio Sync.1138808 19:52:03.015935 getxattr <path-to-some-synced-file> 0.000005 Resilio Sync.1138808 19:52:03.016021 open F=53 (R_____) <path-to-some-synced-file> 0.000006 Resilio Sync.1138808 19:52:03.016022 fcntl F=53 <GETPATH> 0.000001 Resilio Sync.1138808 19:52:03.016023 close F=53 0.000002 Resilio Sync.1138808 [...] 19:52:16.106473 lstat64 <path-to-some-synced-folder> 0.000002 Resilio Sync.1138808 19:52:16.106490 write F=6 B=0xa3 0.000003 Resilio Sync.1138593 19:52:16.106491 lseek F=6 O=0x02bfd39d <SEEK_SET> 0.000001 Resilio Sync.1138593 19:52:16.106536 open F=127 (R_____) <path-to-some-synced-folder> 0.000003 Resilio Sync.1138808 19:52:16.106537 fcntl F=127 <GETPATH> 0.000001 Resilio Sync.1138808 19:52:16.106538 close F=127 0.000001 Resilio Sync.1138808 With this behaviour of frequent re-indexing and constant file access it is not feasible to run resilio on that folder permanently without draining the battery. Has anybody else encountered a similar issue on their system? Thank you for your help.
  9. I'm running on Windows 10. I recently did a fresh install, and wanted to reconnect some sync folders. To start with, I just pointed at the original folder, and copied the link, and then removed the folder on the freenas server, and re-added it. I've done this before and had it working. I receive hundreds of messages about 'invalid modification time', and also messages about failing on downloading files due to 'initial recheck'. Not only this, but the system just scans non-stop. It does not stop thrashing the disk. As soon as one scan is finished, it starts again, completely consuming the read on this disk. So I stopped and removed it, and decided to delete the .sync folder, hoping that it would fix the problem, but it seems not. I have decided for the moment not to share a link to the folder, just to see if it can index correctly, but it never finishes and still has hundreds of 'invalid modification time' messages. Any idea what I can try to stop this?
  10. SmallwoodDR82

    Syncing 2 large NAS shares

    First off thank you in advance for assistance and love the product and if I can get this to work as I hope, I will be purchasing Pro for sure. (I'm on 30 day trial right now) Quick Facts - I have 2 NAS (Xeon CPU) unRAID servers and both are running Docker with Sync as a container. Both running Sync 2.4.4. - The idea here is server #1 is my "master" all read/write happens on this server and server #2 is strictly read only from server #1. - I have 6 folders created on server #1 shared as read only to server #2. The size of these folders range from 11GB to 13TB. - Because both servers were sort of mirrors of each other before (more manually) most files are on both servers with same folder structure. - I'm trying to introduce sync now so it will be automated. So after adding the 6 folders to server #1 all the folders indexed correctly (took about 30 hours) after that I added each folder one by one to server #2. Upon accepting the folder it asks are you sure because the destination folder already has files, I click yes and the "syncing" starts. Now because most items are identical, on the first pass it didn't do much transferring most of it was just rechecks and what not. Rinse and repeat for the next 4 folders. Now here is where my issue starts. My final folder is 13TBs and about 48,000 files. The indexing of this folder on server #1 went just fine. However when I share the folder as read only with server #2 and the indexing/recheck starts it goes for about 10 to 12 hours and then just stops. Usually around 7.5TBs or so but never finishes. I looked in the logs and didn't really see anything. I've removed the folder and re-connected it twice now and it's the same result. It's running again right now for the 3rd time and so far so good (about 2TBs in). So I don't have any logs to share just yet and I'll make sure I turn debug on during this process. I'm just curious if anyone has run into this and is there any setting or feature I'm missing? Hope to post more info tonight. Thanks, Daniel!
  11. Hello, Is it possible to tell resilio to reindex the files.. I am using a shared folder and sometimes when I add files from a specific computer, resilio does not see that file change. In Dropbox, if I pause dropbox and then resume it , the dropbox reindex all the files.. Is there an option like that in Resilio? Thanks, Oren.
  12. I think a key missing feature from Sync is stopping errors from propagating by interacting with the user and presenting her with options on what one can do. As an example, I recently filed an issue #49958 where the indexing was out of control. It turned out that due to some naming conflict (probably due to the composed/decomposed issue) *.Conflict folders were being created. By the time we found out why indexing never finished 1.4 million such conflict folders had been created for one particular weirdly named folder. When stuff like that happens Sync should notify the user that something really fishy is going on (ok, I am a programmer so I know that is way too wide description to implement, but you get the gist) and some possible solutions like: Folder X is causing conflicts because of naming issues. These are your options for fixing the problem: - Notify user X that the created file Y has a filename causing problems and ask her to rename the folder - Do X - Do Y We will stop sync for this subfolder/item until the problem has been fixed. If there had been some kind of "cross-network error diagnostics" built into the protocol maybe errors that arise on another peer than the one that created the filename can be propagated back to the origin to fix the error in the source. AFAIK only the "destination" peer gets any info through the GUI that a filename contains illegal characters, and from my experience they have no idea what to do about it (if they even understand it's a problem).
  13. Hello, I have a raspberry pi running sync and synchronizing folders from a home NAS (Apple Airport Extreme). It's taking a long time for sync to re-index and find file changes on the NAS as they are modified/added/deleted by other machines accessing it and as far as I know the only way sync finds these changes is on re-scanning/re-indexing the NAS. I have two point's I'd like bring here: 1) Is there a way to speed this process ? For instance prioritizing more important folders over others that I know are not often changed or are less important/critical? 2) Is there a way to tell sync to "look into that folder now" as I know things changed and want them to be shared asap? These were somehow discussed years back without a clear answer so I am bringing the issue back, after several new versions as it seems they still exists or the solution or work-around (if any) was not shared. Despite the points above, sync is a great product and refining it considering these points would make it even better. Suggestions, hints are welcome. Thanks in advance. Carlos Benjamin.
  14. I just wrote a long post, but the forum software made it disappear after a permissions issue. Anyways, I'm just I'm going to write something short. I set up btsync to sync my home folder to another computer that should have an already pretty much synced version of the home folder on it (I was previously using syncthing). It's been indexing for hours. I expected cpu usage to be 70% and up during this indexing stage, but it's currently running at around 2%. I can see that it needs to upload 14,819 files, but it doesn't upload. I installed Sierra beta a few days before I installed btsync, so I'm not sure if that has anything to do with it. Any help would be appreciated.
  15. I have some virtual disk images that needs to be synchronized between different system. At a given time, the VM is active on only one computer. So there is no problem with corruption or similar. However, when the VM is running, btsync is always indexing the file over and over again. How is it possible to stop btsync indexing the disk image as long as the vitual machine is running? Best regards Daniel
  16. hi, i have version 2.3.6 (378) of btsync running and i have the problem, that indexing somehow needs too much resources of my pc. directly after start of btsync task manager shows relatively high cpu consumtion of btsync (about 30%) - see "processes.png". however, the pc works fine at this moment. the strange thing is, that after about a minute cpu usage of btsync drops to about 5% (see "processes2.png"), but now the pc is becoming nearly unusable. all reactions are very slow and e.g. videos in browser are hardly working anymore. if i stop btsync, then after some moments everything is fine again. "cpu.png" shows the chart of cpu usage. the first part, where the line is relatively flat, is the part where the pc is hardly working anymore (50% usage, although in processes btsync is only ~5% -> but somehow anyway its btsyncs fault). the drop of the usage in the chart occurs, if i terminate btsync. "settings.png" shows my advanced settings of btsync - i already tried some things like changing "folder_rescan_interval" to a higher value or "disk_low_priority" to true. my system: win 10 64bit intel core i5 m 520 8GB RAM i would be very thankful if you could help me with this very strange problem! if you need some logfiles or more infos, please let me know. thanks!
  17. I use BTSYNC Pro on a windows PC, but the shared 70GB directory is on a NAS. Because of that I can't use window's OS notification system for indexing file changes. The work around is to set scanning intervals using rescan interval in Sync settings -> Advanced -> Power user. I've set it to 36000 which is fine for me. This has been working fine on version 2.2.5 Recently I've updated to the latest 2.3.3 and my NAS was giving heat warnings. Finally, found out BYSYNC was indexing very frequently and for prolong periods of time causing my HDD to spinup. Temporarily I've downgraded back to 2.2.5 and indexing is back to normal, but I still want to keep updated. Is there a workaround fix or am I doing something wrong?
  18. Hello! I am trying to use BTsync with an external HDD, as my main PC SSD is pretty small. I'm trying to sync ~100GB of home movie files in a couple of different formats, but what I find is that as soon as btsync starts indexing the folder on my HDD the HDD becomes extremely slow, but not technically frozen. It takes ~4 minutes to load a folder on this HDD while indexing is running. I tried to get around this last night by leaving my PC on overnight indexing, but when I came in the morning it hadn't actually indexed over night? -- Might be user error, consider an anecdote. Is there a power use setting I can adjust to prevent indexing from making the drive unusable? I've already turned down the folder rescan time to every few hours. Let me know! JW
  19. Hi, Having had a ton of issues that I've been able to surmount (namely permissions issues that I could only fix by copying all my files to a new share on the same source NAS), I'm now unable to get this working, and here I came for help. The purpose of my bitsync install is to keep a working server in my company synced with a backup server which I'll keep home after the initial sync (~3.3Tb) has taken place locally. My target NAS is using bittorent sync 1.4, as it is the latest available version for qoriq based Synology NAS, but I have been able to install everything, setup a shared folder and start the sync process, so to my untrained eye this does not seem to be the source of my problems. Source NAS is running the latest version available through synocommunity. Target NAS only has read permissions (thus my source/target names for the NAS'). Problem 1: My source folder is ~3.31 Tb; Sync only "sees" 2.94 Tb. It shows "indexing" for days, but it does not seem to be able to index the latest ~370Gb or so. (picture 1). Permissions' issues with the original share were what made me use the nuclear option, create a share and define all the permissions from scratch, sos I'm pretty sure this isn't a permissions issue as outlined in the FAQ and in many forum posts. Problem 2: Target NAS synced folder stopped at around 260Gb, and it stopped increasing. Although both NAS are currently connected to the same gigabit switch, transfer rates went from ~10Mb/s with peaks of ~30Mb/s to 1-2kb/s. I have since removed the shared folder in the target NAS, in the hopes that this would decrease the load on the server and improve the speed with which it was being indexed. Hasn't happened. Any ideas on why it stopped syncing with more than 90% to go? Problem 3: Bittorent Sync's web interface stopped generating the QR codes for syncing with mobile apps. (picture 2) This was working before, and I have been able to successfully link the shared folder with my iPhone. Two related questions: 1 - Bittorrent Sync generates a ton of logs and journals, any hints on what I should try to search for in order to try to understand what the problems are? Also, why would I have a B07A0AB68F26F6E0CCEEF0CFBBC7FBBFEFAB3455.5711241.db file at /usr/local/bittorentsync/var that is ~3.6Gb and growing? It seems quite large, what is supposedly stored in this database? 2 - I've had a (quite bizarre, IMHO) behaviour where the files in a test source folder were deleted when I removed the shared folder in the source NAS through Sync's web interface. I believe this is not intended behaviour, but this is in a way hold me back trying the "nuke" option of uninstall/reinstall and starting from scratch. I realise these may not be the cleverest questions, but I've pretty much exhausted all the how-to's and forum material that I could find and I'm stuck. Any help towards a despairing noob would be dearly appreciated. Thanks in advance, Vasco
  20. Hi, i've installed BTSync on my Synology and on two Macs to sync around 100k files in about 25 GB (lots of JPG and RAW files and a Lightroom database). So far it works… but… The Mac-Client – on is a MacBook Pro – is doing an 'indexing' every 10 Minutes. Every 'indexing' takes a few minutes where BTSync is consuming up to 100% CPU. I've searched a bit and read some longer threads about this indexing-feature but i think i didn't really get it: - Indexing is building a catalogue of all files that i like to sync to detect new/deleted files? - But then, changes made on one mac are transferred to the NAS in nearly realtime… not every 10 Minutes… And if i wake up my MacBook, the files are downloaded from the NAS nearly instant … not after 10 Minutes So again - what is the purpose of creating an index every 10 Minutes? Are there save ways to optimize the clients behavior? I like to have my Lightroom-Files in sync on both Macs and i'm using a NAS because both Macs are not running at the same time. Both Macs are creating/deleting files (JPG-Previews etc.). The NAS is running 24/7 so the 'truth' should always be on that NAS… In this case: Is it save to say, that i don't need to build an index on the macs every 10 Minutes but for example only once a day? I mean: If i'm working on the local iMac, changes are synced to the NAS in real-time. If i'm working on the MacBook it's the same when i'm at home. Only if i'm working on the MacBook while i'm traveling i need to be sure, that the MacBook will sync all new/changed files to the NAS when i'm back home - While BTSync is running, it should know those changes in realtime, right? So - Why do i need such index in my use case? Is it save to turn it off or set it a way longer interval (once every 10 hours or so) on the mac-clients? Could i even change the interval on the NAS (or turn it off completely) to let the HDDs sleep again from time to time? Thanks a lot for your help to clarify this topic
  21. I recently got a new phone (Moto X Pure running 6.0) and am having trouble with the android sync app. I have it set up to sync files to an SD Card, but whenever I connect the folder it just starts and stops "indexing" and never actually transfers anything. If I turn on the selective sync and grab a file or two manually they download fine so I know it's not an issue with the SD card/permissions.
  22. I've got tons of issues with the latest version (2.3.3), the biggest one being that folders are suddenly disappearing from BTS all the time after a restart. When re-adding them, indexing will get stuck after a few GB's. I've noticed that if I quit all other clients (so it can't sync the new folder) indexing will finish properly. I've already downgraded to 2.2.7 because of this and the issue of BTS constantly indexing, using up 100% CPU draining my battery (there is already another topic about this).
  23. I have a folder containing a ton of files (probably over a million), most of which will not change, save maybe because of corruption on the disk. From what I understand, indexing causes all the contents of all the files to be read. This takes forever. Is there any way to, say, reindex files only based on their last modified date? Edit: Just realized that wouldn't work, given BTSync is based on the BT protocol.
  24. Hello ! I've been testing btsync for a few days and I was able to sync folders containing a few files/folders succesfully. Btsync seems to be the solution I was waiting for. Now I'm running into a problem : I'm trying to sync a folder containing about 35000 files (about 20Gb) between two synology NAS boxes so I added a folder share on the first one and generated a secret. The btsync web interface tells me this : 21.7 GB in 36915 files (Indexing...) Also, the CPU usage is abnormally high (above 70%) and I'm talking about constant usage, not peak usage. My folder contained originally 500k+ files and I thought that it was too many, that's why I reduced the number of files to about 35k. Indexing seems to be stuck, could someone help me troubleshoot this ? I've looked into the logs but I could not find anything relevant. Thanks in advance, David
  25. larsen161

    Multithread Indexing

    It seems that indexing currently only happens on a single share at a time. When adding multiple new shares this would drastically improve speed as now it takes hours/days to sync multiple folders with 100k's of files and TBs of data. Currently, btsync is only using ~15% cpu on an m1.xlarge aws instance.