marmite_sandwich Posted September 24, 2014 Report Share Posted September 24, 2014 (edited) I am syncing 2 folders on W7 PCs via the internet. They already contain the same 100GB of media files (c15000 files and 1500 folders, nested). I am just trying to get BTSync to recognise that they are in sync, before I actually change something on the master. It seems to choke on this volume. For several hours now the master has been showing "out of sync" for this, its only, peer. The receiving and sending data is missing as is the progress. On the receiving side, this is also "out of sync", with a progress of 1% after about 3 hours. On both sides the data rates show up intermittently and something seems to be happening for about 30 seconds, then nothing for c10 minutes. If I check the Windows task manager on the sending side, I can see lots of heavy processing (60%) for minutes at a time, split between BTSync and the OS. In between, there are short bursts of network traffic. When I tested this principle on a folder containing c1.5GB, 18 folders, 200 files, the process was complete in 10 minutes, with steady network traffic at the max rate that the sending side could upload. It seems like the extra file/folder numbers and/or the extra data size is causing the sending side to choke on the volume - it is only trying to compare what is there with the receiving side, which is the same, so why does it seem to grind to a halt? Any ideas welcome.v1.4.83Sending W7 Pro SP1 32 bit, upload max 1Mbps.Receiving W7 Pro SP1 64 bit.Use relay, use tracker, search LAN set.Overwrite changed files set. Edit: Sending side says "no peers online" most of the time, receiving side says "<source> offline" similarly. Why aren't they talking to each other all the time. They do intermittently, so why not continuously? Edited September 24, 2014 by marmite_sandwich Quote Link to comment Share on other sites More sharing options...
marmite_sandwich Posted September 26, 2014 Author Report Share Posted September 26, 2014 So I did some more investigation. Deleted both folders and reinstated the sending ("master") side. For 45 minutes, there was a lot of processing activity, while the indexing took place. After this stopped, I created the receiving ("slave") folder and monitored the windows task manager network activity and processing at both ends (different PCs on different LANs at different locations). Both ends contained the same 100GB of data in 15000 files and it took the 2 communicating instances of BTSync 8 hours to figure out that they were the same and did not require any data to be transmitted. This seems like a long time, but I guess you only need it to happen once, unless you are in the habit of un- and re-installing BTSync. I noticed that the network activity during this comparison phase came in bursts, at a very slow and variable speed, typically 10KB/s, with a pattern of about 2 minutes on and 5 minutes off. During the 5 minutes off, the slave did processing. I had created one sub-folder with new data on the master, and when it came to these files, the network activity changed totally, becoming a constant 100KB/s until the data was all transferred, then it resorted to the stuttering file comparison mode. Since completing this process, I have made changes at the master and monitored the changes at the slave. These were immediate and at the maximum upload speed of the master, i.e. 100KB/s. So no complaints there. Just need to figure out how to install BTSync as a Windows service, which I saw a post about, then it should be a permanent feature for me. Quote Link to comment Share on other sites More sharing options...
marmite_sandwich Posted October 31, 2014 Author Report Share Posted October 31, 2014 Still getting this problem with 1.4.99, on much smaller folders. http://forum.bittorrent.com/topic/31986-out-of-sync/ 2 folders of c7GB, with 4000 files/folders, will take 6 days to get in sync even though the data in them is the same (well, almost - there is one file of 40B to update). Quote Link to comment Share on other sites More sharing options...
RomanZ Posted November 3, 2014 Report Share Posted November 3, 2014 @marmite_sandwichWhat is your hardware?100Gb in 17K of files and folders is not that much for sync. Taking around 1Kb per file/folder it will take around 17Mb of memory space, while 100Gb can be hashed within around 30-60 minutes on modern SATA drives and high-power CPUs. Quote Link to comment Share on other sites More sharing options...
marmite_sandwich Posted November 3, 2014 Author Report Share Posted November 3, 2014 R/W node: Computer: Computer Type ACPI x64-based PC Operating System Microsoft Windows 7 Professional OS Service Pack Service Pack 1 Internet Explorer 9.11.9600.17358 DirectX DirectX 11.0 Date / Time 2014-11-03 / 14:23 Motherboard: CPU Type 4x , 2000 MHz (12 x 167) Motherboard Name Unknown Motherboard Chipset Unknown System Memory 3978 MB BIOS Type AMI (03/28/2014) Storage: IDE Controller Standard Dual Channel PCI IDE Controller Disk Drive WDC WD5000AAKX-22ERMA0 ATA Device (465 GB, IDE) Optical Drive CD-ROM Drive SMART Hard Disks Status OK Partitions: C: (NTFS) 458.8 GB (220.6 GB free) Total Size 458.8 GB (220.6 GB free) R/O node: Computer: Computer Type ACPI x86-based PC (Mobile) Operating System Microsoft Windows 7 Starter OS Service Pack Service Pack 1 Internet Explorer 9.10.9200.17116 DirectX DirectX 11.0 Computer Name ASUS3 (1015PX) User Name Guy Logon Domain Asus3 Date / Time 2014-11-03 / 14:35 Motherboard: CPU Type DualCore Intel Atom N570, 1666 MHz (10 x 167) Motherboard Name Asus Eee PC 1015PE Motherboard Chipset Intel Tiger Point NM10, Intel Pineview-M System Memory 2038 MB (DDR3-1333 DDR3 SDRAM) DIMM1: Micron 8JSF25664HZ-1G4D1 2 GB DDR3-1333 DDR3 SDRAM (8-8-8-22 @ 609 MHz) (7-7-7-20 @ 533 MHz) (6-6-6-17 @ 457 MHz) (5-5-5-14 @ 380 MHz) BIOS Type AMI (05/06/11) Storage: IDE Controller Standard AHCI 1.0 Serial ATA Controller Disk Drive Hitachi HTS543232A7A384 ATA Device (320 GB, 5400 RPM, SATA-II) Disk Drive Multiple Card Reader USB Device (3 GB, USB) SMART Hard Disks Status OK Partitions: C: (NTFS) 100.0 GB (23.3 GB free) D: (NTFS) 183.1 GB (17.7 GB free) Total Size 283.1 GB (41.0 GB free) I can't imagine it is the hardware. I have checked the windows task manager at both ends, and nothing much is happening. It only happens when trying to sync two folders which have the same contents already. After that it is very fast at updating new changes.Marmite Quote Link to comment Share on other sites More sharing options...
RomanZ Posted November 3, 2014 Report Share Posted November 3, 2014 Marmite, While your RW node is pretty powerful, there RO one is based on Atom CPU - which is low-end power economy CPU. I suspect that it might be a bottleneck as the CPU itself will have to read and hash 100Gb of data. Can you please do a simple test:1. Disconnect the folder on your Asus netbook.2. Add it as a new RW folder (just hit the huge blue button)3. Observe - how long will it take to index all the files. I suspect that it is going to take around 8 hours. Quote Link to comment Share on other sites More sharing options...
marmite_sandwich Posted November 3, 2014 Author Report Share Posted November 3, 2014 Sorry, that combination of hardware was the one which was trying to sync 7GB, over a LAN. When I was trying to sync 100GB of media files, both devices were similar to the Acer RW device above, but it was over the internet. In both cases, there were long pauses at both nodes, where the Windows Task Manager showed processing at 10% or less (and no network activity). If there was a CPU bottleneck, one of the CPUs would have been flat out at 100%. That's why I described them as going to sleep. I'll have a go at trying your experiment shortly.Cheers,Marmite Quote Link to comment Share on other sites More sharing options...
marmite_sandwich Posted November 3, 2014 Author Report Share Posted November 3, 2014 OK, results of experiment:-2 devices as above.100GB of mixed media files, already synced using SyncBack.A folder was created on each device with no connections.Quadcore Acer desktop took 31 minutes to index.Dualcore Asus netbook took 41 minutes to index.Then I disconnected the Asus folder and created a folder on it from a read-only key on the Acer.Set the read-only node to overwrite changed files.After 3 hours, the Acer (master) is showing "out of sync" and the Asus (read-only) is showing 34% complete with a size of 34GB and a forecast to completion of 3 months. Windows Task Manager on the Acer is showing CPU activity of about 10%, with much of that due to other processes. On the Asus, CPU activity is between 10 and 50%, with only TeamViewer running additionally. Network activity is spiky. Hope that helps. The syncing seems to add a whole load of activity which is not related to the indexing process. With a fast connection it would be quicker to sync to an empty folder than to one which already has the data in it. Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.