W7 Btsync Goes To Sleep Trying To Sync Large Collection?

[marmite_sandwich]

I am syncing 2 folders on W7 PCs via the internet. They already contain the same 100GB of media files (c15000 files and 1500 folders, nested). I am just trying to get BTSync to recognise that they are in sync, before I actually change something on the master. It seems to choke on this volume. For several hours now the master has been showing "out of sync" for this, its only, peer. The receiving and sending data is missing as is the progress. On the receiving side, this is also "out of sync", with a progress of 1% after about 3 hours.

 

On both sides the data rates show up intermittently and something seems to be happening for about 30 seconds, then nothing for c10 minutes.

 

If I check the Windows task manager on the sending side, I can see lots of heavy processing (60%) for minutes at a time, split between BTSync and the OS. In between, there are short bursts of network traffic.

 

When I tested this principle on a folder containing c1.5GB, 18 folders, 200 files, the process was complete in 10 minutes, with steady network traffic at the max rate that the sending side could upload.

 

It seems like the extra file/folder numbers and/or the extra data size is causing the sending side to choke on the volume - it is only trying to compare what is there with the receiving side, which is the same, so why does it seem to grind to a halt?

 

Any ideas welcome.

v1.4.83

Sending W7 Pro SP1 32 bit, upload max 1Mbps.

Receiving W7 Pro SP1 64 bit.

Use relay, use tracker, search LAN set.

Overwrite changed files set.

 

Edit: Sending side says "no peers online" most of the time, receiving side says "<source> offline" similarly. Why aren't they talking to each other all the time. They do intermittently, so why not continuously?

Edited September 24, 2014 by marmite_sandwich

[marmite_sandwich]

So I did some more investigation. Deleted both folders and reinstated the sending ("master") side. For 45 minutes, there was a lot of processing activity, while the indexing took place. After this stopped, I created the receiving ("slave") folder and monitored the windows task manager network activity and processing at both ends (different PCs on different LANs at different locations). Both ends contained the same 100GB of data in 15000 files and it took the 2 communicating instances of BTSync 8 hours to figure out that they were the same and did not require any data to be transmitted. This seems like a long time, but I guess you only need it to happen once, unless you are in the habit of un- and re-installing BTSync.

 

I noticed that the network activity during this comparison phase came in bursts, at a very slow and variable speed, typically 10KB/s, with a pattern of about 2 minutes on and 5 minutes off. During the 5 minutes off, the slave did processing. I had created one sub-folder with new data on the master, and when it came to these files, the network activity changed totally, becoming a constant 100KB/s until the data was all transferred, then it resorted to the stuttering file comparison mode.

 

Since completing this process, I have made changes at the master and monitored the changes at the slave. These were immediate and at the maximum upload speed of the master, i.e. 100KB/s. So no complaints there.

 

Just need to figure out how to install BTSync as a Windows service, which I saw a post about, then it should be a permanent feature for me.

[marmite_sandwich]

Still getting this problem with 1.4.99, on much smaller folders.

 

http://forum.bittorrent.com/topic/31986-out-of-sync/

 

2 folders of c7GB, with 4000 files/folders, will take 6 days to get in sync even though the data in them is the same (well, almost - there is one file of 40B to update).

[RomanZ]

@marmite_sandwich

What is your hardware?

100Gb in 17K of files and folders is not that much for sync. Taking around 1Kb per file/folder it will take around 17Mb of memory space, while 100Gb can be hashed within around 30-60 minutes on modern SATA drives and  high-power CPUs. 

[marmite_sandwich]

R/W node:

 

    Computer:

      Computer Type                                     ACPI x64-based PC

      Operating System                                  Microsoft Windows 7 Professional

      OS Service Pack                                   Service Pack 1

      Internet Explorer                                 9.11.9600.17358

      DirectX                                           DirectX 11.0

      Date / Time                                       2014-11-03 / 14:23

 

    Motherboard:

      CPU Type                                          4x , 2000 MHz (12 x 167)

      Motherboard Name                                  Unknown

      Motherboard Chipset                               Unknown

      System Memory                                     3978 MB

      BIOS Type                                         AMI (03/28/2014)

 

 

    Storage:

      IDE Controller                                    Standard Dual Channel PCI IDE Controller

      Disk Drive                                        WDC WD5000AAKX-22ERMA0 ATA Device  (465 GB, IDE)

      Optical Drive                                     CD-ROM Drive

      SMART Hard Disks Status                           OK

 

    Partitions:

      C: (NTFS)                                         458.8 GB (220.6 GB free)

      Total Size                                        458.8 GB (220.6 GB free)

 

R/O node:

   Computer:

      Computer Type                                     ACPI x86-based PC  (Mobile)

      Operating System                                  Microsoft Windows 7 Starter

      OS Service Pack                                   Service Pack 1

      Internet Explorer                                 9.10.9200.17116

      DirectX                                           DirectX 11.0

      Computer Name                                     ASUS3 (1015PX)

      User Name                                         Guy

      Logon Domain                                      Asus3

      Date / Time                                       2014-11-03 / 14:35

 

    Motherboard:

      CPU Type                                          DualCore Intel Atom N570, 1666 MHz (10 x 167)

      Motherboard Name                                  Asus Eee PC 1015PE

      Motherboard Chipset                               Intel Tiger Point NM10, Intel Pineview-M

      System Memory                                     2038 MB  (DDR3-1333 DDR3 SDRAM)

      DIMM1: Micron 8JSF25664HZ-1G4D1                   2 GB DDR3-1333 DDR3 SDRAM  (8-8-8-22 @ 609 MHz)  (7-7-7-20 @ 533 MHz)  (6-6-6-17 @ 457 MHz)  (5-5-5-14 @ 380 MHz)

      BIOS Type                                         AMI (05/06/11)

 

    Storage:

      IDE Controller                                    Standard AHCI 1.0 Serial ATA Controller

      Disk Drive                                        Hitachi HTS543232A7A384 ATA Device  (320 GB, 5400 RPM, SATA-II)

      Disk Drive                                        Multiple Card  Reader USB Device  (3 GB, USB)

      SMART Hard Disks Status                           OK

 

    Partitions:

      C: (NTFS)                                         100.0 GB (23.3 GB free)

      D: (NTFS)                                         183.1 GB (17.7 GB free)

      Total Size                                        283.1 GB (41.0 GB free)

 

I can't imagine it is the hardware. I have checked the windows task manager at both ends, and nothing much is happening. It only happens when trying to sync two folders which have the same contents already. After that it is very fast at updating new changes.

Marmite

[RomanZ]

Marmite,

 

While your RW node is pretty powerful, there RO one is based on Atom CPU - which is low-end power economy CPU. I suspect that it might be a bottleneck as the CPU itself will have to read and hash 100Gb of data.

 

Can you please do a simple test:

1. Disconnect the folder on your Asus netbook.

2. Add it as a new RW folder (just hit the huge blue button)

3. Observe - how long will it take to index all the files.

 

I suspect that it is going to take around 8 hours.

[marmite_sandwich]

Sorry, that combination of hardware was the one which was trying to sync 7GB, over a LAN. When I was trying to sync 100GB of media files, both devices were similar to the Acer RW device above, but it was over the internet.

 

In both cases, there were long pauses at both nodes, where the Windows Task Manager showed processing at 10% or less (and no network activity). If there was a CPU bottleneck, one of the CPUs would have been flat out at 100%. That's why I described them as going to sleep.

 

I'll have a go at  trying your experiment shortly.

Cheers,

Marmite

[marmite_sandwich]

OK, results of experiment:-

2 devices as above.

100GB of mixed media files, already synced using SyncBack.

A folder was created on each device with no connections.

Quadcore Acer desktop took 31 minutes to index.

Dualcore Asus netbook took 41 minutes to index.

Then I disconnected the Asus folder and created a folder on it from a read-only key on the Acer.

Set the read-only node to overwrite changed files.

After 3 hours, the Acer (master) is showing "out of sync" and the Asus (read-only) is showing 34% complete with a size of 34GB and a forecast to completion of 3 months. Windows Task Manager on the Acer is showing CPU activity of about 10%, with much of that due to other processes. On the Asus, CPU activity is between 10 and 50%, with only TeamViewer running additionally. Network activity is spiky.

 

Hope that helps. The syncing seems to add a whole load of activity which is not related to the indexing process. With a fast connection it would be quicker to sync to an empty folder than to one which already has the data in it.