10 FreeNAS sync questions

[billgreenwood]

I’m running BTSync 2.3.1 on two Freenas servers both in different locations, connected over internet. 

 

I have 10 questions for the gurus here.

 

1) I’ve read conflicting information on interweb. Please can I get a definitive answer on what the green and blue arrows represent. 

 

2) The web GUI amongst other options, has columns showing Status, Peers Online, Receiving, Sending, Progress.   Peers online is always showing how many peers online. Status Receiving, Sending and Progress columns are mostly blank. Can these be forced to show current information?

 

3) Can I turn off keyboard shortcuts in the web GUI?

 

4) Size.  I have a folder synced on two servers. (The machines have been syncing over a very fast connection for a few weeks … for a short time they were on the same LAN.)

On Freenas 1, the folder size (using OSX, ⌘-I) is 85gb, the BTSync web GUI for this system is showing only a size of 1.3gb.  

On Freenas 2, the folder size (using OSX, ⌘-I) is 107gb, the BTSync web GUI for this system is showing only a size of 44.69gb.  

Is there a technical reason for these web GUI showing these differences? Or at least not reporting the correct sizes.

 

5)  How does BTSync work when indexing? Can I set indexing at a folder level so I can have priority folders? If so how is this achieved please?

 

6) Meanwhile, In the advanced preferences there is the feature to set the rescan time of the folder (folder_rescan_interval set at 600 - i guess seconds). 

What is the optimun time for this setting given what I have to sync? 

Here’s what is has to sync. 

I have 15 sub-folders, each of these sub-folders contains 30 further subfolders, each 3rd-level sub-folder containing between 80 - 150 images each. Total folder size 95gb.

What I am thinking is; would a slower folder rescan be better to allow files to be transferred between the machines over the internet, before the rescan kicks in. Does this matter?

 

7) I understand that the FreeNAS/FreeBSD does support kqueue that could allow BTSync to support immediate file changes on this platform. Will BTSync add this feature in on FreeBSD?

 

8) Would it be worth be adding a mac / linux with BTSync in-between these machines to help with the sync. Would this help?

 

9) Both machines are on 100mb links. Are there any settings that would help a with a faster transfer?

 

10) Does adding an entry in the  folder_defaults.known_hosts lock BTSync into only that host?

 

Thanks in advance for any of the answers.

 

b.

 

[GreatMarko]

1 hour ago, billgreenwood said:

1) I’ve read conflicting information on interweb. Please can I get a definitive answer on what the green and blue arrows represent. 

Are you referring to up/down arrows in the Sending/Receiving columns? 

1 hour ago, billgreenwood said:

2) The web GUI amongst other options, has columns showing Status, Peers Online, Receiving, Sending, Progress.   Peers online is always showing how many peers online. Status Receiving, Sending and Progress columns are mostly blank. Can these be forced to show current information?

No, but there are a number of threads in the Feature Request forum you may wish to contribute to - for example: 1 | 2

1 hour ago, billgreenwood said:

3) Can I turn off keyboard shortcuts in the web GUI?

No, again, possibly a request for the Feature Request forum

1 hour ago, billgreenwood said:

4) Size.  I have a folder synced on two servers. (The machines have been syncing over a very fast connection for a few weeks … for a short time they were on the same LAN.)

Is there a technical reason for these web GUI showing these differences? Or at least not reporting the correct sizes.

There are a number of reasons why folder sizes in Sync may differ to those in your underlying OS. Commonly, it can be because you're "excluding" certain files/file types/folders from syncing (via .sync/IgnoreList, etc). Also, note that sizes shown in Sync won't take into account the size of the hidden .sync/Archive folder

1 hour ago, billgreenwood said:

5)  How does BTSync work when indexing? Can I set indexing at a folder level so I can have priority folders? If so how is this achieved please?

You can't currently "prioritize" syncing on a per-folder basis, but again, the Feature Request forum is your friend - try this thread.

1 hour ago, billgreenwood said:

6) Meanwhile, In the advanced preferences there is the feature to set the rescan time of the folder (folder_rescan_interval set at 600 - i guess seconds). 

What is the optimun time for this setting given what I have to sync?

"optimum" in terms of what? Sync speed? CPU usage?

Folder "rescanning" will have negligible impact on actual syncing speed (assuming your hardware is sufficient)

1 hour ago, billgreenwood said:

7) I understand that the FreeNAS/FreeBSD does support kqueue that could allow BTSync to support immediate file changes on this platform. Will BTSync add this feature in on FreeBSD?

Once again, the Feature Request forum is your friend!

1 hour ago, billgreenwood said:

8) Would it be worth be adding a mac / linux with BTSync in-between these machines to help with the sync. Would this help?

Generally speaking, the more devices you have in your Sync "mesh" the faster files will be available on your other devices, as each device will be able to retrieve files/parts of files for multiple peers.

1 hour ago, billgreenwood said:

9) Both machines are on 100mb links. Are there any settings that would help a with a faster transfer?

If they're all on the same local network, you could try disabling settings such as "lan_encrypt_data" and "disk_low_priority". These might give you a slight increase in transfer speed.

See also How can I improve my data transfer sync speed? in the Sync Help Center

1 hour ago, billgreenwood said:

10) Does adding an entry in the  folder_defaults.known_hosts lock BTSync into only that host?

No - "known hosts" provides a "hint" to Sync - they are not "exclusive" and do not "lock" sync to specified IPs

[billgreenwood]

Hi @GreatMarko

 

Thanks for the reply

1) yes I am referring to up/down arrows in the Sending/Receiving columns?  Can you confirm what the green & blue colours mean please?

2) Yes, I'll add in a feature request. It's not great having these useful columns and no info on them. 

3, 5, 7) will add in feature requests

4) Size. The examples I gave were gigabytes apart in size, and i've no ignore list.  Maybe this needs a thread of its own.  Though I have noticed files buried in subfolders that are same name on bother servers but different timestamps which are days apart.

6) rescanning. Yes the hardware is good. i was wondering if having a 10 min scan time between intervals is to short for the program to index the files and search for updated ones on other servers before the next scan is due. I was wondering if to short a scan would keep the machine in a constant state of indexing. Just a thought. 

8) on one folder I have 5 devices, though the master server is a freenas box, so if this is not telling others that the file has changes then the whole thing collapses. I'll do some more tests. 

9) I've disabled the low disk (it's the default for the Freenas setting, but not the OSX) and alos the an encrypt. I was wondering about the buffer size, which again defaults are different in FreeNAS 128 (MB i think) and OSX 10 MB

10) superb, what I wanted to hear .... well, read ... Thanks

[GreatMarko]

7 hours ago, billgreenwood said:

1) yes I am referring to up/down arrows in the Sending/Receiving columns?  Can you confirm what the green & blue colours mean please?

The green "Up" arrows denote data is being uploaded from the current device.

The blue "Down" arrows denote data is being downloaded to the current device.

[Helen]

4) discrepancy between FreNAS1 and FreeNAS2 may come from not all files synced? That might need submitting logs from both peers to support team. And a few details about your setups - folders, types of files, perhaps a few screenshots to better illustrate it. Or, you can troubleshoot it with these most widespread reasons. 

And discrepancy between what size system reports and Sync in UI may come from Archive, or not all files being indexed (usually caused by their unavailability) or system reporting the space allocated for files while Sync - actual size of file.

6) depends on the number of files you have in the folder - the indexing itself may vary from a few seconds to half and hour. So if you have a lot of files, you can increase rescan.

9) I suppose on Mac you have some 2.2.x version of Sync, where buff-size in indeed 10 Mb? 

[billgreenwood]

Hi @Helen

 

Thanks for the reply. BTSync is a great Idea.

I have BTSync, running on 15 folders. All the folders are standard manual folders. A hangover from the 1.3 days. And some of my less techy colleagues understand these. Could this be an issue?

The two main servers are FreeNAS machines - syncing over the internet - where I have BTSync installed in a couple of Jails. I'm trying to sync approx 6tb of data. Though this will increase in time.

Other machines are macs dotted around the London, each syncing different folders from this main pool. These Macs seem to pick up files created on other Macs. I think my issue here with syncing has to be with the FREENAS boxes.

In my case the reporting of data size in the web GUI IS incorrect to the data actually on the system. I am assuming that the data size in the web GUI represents the supposed size of the folder? I understand about the sizing issues you've mentioned, but I don't think they come into play here. The disparity is to great.

I could start-over with fresh installs of BTSync and fresh folders on each FreeNAS system (making sure there are no .sync folders) and resync everything to see how it goes. I may do that over the weekend, then give the system time to resync and (hopefully not) speak to support if there are issues. Both folders will not be empty and will have a data inside. I think BTSync should aim "merge" the data sets so both folders on each system become the same.

I'll sort out the sending of logs and hopefully you can give me some pointers.

 

10) Yes on mac I have 10mb Buff size. But FreeNAS is the default 128. Should I change this?

 

A compare function that reports files between two machines would be a great addition. The reporting of what is synced / not synced so the administrator can keep a check would be really helpful

I understand that the FreeNAS/FreeBSD does support kqueue that could allow BTSync to support immediate file changes on this platform. Will BTSync add this feature in the future?

Many thanks

 

b.

 

Edited February 19, 2016 by billgreenwood

[Helen]

billgreenwood,

note that peers create a mesh, so if two macs connect to a same folder on NAS, then two mac will also communicate with each other, and sync files between each other .

I don't think that 1.3 hangover (: folders can be the root of the problem but it would be better if you could re-add them. So please try to re-setup everything, perhaps this instruction on syncing pre-populated folder will be of help for you. Yes, Sync will merge the folders.

And finally, you can trace the non-synced files in sync queue, available in Peer list. 

Well, I've not heard any plans about kqueue discussed around, so it won't be there in the near future at least. 

[billgreenwood]

Hi @Helen

 

I'm not sure if Bittorrent Sync on FreeNAS / FreeBSD is working.

I've many files just not syncing. I've set known peers with fixed IPs and ports. I've Turned off any low disk priority, lowered the folder rescan times, changed all permissions on files/folders so any user can Read / Write.

 

It just does not work!

 

 

Any ideas?

 

 

Thanks

 

b.

 

[Helen]

@billgreenwood

To give you a more specific reply on your particular case and know why exactly files do not sync in your setup, we need debug logs from your peers, here's how to submit logs . Without logs we can only make assumption and this might be any of the reasons mentioned in this article

[billgreenwood]

Unfortunately Helen I dd not have logs set up. But I've updated the BTSync Freenas plugins from 2.3.1 to 2.3.3 this afternoon and things are starting to look ok. Your team must have ironed out some wrinkles. 

 

I'll have to give it a few days and see how this does.

 

But I will be switching on the logs on all my devices in case there are future issues.

 

b.

[billgreenwood]

UPDATE

@Helen

 

Have sent logs for all 4 devices to the support team.