mr.canada

I have a very large folder that i'd like to sync. It's about 2TB in size, with an extremely complicated set of folders within it (very long paths, tons of small files, lots of VERY big files). Basically, a torture test. I'd like to sync it between a server in a data centre with a 1gb fiber connection, an office with a 500mbps fiber feed, another office with a 1gb fiber feed, and about 5 end users with residential cable modem feeds, some with 1gb down/50mbps up, some with much slower 50mbps/10mbps. My question is this - what settings, if any, should I adjust in Power User preferences, to deal with this much data, and to take advantage of these fast internet lines? All machines are Mac with min 8gb ram. most have 16gb or 32 gb ram. I have changed one setting so far - folder_rescan_interval. It's increased to once a week to avoid constantly rescanning. All other settings are currently at default.

access is via SMB and AFP. If anyone here has a NAS with a particularly large shared folder (multi TB...) with Mac clients I'd really like to hear about your NAS and how it's working for you.

If it were my money, sure, i'd cobble together a nas out of a pc and freenas. For clients, i'll defer to a proven brand with warranty and hardware/software support.

Thank you Helen. I don't expect you to go shopping for me But just to be perfectly clear, I understand that you don't have a particular NAS in mind, but is it fair to say that you feel a NAS with lots of RAM (16 gig) is just as capable as a Mac with 16 gigs ram, if it's PRIMARY PURPOSE is to sync a very large share? Are you aware of any problems with the current firmware running a modern Synology NAS that creates problems with Sync? There are lots of threads in this forum talking about problems with Synology DSM, but they're all 2-3 years old, so I don't know if they are still relevant or not. In the next few weeks, I have to choose between a Mac vs. a NAS as my company fileserver, and compatibility with Sync is my deciding factor. Specifically, Sync being able to accurately of observe new/changed files without relying on scheduled folder scans is my #1 concern. Thanks!

I have the luxury of being able to choose whatever NAS I want. I'm looking for input please. Here are my needs: -Large archive of files to sync (almost 3TB today, growing maybe .5TB per year). -very deep folder hierarchy - not my fault, but too late to change it (evolved over years) -very long file names -Lots of small files, but also lots of very large (1gb) files. -up to 70gb of new data produced each day (typically much less than that, average maybe 10-20GB) I am looking for a NAS to serve two functions - be a traditional fileserver for LAN clients, and be a "parent" for a sync'ed folder in Sync. Sync clients will be two other NAS servers, plus about 10 Mac workstations. No PC clients. I need speed of sync'ing and reliability of instantly observing file changes and initiating a sync. I do NOT want to rely on scheduled file system scans, as I've already seen that this takes upwards of 30 minutes to complete on a modern Mac. How long would it take on a NAS? Don't know, but probably longer given the lower-powered CPU. I'm leaning towards a Synology 1815+ with 16gb ram. Input GREATLY appreciated. Thanks.

I'd like to know about the scalability of Sync. I understand how the Bittorrent protocol benefits from more nodes - the more, the better. Is this also true of Sync? Specifically, is i have 10 nodes participating in a syncronized folder, and one node adds a large file, what exactly happens? -is it uploaded 9 times, once to each of the other nodes? -is it uploaded once to one node, who uploads it once to another node, and so on 9 times? -is it split into pieces and uploaded a total of one time, but with roughly 10% going to each node, who, in turn, share with each other until everyone has 100% of the file? If Sync uses the 3rd method, then I expect it will scale very well, as more nodes will be a benefit.

Thanks. But I do use SMB - it's a file server which shares files via SMB to users on the lan. I believe it's SMB 3.0 these days, not 2.0. It can also be AFP. I guess I should try setting it to 0 to prevent folder_rescan from happening in the first place, and see what happens. Can you think of any other times where a file may make it into a folder without Sync knowing about it, other than with SMB 2.0?

I've found some more detailed explanations here... https://help.getsync.com/hc/en-us/articles/205458185-Setting-how-often-Sync-should-check-for-file-changes- This article points out that OS filesystem notifications are not always perfect... ------- Different operating systems have limitations for folder-monitoring applications. For example, if a file path is located very deep in a Windows file structure, it will not send change notifications for these deeper levels. To cover such cases, Sync periodically re-scans all folders for changes. ------- Can you please tell me how reliable Mac OSX 10.11.6 is in this case? I have a LOT of files, and VERY deep folder paths. I would love to be able to increase the rescan interval to a very long time - perhaps 1 week. But I also need Sync to keep up with new files in near realtime.

"If you don't use network shared folders in the Sync you can open Sync preferences => Advanced => Power user preferences and increase folder_rescan_interval from 600 (which is 10 minutes) to 18000 (which is 5 hours) or even more." Can you please clarify what the pros/cons are of increasing the folder_rescan_interval is. I have a large Mac fileserver with about 3TB of data. It is shared for a dozen or so staff members using standard Apple File Sharing. I would like to use Sync to maintain local copies of this data in several locations, where it will also be shared for local staff members. In other words, i would like to run Sync on several Mac file servers - not on end-user workstations. I have increased folder_rescan_interval to 1 day. when it was set to 10 minutes, it ran constantly, never actually completing (or, perhaps, completing then starting again right away). Now that it's set to 1 day, it no longer runs constantly. but why one day? why not one month, one year, or never? what are the consequences? Does this mean that Sync will not be aware of new data until the timer for folder_rescan expires and it learns of new data that needs to be synced?

This line from the release notes... Sync replaces new file by older one after first rescan sometimes Can you elaborate on this further please? what does "first rescan" mean? I have selected a large folder on a shared Mac fileserver as the source of a sync. I'm syncing it to another machine, just for testing at this point. Some users who interact with files on the Mac fileserver (I mean, they mount the volume and add/change files - they are not running Sync at this point), have reported some odd behaviour. several people have reported several file that they have recently modified have mysteriously reverted to a prior revision. I have no way of knowing how many files have reverted - my sense is maybe 10% of files recently modified are reverting to their prior state, and 90% are saving just fine. Just a guess really, based on user feedback. To be clear, i'm running 2.3.8 on a Mac. I am not running 2.4. Am i describing the bug that 2.4 fixes?

Thank you Helen - this was helpful. Revealing the Size column in the UI allows me to see the difference between what is already sync'ed, and what needs to be sync'ed. It does not reveal the progress of the indexing process, but i suppose that's a chicken-egg dilemma - no way for Sync to know how much data is has not yet indexed until it has indexed it. I increased the index interval to 1 day. It no longer indexes endlessly. But i don't really understand what this process does.... now that indexing only happens once a day, how does Sync know what files to transmit during the day? does it rely on filesystem change notification from the OS to be informed each time a file is added or modified? If so, why index every day - why not just do it once and be done with it? indexing 2.3TB of data takes a long time - maybe an hour or so. why repeat this process daily?

I'm attempting to setup a sync between two mac servers. The folder i'm trying to sync is very large - 2.3 TB (yes, TB, not GB). It contains 215,000 items. The two macs are running current OS (today it's 10.11.6), and current Mac Server. They each have full 1gbps fiber internet. Honestly, is this just too much for Sync to deal with? It's been "indexing" for 3 days. If i allow the machines to communicate, then the sync does start to work... i've left it for 24 hours and several gigs has synced, but it's also Indexing at the same time. I imagine these processes compete for the CPU, so i've paused syncing in hopes of Indexing being able to complete. I wish there was a progress bar, % compete, ETA for the Indexing process as there is for the Sync process.....

Hi - can anyone explain how Sync deals with multiple copies of the same file? I mean, say I have a folder with 10 subfolders, and for some stupid reason, the same large file exists in all 10 subfolders. So i have 10 copies of the same file. Then I tell Sync to sync the parent folder. Does it transmit the same file 10 times? Or is it smart enough to send it once, then fill in the blanks at the destination by duplicating that one file 9 more times? Here's my scenerio - a fileserver with 6TB of historical data. It's 1000's of large Photoshop images. Folders are organized by project, year, month. Even though it totals 6TB, it's really only about 0.5TB of unique data, because most images are duplicated throughout the folder structure. As a matter of workflow, staff regularly create more and more copies of these files by placing copies of them into this month's working folder. So on one hand, every day, lots of new data is created (like, many many gigabytes - more than is reasonable to sync with a modest internet connection). But on the other hand, very little new data is created each day, because most new files are just copies of preexisting ones. Is Sync for me?