Shaav
-
Posts
21 -
Joined
-
Last visited
Posts posted by Shaav
-
-
@adam1v
Not as far as I am aware... but I'm not necessarily in the the loop either. Just didn't want to leave you completely hanging since there have been no other replies...
-
+1
Would also be nice if there were two levels --- a default global for all shares and then a share-level IgnoreList that would sync across all devices using the share.
-
@GreatMarko
Thansk for the link; will do that.
-
@RomanZ
It is entirely possible that I mistakenly put it there in the past; I just don't remember. Also, because the documentation isn't entirely clear on that point (something that could maybe be addressed?), I possibly put it in both locations "just to be safe". However, increasing the number of devices/shares certainly makes that less feasible quickly.
It would be nice if there were a global IgnoreList and a local IgnoreList BTW... But I realize that's a very linuxy kind of thing to do.
It would also be nice if the IgnoreList become a propogated property of the share so that if you edited one, that change would propogate rather than having to try and edit it on each installation...
But thank you for the reply, as always your expertise is much appreciated.
-
@Moe
Frankly I'm having enough of a miserable time experimenting around trying to understand the behaviour of this software --- I thought *maybe* it was a simple enough question to get straight, conclusive answer instead of trying to reverse engineer it from observation.
-
Just want to quickly clarify something...
The documentation says the IgnoreList goes in the hidden .sync file; my interpretation is that it means the /share_folder/.sync folder rather than the /run_path/.sync folder. But I have IgnoreLists in both locations and I'm not sure if that's by design, because of upgrades where the location changed, or if it was something that I just mistakenly did at some point in the past.
What I kind of *hope* is that you can put a global IgnoreList in /run_path/.sync that applies to *all* folders and then a share-specific IgnoreList in /share_folder/.sync and that btsync combines them for the share...
What exactly is the behaviour with IgnoreLists?
-
I realize this is an issue that people bring up here a lot; I haven't really seen much by way of solutions---seems like a lot of them abandon btsync before it gets resolved. I'm hoping that I have more detail to offer about what happens than others...
Situation: 4 QNAP NAS drives syncing about 2.3TB of data; ~700K files. All 4 drives are using the latest 2 version. All the drives currently have ~99% identical content; just differences that have occurred from their use since the problems began. 2 of the drives have been functioning perfectly for about 8 months; the problem came when we tried to add 2 new drives. Left them to index for days and days without there seeming to be any change. Because of an accidental deletion, one of the original NAS's ended up dumping all 2.3TB into "Archive"; those were moved back in place, but of course, then that one needed to reindex everything too. All of them have their folder rescan set to 24hrs, which until now, I thought would be more than enough to handle the indexing of what I currently am shifting...
Not getting anywhere, I eventually stopped the original folder from syncing at all and set up a new sync folder and started shifting small amounts of data from the old to the new with the idea that if there's something corrupt about the share, then that will get fixed; if it's a problem with files or communication, that would be easier to diagnose with a smaller data set and if it's a problem with resources then it should be easier to manage and actually see tangible progress, get a handle on how long it takes to actually do stuff.
I started with 12GB --- went OK. Next I did about 30GB -- took *I think* about 4hrs to index and sync. Last night I did approximately 76GB and now more than 24hrs later, it looks like progress has halted. All four report different sizes, only one of which I think matches the size of the share (trying to take into account excluded files and rounding differences etc). Two report all 3 peers on line, 1 reports 2 peers online and 1 reports 0.
I have written some scripts to pull file lists from the drives and filter them with the excludes so that any differences are reflective of real differences and I can confirm that right now, no files are being updated.
I'm happy to provide log files --- I'm not making much sense of them --- but I'd rather not post them publicly. This is a client's system and they use a lot of identifiable personal information in filenames/paths. Is there another way I can upload them a little more privately?
There is definitely a lot of looping going on. Two have a lot of "rejecting until file info is updated". Another has a lot of "Failed to create empty suffix for file". The one that looks like it actually indexed everything appears the most normal, but still have a bunch of stuff like "Trash: requested file was not found".
Clearly, a big part of the problem is the combination of data volume and the processing capabilities of the NAS drives themselves. So one thing I'm wondering is whether it is possible to bootstrap that process by, say mounting the share on my hexacore behemoth as an nfs drive or something, and have it do the initial indexing? When I do an initial mirror of data from one to the other, could I copy the hashes too?
One thing that is profoundly frustrating about the UI, is that it is extremely difficult to determine what is actually happening. Especially when the devices are under a heavy load, it doesn't accurately reflect the actual state of the sync. Even trying to load the options, I sometimes have to wait 10s of seconds for the options to appear, particularly pre-defined hosts. The "indexing" indicator flashes on and off seemingly at random and you have no idea if it done or not done. I'll try to pause/resume the syncing and sometimes it doesn't work. Often a 'du' of the folder is wildly out of whack with what it is reporting as the size in the UI, even when it doesn't seem to be indexing (but possibly is). Sometimes, the size changes and seems to reflect indexing going on; sometimes it just sits at the same amount for minutes/hours even when it seems to be indexing (but could be stuck).
It would be really useful if, especially for these massive initial indexes, there was a mode where you could just tell the device to just index the folder and provide a progress indicator and clear indication when it is done. If there's a lot to index --- maybe pick a threshold --- I wish it would put everything else on hold until it's done that process. Or at least it was a configurable option, especially when you add a folder that already has content.
Any help would be greatly appreciated.
-
@RomanZ
I was eventually able to verify that they could talk to each other, FYI. There's just other stuff going on. Topics for other threads.
-
@RomanZ
Will look into trying that. Unfortunately, these are all QNAP NASs that don't have netcat on them. Will need to figure out getting in on there first...
-
@Shaav
Thanks for the feedback. We definitely want to improve Archive functionality in future.
Specifically here, I think it would be a lot more functional if rather than renaming the files the way you do now, is always have the most recent version keep the same file name and then rename the versions such that the highest number is always the oldest. I know that means renaming more files on a regular basis, but it would become much easier to, say, write a script to deal with problems like mine.
-
@Shaav
It would be test.txt. The best way to find the last one file there would be take its modification time.
@RomanZ
Yeah... sadly in the situation I'm in, all the files from one peer were deleted which propogated... I rsynced the files back (including all the .n archives cause... hard to separate them out) and neglected to use -a so all the modification times got updated.
So no help there. Thanks for the info though. -
@Shaav
I suggest trying to enter predefined hosts to folder properties. The IP:port pair should point to the peer behind VPN gateway, i.e. use not a gateway IP, but IP you would use to reach remote peer. See if it works.
@Romanz
That is what I mean I did when I said "EVEN when I provide explicit VPN IPs and ports for the peers". Does not work.
-
When files are changed and archived, .sync/Archive/path/ has multiple versions of the file named: file.n.ext and it appears that the highest "n" is the most recent file.
Consider this scenario where the default retention period of 30 days is in effect and the same file (test.txt) is edited on a weekly basis:
.syncArchive/path:
2015-06-12 test.4.txt
2015-06-05 test.3.txt
2015-05-29 test.2.txt
2015-05-22 test.1.txt
2015-05-15 test.txt
by the time 2015-06-19 rolls around, the 2015-05-15 test.txt will be deleted. What will the name of the new archived file? test.txt? or test.5.txt?
-
@Moe, @Shaav
Its a bit more here. Sync is sending multicast packets for other peers discovery with TTL=3. So it actually depends on a router how to behave. If router is simple and is not aware of multicast - it will just route packet and decrease TTL.
If it is aware, then multicast with TTL=3 will be routed within same organization / site. So it more depends on VPN gateway implementation.
Thanks @RomanZ --- this is kind of an unusual case and I'm using VPN because I don't have direct access to the networks in question (except 1 --- not the one I'm particularly interested in.
). So I don't have any idea what the routers or their configurations are like for 3/4 devices that are syncing.I've been experimenting quite a bit and the peers do not seem to find each other on the VPN. I.e. if I turn off relays/tracking/DHT and just leave searching the LAN, they do not find each other. But more surprising, EVEN when I provide explicit VPN IPs and ports for the peers, they won't find each other and won't sync until I turn on using the tracker. Any thoughts on that?
They definitely are accessible to each other through the VPN. I ssh into one and then can ssh into the others without any difficulties (for instance).
-
I am using BTSync with devices that connect to each other with a VPN. So they each have an IP address on the local LAN (where they are alone) and also an IP on the VPN. When I check "Search LAN" will it search both? Or just the local LAN? Obviously, my preference would be for BTSync to work over the VPN.
When it is "searching", is each client sending a broadcast and waiting for a response? Or just listening for a broadcast?
-
Facing a similar issue... what did you eventually do? Just wait until it was finished indexing?
-
Here's the situation:
I have had two QNAP NAS ("Q1" & "Q2") drives syncing happily for 7 months using 1.4; just over 2TB of data (about 650K files). This last week I wanted to add two more QNAPs to the mix ("Q3" and "Q4"). They have 2.0 installed. (For the time being, I have reasons for not wanting to upgrade Q1 and Q2 although will if necessary).
I pre-copied over all of the data from Q1 to Q3 and Q4 which are all on the same LAN. Q2 is still in the swarm but over the internet. I set up the syncing; everything seemed to be OK. However the indexing was taking *forever* --- 3 days and it only seemed to be maybe half done --- so I thought it might just be faster to remove the data from Q3 and Q4 and then let them sync from Q1 (and Q2); downloading seems to be faster than indexing.
I stopped btsync on Q3; deleted the date I had copied over; started btsync again and everything seemed to be going OK; stuff was starting to copy over; so I did the same with Q4; again everything seemed to be working OK. Q1 started behaving a little oddly; couldn't get the GUI to load and the others were showing it offline. Finally it came back; didn't think much of it.
Then I discovered that most of the files had been deleted from Q1. In retrospect, I guess because of deleting stuff from Q3/Q4; probably everything that *had* been indexed got deleted.
Sort of a no biggie, because everything is still in the .sync/Archive HOWEVER, I have a couple of questions on how to proceed.
1. Best way to restore from .sync/Archive? Is there a way for me to identify the newest version of the files and not restore all the previously deleted ones?
2. It looks like all the timestamps have been updated to the delete times. :[ Don't much care about that with Q3 and Q4, but what's going to happen with Q2? Is it going to try and reupload all 2TB to Q2 because they have new timestamps? Or will it recognize that the hashes are the same?
3. Which is the better (i.e. faster) approach on a LAN? Prepopulating the share with rsync and then letting them index or syncing all 2TB using btsync?
4. What should I delete on Q3 and Q4 to make sure this doesn't happen again?
Any advice would be greatly appreciated!
-
It's been a bit of a nightmare, but I finally got this sorted on my two QNAP TS-421
Two step process:
1. I had to ssh in and edit the startup script: /etc/init.d/btsync.sh (which just links to /share/MD0_DATA/.qpkg/BitTorrentSync/btsync.sh
and add the line:
umask 002
somewhere near the beginning. I added it just before the "case".
Note: it seems extremely likely that if you update the package you will need to redo this step!
2. Make sure that all your existing files and directories have sufficient privileges and the correct ownership. From the top level share (/share/MD0_DATA/<folder>):
chgrp -R everyone * <--- make sure all the existing files and folders belong to the group "everyone"
chmod -R 775 * <--- make sure all that the group has full permissions (this is what the umask above takes care of for new files and directories).
chmod -R g+s * <--- set the "sgid" bit which forces new files and directories to inherit the group ownership.
----
One additional thing that was weird and frustrating while trying to test stuff; while logged in as admin the sgid bit was not being inherited by directories that I manually created. (!!) So you'd create be in a directory; create a new subdirectory---it would have the correct group but NOT the sgid which it should also inherit. Then you'd create a new file in the subdirectory and it would NOT have the correct group. Fortunately, it seems that btsync still runs OK and sets it correctly.
-
1.4.91 seems to have resolved the issue for me.
-
Also having this problem on a pair of identical QNAP T-421 both running 1.4.76. Oddly enough, it only ever happens on one of them; not sure if that's just because of the direction of the syncing it's trying to do or what... Everytime it fails, btsync crashes (process dies) and it's always with this set of log entries (although the file can change):
[20141014 00:51:20.989] TorrentFile: 0x7460bea0 created
[20141014 00:51:21.041] FC[C042]: LoadTorrent: wrong number of pieces for file /Path/Document.pdf without metadata, resetting it
[20141014 00:51:21.041] [OnNotifyFileChange] "/Path/Document.pdf", source = "NULL"
[20141014 00:51:21.041] TorrentFile: 0x7460bea0 deleted
So no help there. Thanks for the info though.
). So I don't have any idea what the routers or their configurations are like for 3/4 devices that are syncing.
Insight Into Slow Indexing
in Sync Troubleshooting
Posted
I have been struggling for a couple of months now trying to get 4 QNAP NAS drives syncing about 2TB of data, all of which had been pre-loaded.
Indexing has been a nightmare and never seemed to move forward. For weeks and weeks.
Finally, I found that if I stopped btsync on all but one NAS at a time, it would successfully sync all 2TB in about 12hrs. Note, btsync had to be *stopped* completely (i.e. I would ssh in and run ./btsync.sh stop --- not just paused in the UI.
Pausing seems to stop the transfer of files (?) but not the back-and-forth communication about the files and somehow, it seems that interferes with the indexing sufficiently to grind it completely to a halt.