AquaRelliux

Huge Sync problems with 2 NAS units

Recommended Posts

Hello Sync Community

This is my first post here so bare with me if I am noob on this. OK before I go into details I think it best if I explain my setup I will draw it first:

 

FreeNAS(3.7TB media)  --->ubuntu 14.04 LTS(btsync owner) --->Internet--->Western Digital EX4(btsync receiver)

 

I wanted a backup of all my files that keeps up to date so I placed one of my NAS unite(EX4) at my friends house and I run a 250/100Mbit fiber and he uses a 100/10Mbit fiber. All devices runs the latest version of btsync and am baffeled on how bad this program seems to work. Never have I succesfully synced my whole liberary to my friends house. I have tried adding the library 4 times now and it stopped beeing funny the second time. There is around 5600 files and it takes around 10 days for it to complete the sync over the Internet. So please I really need help now because I want this to work!

Attempt 1: Stopped syncing all together after around 50% of the files synced, tried rebooting the devices and reindexing the folder nothing worked. So I reinstalled everything..

Attempt 2: After a couple of days it hanged at "indexing" on the owner machine and could not index new files. Removed the folder and added it again, seemed to work for a while but then hanged again so I reinstalled the Owner node

Attempt 3: Failed like the 1st

Attempt 4: Some how the amount of files the reviceing and owner nodes needed got "out of sync". Now the reveining node reports "Synced" but the Owner node reports "Sending 528 files". I checked the file system on the receving node and it is exactly the 528 files that it is missing..

 

I don't know what to do now here are some debug logs from receiving node:

 

[22:21:27]  SF[F715] [5149]: Received request "files"
[22:21:27]  MC[F715] [5149]: processing files message with 565 files
[22:21:28]  ACL[F715]: VerifyFileSignature: failed to verify ed25519 signature, user: 59E087553358D0695544356311912DD5AF2DE5E13209AFB4EFAB0E10672EDAC8
[22:21:28]  MC[F715] [5149]: failed to verify signature of remote file XXXXXXXXXXXXXXXXXXX, aborting
[22:21:28]  assert failed /opt/sync/SyncFolderMergeController.cpp:880
[22:21:28]  SF[F715] [5149]: State sync finished
[22:21:28]  SyncReq[0x41903b88][uTP-TUNNELL] [0000]: cancel 41903B88 - incoming merge
[22:21:28]  SyncReq[0x41903b88][uTP-TCP] [0000]: destroing 41903B88 cbcnt:9 - incoming merge
[22:21:31]  SF[F715]: UpdatePeersStat
[22:21:31]  SF[F715] [5149]: up:0 down:0
[22:21:31]  ScheduledTask: "UpdatePeersStat" invoked: timer, reason:FinishStateSync
[22:21:36]  PC[0x41900c60][0x00000000][XXX:XXX:XXX:XXX:50160:TUNNELL] incoming connection fro

 

And there is some from the Owner node:

 

[22:25:37]  MC[F715] [A800]: Local file Series/XXXXXXXXXXX/S4 is newer than remote t:1453770242/1453770201 ot:59351/59182 o:1036DF5B0C3BD36D72249A575BC5732CCA975149/1036DF5B0C3BD36D72249A575BC5732CCA975149
[22:25:37]  MC[F715] [A800]: will send file /Series/The Walking Dead/S4
[22:25:38]  TCPReq[F715] [0000]: eof F8001250 - outgoing merge
[22:25:38]  SF[F715] [A800]: State sync finished
[22:25:38]  TCPReq[F715] [0000]: cancel F8001250 - outgoing merge
[22:25:38]  TCPReq[F715] [0000]: destroing F8001250 cbcnt:10 - outgoing merge
[22:25:40]  LicenseController:
[22:25:40]  HasValidLicense: 0

 

And it looks like before the Owner node has completely stopped reindexing the filesystem when I add new files. Why does this even happen? All the files are located on one folder on the NAS and the Owner node has FULL access it can even delete files..
 

 

Edited by GreatMarko
Edited for formatting

Share this post


Link to post
Share on other sites

 AquaRelliux,

Folder cannot be merged because of file "XXXXXXXXXXXXXXXXXXX", as you see in log "failed to verify signature of remote file XXXXXXXXXXXXXXXXXXX, aborting". The file signature on receiving peer was wrong. 

Remove it from the folder on the uploading peer and add it back. Before adding, check its filename, it shall be in utf-8 encoding (try renaming the file), and timestamp (make sure it's valid).

Share this post


Link to post
Share on other sites

I cannot do this on 533 folders... It will take until the end of time, This keeps happening and when I add new folders it does not sync them either.

I only sync on folder that contains around 4500 folders of media, it has succesfully synced all but 533 files. If I remove them like I tried serveral times it again fails after around 8-9 days of transfering.

Share this post


Link to post
Share on other sites
2 hours ago, AquaRelliux said:

I cannot do this on 533 folders... 

You cannot do what? Remove or rename "XXXXXXXXXXXXXXXXXXX" file (or subfolder) mentioned in the log? The signature of this item is corrupted, the file (subfolder) cannot be synced. Usually all syncing stuck because of one file/subfolder mentioned in the line that you cited. 

Share this post


Link to post
Share on other sites

It says the same for all 533 files it not just this one this is just a part of the log

 

Also the Linux version that you have in the Ubuntu server reposistory is version 2.0.X something? why have you not updated this in several months?

 

Ok I just tried to rename the entire folder, it reindexet it and still not sending any files...

And the receving server says this so why is there any error?

 

synced_zps0tmjusaa.png

Edited by AquaRelliux

Share this post


Link to post
Share on other sites

That's the reason why we usually ask to send full log to support. It's pretty hard, seeing only a few lines, guess about the scope. 

Enable "Overwrite any changed files" in this folder's preferences. 

Share this post


Link to post
Share on other sites

I have a similar issue for a file which does NOT exist on the drive (not even hidden). I am using 3 nodes (2 Windows, 1 linux) with v2.3.3 (296). The problem affects the linux client which thinks the folder is synced and doesn't transfer any other file within that folder. The Windows hosts sync all the files without problems.

Log from Linux client:

[20160224 16:16:43.899] MC[DF37] [FDB8]: failed to verify signature of remote file IMGPXYZ.JPG, aborting
[20160224 16:16:43.900] assert failed /opt/sync/SyncFolderMergeController.cpp:880

By the way this path does not exist too: "/opt/sync/".

Log from Windows client:

[2016-02-24 16:18:23] MC[DF37] [5FE7]: will send files for /IMGPXYZ.JPG*

 

This file was there months ago with incorrect timestamp and I had deleted it when I saw the error. Since then I have upgraded to several newer versions of BTSync and the problem is still there. I have used different folders to sync, I have deleted all files and started from scratch, nothing worked! This file does NOT exist on any client these days but I still see it in my logs and the linux host doesn't sync this folder. Is it cached somewhere? How can I get rid of it?

 

Share this post


Link to post
Share on other sites

a few questions.

did you isntall btsync on the NAS? iow is it the NAS hardware / cpu / ram that you're using to index your 3.7TB of data?

To sync that many files and folders could require more RAM than your device has built in. We've run into similar issues syncing a similar amount of data.

We're currently making out ok, thought it's not been entirely troublefree, by using dedicated Windows desktop clients to run the Sync software with mapped UNC drives. There has been issues previously with performance on mapped UNC drives.

We do think that we're getting close to the point where we like the software, for as many issues as we've had getting it so far. I think we've been pushing it's limits a bit with the size of our sync data.

Share this post


Link to post
Share on other sites

zczcz,

because of incorrect timestamp (thus file signature corrupted) the deletion cannot be propagated. re-add the folder. or try recreate file with the same name in same directory on the source. 

AquaRelliux,

sorry, but there is only logs from source. again, merging stops because the files have bad signatures. I wonder if your files indeed have * at the end? Also, did you try re-adding the folder? 

Share this post


Link to post
Share on other sites

I did try to readd it serveral times, why can't it handle this? I just wanted a replication of a files not to go through all my thounds of files to find bad file names. ownCloud handle this replication without changing the folder names, please fix!

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.