rk8102 Posted March 12, 2014 Report Share Posted March 12, 2014 I've been looking around and searching on this, but haven't found much of anything... Basically, I have about 1tb of files (music, movies, etc.) that I want to sync between my home Plex server and a system at work, for backup purposes, etc. All the files are currently at home, behind your standard cable modem connection (12mb down, 3 up or something like that) and work is a major university so we have tons of bandwidth. How do I avoid a huge initial sync of the existing files? Can I bring a copy of the data from home on a drive, drop it onto the work machine and then set up the sync? I don't want to crush my cable connection at home for days, either up or down... Quote Link to comment Share on other sites More sharing options...
GreatMarko Posted March 12, 2014 Report Share Posted March 12, 2014 How do I avoid a huge initial sync of the existing files? Can I bring a copy of the data from home on a drive, drop it onto the work machine and then set up the sync?Yes! If your files are in sync on both devices before you actually add them both to Sync, they your files will not transfer again. Quote Link to comment Share on other sites More sharing options...
rk8102 Posted March 12, 2014 Author Report Share Posted March 12, 2014 So how does it decide the files are the same? MD5sum, filename, date/time, etc. Quote Link to comment Share on other sites More sharing options...
Harold Feit Posted March 12, 2014 Report Share Posted March 12, 2014 segmented sha1 validation. Quote Link to comment Share on other sites More sharing options...
rk8102 Posted March 13, 2014 Author Report Share Posted March 13, 2014 Oh that's just great, thanks very much for the help! Quote Link to comment Share on other sites More sharing options...
Bartus Posted March 13, 2014 Report Share Posted March 13, 2014 Hi I'd like to hijack this topic since I have a similar question. Originally I falsely assumed that pre-seeding of nodes was not possible because the copied files would have a different timestamp. Since this actually is possible as I read here, would it be possible to delete a partly synced folder on one of my nodes and copy paste the files from one of the already synced nodes on there? Or is it so that this only works with files that haven’t been in contact with bitsync before at all?!? Quote Link to comment Share on other sites More sharing options...
Harold Feit Posted March 13, 2014 Report Share Posted March 13, 2014 In your case I would simply recommend pausing the sync on that machine and doing a copy-and-paste of the remaining files. Deleting files in a synced folder while a sync is in progress may cause unintended data loss. Quote Link to comment Share on other sites More sharing options...
AzaK Posted April 8, 2014 Report Share Posted April 8, 2014 (edited) Yes! If your files are in sync on both devices before you actually add them both to Sync, they your files will not transfer again. Can I do the copy after the folders have been set up on BTSync on the source? I have a large number of folders I have just set up and I don't want to have to remove them and start again. Could I just delete all the .Sync files (Except the SyncID) in the source folder (and destination folder where some have been copied) and it'll work? Additionally, copying an already added folder to another disk, deleting all the .Sync files and adding it to BTSync on the destination machine seems to resync all the files if it's set up as read only (What I need). Thanks Edited April 8, 2014 by AzaK Quote Link to comment Share on other sites More sharing options...
GreatMarko Posted April 8, 2014 Report Share Posted April 8, 2014 Azak, if you wish to "pre seed" your files, you should do this with Sync not running whilst you manually copy files/folders across (excluding copying the .SyncID file), then start Sync on your devices. Sync will then re-index the folders, but as long as their contents are identical, nothing should transfer between devices. Quote Link to comment Share on other sites More sharing options...
AzaK Posted April 8, 2014 Report Share Posted April 8, 2014 (edited) Azak, if you wish to "pre seed" your files, you should do this with Sync not running whilst you manually copy files/folders across (excluding copying the .SyncID file), then start Sync on your devices. Sync will then re-index the folders, but as long as their contents are identical, nothing should transfer between devices. Hi, thanks for the reply. I just tried your suggestion and these are the steps I took and this is what happened: I closed Sync on both machines.Copied the source directory to destination machine.Deleted the SyncID file on the destination machine. I left the .SyncIgnore and .SyncArchive file/folderRan Sync on the source machine, folder was indexed.Ran Sync on destination and added the folderIndexing flashed up quickly. Folder showed as 0 files and 0 bytes (There were lots of files in the folder)The Data up/down was showing bytes transferring, but the transfer window was empty....at first.After a few seconds, transfers started and it looks like they are all 'somepath\com.apple.FinderInfo' files although I can't be sure there weren't other sneaking in there. The History on the destination says "Finished syncing file XXX" and those XXX's are the names of the files in the folder, not just the com.apple.FinderInfo files. Edited April 8, 2014 by AzaK Quote Link to comment Share on other sites More sharing options...
AzaK Posted April 8, 2014 Report Share Posted April 8, 2014 Sorry for adding another reply, I can't seem to edit my post, but in the test above, I was using a Read\Write folder. I did another test today at work and if I add the folder on the destination as Read Only, then it shows up as having no files (Even though there are files copied there) and when I then start up Sync on the source, everything is copied again. I think read only folders should still index and include their existing files in the test for de-synchronised files. Quote Link to comment Share on other sites More sharing options...
piotrnik Posted April 9, 2014 Report Share Posted April 9, 2014 For me, I went through a similar process. It seemed to work for me without re-copying the files, so perhaps you may find what I did helpful. I was using a hamachi/synctoy setup previous to BTSync, with both r/o and r/w folders. Previous to my transition to BTSync (made due to pause/resume, parallel transfers, and no LAN required), I made sure the folders were fully up to date via synctoy. I then added the folders on one computer to BTSync, waited until it finished indexing there, and then added the folders on the other computer (with appropriate r/o or r/w settings). Initially, it showed 0 kb in 0 files on the main tab on the destination, and reported the full file size as needing to sync on both computers in the "devices" tab (showing as needing to upload from the source, and needing to download on the destination). As it scanned, on the destination computer it would flash files in the "transferring" tab and then show them as finished syncing in the "history" tab (like you described above). On the source computer, it would flash the same filenames in the "transferring" tab, but with up/down rates of 0.0kbs, and it never showed them in the "history" tab. During this process, the filesize showing in the "device" tab remaining to be synced would gradually decrease as the destination computer indexed (much faster than my internet connection is possible of, I might add; thus I'm reasonably certain that it never transferred more data than file hashes to verify the integrity of the files already on the destination), until it finished and I got the "Finished syncing with [computer]" in the "history" tab on both computers. After it finished syncing/indexing, I added files on both computers to test the r/o and r/w capabilities, and it worked flawlessly at detecting them and transferring as needed. As noted, I did this for both r/o and r/w folders (with some r/o folders also having the "restore modified files" option checked) and it did index the files already there successfully. I believe that out of 5+TB of data, it only re-copied perhaps 20-30MB in partial pieces of files (I think those being metadata that had been changed in a r/o folder by a media player on the destination). Perhaps your problem could come from leaving some of the .sync files/folders? My directories were identical prior to adding the folders to the source/destination computers, so I never had the .sync files/folders on either until they were autocreated when I added them to BTSync. Quote Link to comment Share on other sites More sharing options...
AzaK Posted April 10, 2014 Report Share Posted April 10, 2014 Thanks very much for the detailed response piotrnik. I'll try your method specifically and see if I have some success. I've since found another show-stopper w.r.t. alternate streams and the fact that the Windows client of BTSync treats extended attributes (They are named 'filename:streamname' ) like regular files and so when my HFS driver software on my WIndows destination machine sees them it treats them like regular HFS filenames where ':' is a valid character. Quote Link to comment Share on other sites More sharing options...
fedele.mantuano Posted February 16, 2015 Report Share Posted February 16, 2015 (edited) I have a host with A key and a Raspberry Pi with B key that sharing about 50 G of files. I use both LAN and Internet.I had to change the keys and now the status of folders flapping from Out of sync to Sending, but they have the same file so I don't understand.In B host there are logs like this:Error: /mypath/file - PostDownload: Permission deniedI use Version: 1.4.110 Beta.Can we help me? Edited February 16, 2015 by fedele.mantuano Quote Link to comment Share on other sites More sharing options...
RomanZ Posted February 16, 2015 Report Share Posted February 16, 2015 @fedele.mantuanoI suggest checking if the user running Sync on your RPI has enough permissions to write data to sync folder as well as to storage folder. Quote Link to comment Share on other sites More sharing options...
boruguru Posted February 16, 2015 Report Share Posted February 16, 2015 I have a relevant question to this, so I am going to hi-jack thread and ask it here. If I disconnect and re-add back a folder, should btsync start a re-index process? Example : If I have 1TB of files & if I disconnect and re-add the folder, does btsync need to re-read 1TB worth or data, or btsync has a way to quickly identify nothing has changed as it keep checksum/timestamp data some where? I also read in a thread, btsync need about 300-400 bytes of memory per file, is it to keep checksums of files? Is there a checksum database that it maintain in disk somethere? If everything is in memory, does btsync need to re-generate the memory checksums everytime it start? Some clarification here is appreciated? Thank you, Quote Link to comment Share on other sites More sharing options...
RomanZ Posted February 17, 2015 Report Share Posted February 17, 2015 @boruguruDisconnecting folder deletes the database of the folder. So yes, re-connecting the folder will force Sync to re-read all the data to build a tree of checksums for files and folders. Sync keeps in memory not only checksums but also some service information on files - like change time, which pieces of the file it has, etc. This information is dumped to DB to avoid recalculation when Sync starts up. Quote Link to comment Share on other sites More sharing options...
fedele.mantuano Posted February 19, 2015 Report Share Posted February 19, 2015 The user of Raspberry Pi is root. I tried to change B key with A key on Raspberry Pi, and I solved my problem. When I came back to B key after some hours the folders was synced. I don't no, it's only a workaroud. Quote Link to comment Share on other sites More sharing options...
RomanZ Posted February 20, 2015 Report Share Posted February 20, 2015 @fedele.mantuanoI can take a look at logs from RPI, but I still suspect that something is wrong with permissions there. Which user is running btsync? Is it also root? Quote Link to comment Share on other sites More sharing options...
fedele.mantuano Posted February 23, 2015 Report Share Posted February 23, 2015 Yes the user that starts btsync is root: 4 S root 7118 1 12 80 0 - 36778 futex_ Feb17 pts/0 17:06:50 /opt/bittorrent_sync/btsync --nodaemon --config /opt/bittorrent_sync/btsync.conf --log /opt/bittorrent_sync/log/btsync.log Now, all is synced but after about a day, with B key and in LAN sync.The btsync.log is on https://www.adrive.com/public/HVjhMd/btsync.log. Quote Link to comment Share on other sites More sharing options...
RomanZ Posted February 25, 2015 Report Share Posted February 25, 2015 @fedele.mantuanoThis issue was fixed in upcoming 2.0 release. I suggest waiting for the new beta build (the 2.0.52 announced yesterday does not contain it yet) and upgrade. Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.