yottabit
-
Posts
143 -
Joined
-
Last visited
-
Days Won
2
Posts posted by yottabit
-
-
Did you share the Linux directory (source) as read-only secret to the Windows machine (destination)?
-
Want me to look things over for you? Run the following commands from a command prompt on both computers, then copy and paste the results here.
- ipconfig /all
- route print
- ping google.com
- ping 8.8.4.4
If you're familiar with the output of ipconfig, also try pinging the other computer from each computer. If you aren't, I'll give you the commands to use once I see the output from the ones above.
- ipconfig /all
-
You guys completely rock! I've rarely had the opportunity to use a product where the developers are so engaged with the active user community and so quickly respond to major requests such as building the product for a different OS (especially when it comes to UNIX!).
I'm greatly looking forward to using the new build.
-
GreatMarko's suggest to use read-only "one-way" sharing will hopefully be the easiest and best solution for you.
If you're CLI saavy in Windows and/or UNIX, you can use links to pull in only certain live data to a folder you share with family (read-only or read-write, as appropriate for your case). Then you can make updates/modify the files in their original locations on your drive and those updates will automatically be shown in the shared BTSync folder(s). The beauty of using links is that there is no duplication of data on your drive, and you don't have to remember to copy the updated files into the BTSync folder(s).
If you have this skillset, here's a lot more info I wrote on the subject:
Whichever OS you're using, links are available. There are slight differences in names and how they work, but all roughly equivalent. Windows, MacOS, and Linux.
-
I successfully sync ~100 GB of data over a CIFS mountpoint from Ubuntu Server Linux to my FreeNAS 8.
I suspect a permissions problem possibly?
-
Can you create an md5 checksum index of the store that shows the most files, and then do a check against the other stores? That will tell you everything you need to know about the consistency ...
-
After adding ip & port to the share definition it works.
It seems that our routers between the subnets are blocking the broadcast packets.
Yes, this is to be expected. One of the reasons to subnet an IP network is to separate your broadcast domains. An IP broadcast will not leave the IP subnet; that is to say, a router will not forward a broadcast to a different subnet and that's by design.
I really like perennate's suggestion to add a few peers manually like you did, and then turn on DHT. On a true LAN environment (not separated by slow WAN links) the overhead chatter of DHT shouldn't matter. This sounds like a great solution until BTSync is updated to allow manual Tracker.
-
FTP can be a fairly fast protocol for sure, especially when running parallel transfers. FTP's primary disadvantages include:
- An old-school, ill-conceived design separating the control channel and the data channel, which causes numerous NAT problems due to random port allocation for the (reverse-direction) of the data channel
- No built-in compression for compressible files
- Insecure (cleartext!) authentication by default, and no encryption for data channel
- Some hacked up FTP can use SSL/TLS for authentication, but it's a real pain to setup because so many servers/clients do it in so many ways
I really like that BTSync seems to "just work" in almost all cases, includes strong encryption by default, uses the venerable p2p protocol technology from BT, and has some options to disable encryption on the LAN when warranted.
And it's alpha! It will only get better from here!
- An old-school, ill-conceived design separating the control channel and the data channel, which causes numerous NAT problems due to random port allocation for the (reverse-direction) of the data channel
-
Yes, definitely easier for most novices, but then again they have to know how to edit a (hidden) text file to begin with.
At least with my script they can just run it and be done. The actual linking/junctioning is really hidden from them and as long as they understand that the new BTSync folder's contents are not copies but simply additional names to the same data, they should be good. Even if they delete one or the other, their data is still intact and safe.
-
Go to a command prompt and do "ipconfig /all" and "route print", then copy & paste put into a reply here. I'll see if anything looks fishy.
-
Well typically when one refers to LAN broadcasts we would be using layer-2 Ethernet MAC broadcasts to MAC address FF:FF:FF:FF:FF:FF.
It is possible to do layer-3 IP broadcasts using the broadcast address of the subnet (e.g., 192.168.1.0/24 network has a broadcast address of 192.168.1.255), and that's how ARP works for IP-to-MAC resolution, but apart from that it's very rare for an application to use an IP broadcast.
I would be very, very surprised if BTSync was using IP broadcast instead of MAC broadcasting.
If I had more time I'd break out a protocol sniffer and check for myself ... maybe sometime next week.
-
The clients are on different subnets - i think the broadcast frames did not reach the other subnet.
Each subnet has its own ip adress range.
Well that's still interesting. Maybe the LAN broadcast functionality doesn't work like this, but I would assume that the IP:Port of the client is being broadcast, so when a client attempts to make a connection, it would go through the default route to reach the other subnet through the router.
You do have a default route and router, don't you? ;-)
And if it doesn't work that way, manually adding the IP:Port of the other computers as I described should definitely work. (Again, assuming you have a default route and router.)
-
The original poster asked about sharing specific files with others because he doesn't want to just "sync files between different machines, but also share (specific) files with other users."
Using hardlinks and/or directory junctions allows him to set up shares of his selective/specific without duplicating/copying/updating/maintaining that data in more than one place.
-
Yup, that's another way. But I find that over time my profile directory becomes more and more full of cruft.
Some applications add their own junk in there without putting it in the proper subdirectory, and I have been known--on occasion--to throw small utilities in their myself!
My way is an opt-in method, whereas using the .SyncIgnore is an opt-out method. Both work. Both have advantages and disadvantages.
-
Someone please help me understand. Why would TCP increase speed with slow CPUs?
TCP packets are larger, and require stateful connection tracking, and retransmissions in the case of corruption.
UDP packets are smaller, stateless, and there are no retransmissions at that layer (4--error detection and retransmission requests must be handled at a higher layer, by the application itself).
In the Internet world, UDP is always faster (on a reliable network), and I don't understand how it would possibly be a higher strain on the CPU except that the application may have to do more error checking since layer-4 isn't handling that for the transmission (but the application would want to do this anyhow, even for TCP, for best reliability and resilience--those properties for which the BT protocol is famously known).
The uTP protocol is based on UDP for the very reason that UDP is less overhead and therefore faster.
Most network interface hardware these days can offload the TCP checksums, but that's a relatively minor operation on today's (and even yesterday's!) CPUs. UDP doesn't even have these checksums.
What about BTSync specifically would make TCP cause a lower load on the CPU?
-
Try adding the IP address and Port of each computer to the share definition:
- Preferences -> make note of the port number on each computer
- Shared Folders -> double click the share
- Hosts -> Add -> put in every other computer's IP address and port number
- Restart BTSync for good measure (might want to check the port number again in step #1 after doing this to make sure it didn't change?)
Also, even if this does work, there doesn't seem to be an option in the GUI version to not randomize the port on each start ... not sure if it does that by default or not ... I haven't checked.
EDIT: you know, I was thinking about this more. BTSync sends broadcast frames on the LAN so the clients find each other without the Internet. So I wonder if you have a software firewall installed on one of the computers that may be preventing this behavior? The default Windows Firewall shouldn't block it since BTSync enters exception rules for it.
- Preferences -> make note of the port number on each computer
-
UDP is default. By setting TCP to True, you're disabling UDP.
-
The upload connection is on a 500mbit connection. The download connection is "up to" 50mbit.
[...]
Pretty regularly, I can get up to 5-6MB/sec to/from these devices [...]
6 MB/s = 6 * 8 = 48 Mbps ... so you're getting very close to your max download, which is imo remarkable given the encryption overhead and various other things that could be contending for time on your Internet connection.
Later in the thread you mentioned 2-3 MB/s, which is still not bad, depending what else you have going on.
I saw a ~huge~ increase in speed on a direct LAN transfer after disabling the encryption. I noticed it was bogging down on the virtualized Linux server since btsync appears to be single-threaded and was maxing out one core.
-
And by UPD traffic, kos13 actually means UDP traffic. :-P
You can also use the advanced options to switch BTSync to use TCP instead of UDP on your LAN and see if that makes any difference. I would be very surprised ... which brings me to another thought...
Are you using a firewall software on your computers? Like ZoneAlarm or Kaspersky Internet Security, etc.? Disable those and see if it works ... BTSync makes firewall exceptions into the default basic Windows Firewall, but it probably doesn't handle the others, and you may not have noticed those programs ask you to open the ports (or have those alerts turned off)...
Just a thought.
-
I just wrote a reply to another thread where a user wanted to share individual files, not folders. Of course BTSync isn't designed to handle this scenario, but there's always a work-around, once again using features stolen copied borrowed from UNIX filesystems now implemented in NTFS. I wanted to post into this thread because it's so closely related to my script above.
In this case, rather than linking in whole subdirectories, we hardlink individual files into a directory that's shared. Three huge advantages to this method:
- Doesn't use additional diskspace because these aren't copies but simply additional filenames that point to the exact same data on the disk, and
- If you update/modify any one of the hardlinked files, all of the others reflect the same changes (again, because they all point to the exact same data on the disk), and
- After hardlinking, you can feel free to delete any of the files you like (even the original one!) and all of the other hardlinked files will still contain the exact same data and relationship.
In other words: Neato! Here's the scoop:
Here's another way! If you're using Windows 6.x (Windows Vista or Windows "7"), use hardlinks to keep the hassle to a minimum.
- Create a new folder that you will share with BTSync.
- Drop to CLI and use "mklink /H" to hardlink the files you want to share into this new folder.
- Drink a beer and relax!
So here's the trick. If you aren't familiar with UNIX, a hardlink is simply another filename pointing to the same data on the disk. So let's say you have a file called MyBigFile located on your Desktop. Well, you don't want to share your whole Desktop, but just that one file. So we do this:
- Create new folder on Desktop called SharedFolder.
- Open a CLI window (Start -> "cmd" at the bottom -> click cmd.exe in the search results at the top).
- cd %userprofile%\Desktop\SharedFolder
- mklink /H MyBigFile ..\MyBigFile
That's it!
You can use this method to even change the name of the file, but not duplicate the data like a copy would. And with a hardlink, now both files are just pointers to the same data on the disk, so you can feel free to delete one without damaging/deleting the other. For instance, if you hardlink file1 to file2 and file3, you can delete 1 or 2 of those (even file1!) and the others will still work just peachy. And just to be clear, if you edit one of those files, the updates will appear in the others as well (because like I said, they're just different names but pointing to the same data on the disk).
Oh, and as usual, Microsoft implements features into Windows without providing any method of using them from the GUI. Some of these features even require third-party utilities! So sorry about the CLI if you're not comfortable with it, but this is the easiest way to go. There is also a third-party GUI utility called NTFSlink that was used prior to Windows 6.x to provide access to this function before Microsoft even had a CLI utility. I don't know if using it in Windows 6.x is supported, but if you install it you can just right-drag files into the new folder, and from the context menu select "Create Junction Here" (for directories) or "Create Hardlink Here" (for files).
http://sourceforge.n...ts/ntfslinkext/
EDIT: I guess NTFSlink is no longer under development. Here's an alternative that also works with Windows x64: http://schinagl.priv.at/nt/hardlinkshellext/hardlinkshellext.html
- Doesn't use additional diskspace because these aren't copies but simply additional filenames that point to the exact same data on the disk, and
-
My question is: will it support sharing files with others? I don't just want to sync files between different machines, but also share (specific) files with other users.
SyncApp is designed to share complete folders, rather than individual files.
The simple solution is to create a folder dedicated for sharing with others, and only drop the files you wish to share into that folder.
Here's another way! If you're using Windows 6.x (Windows Vista or Windows "7"), use hardlinks to keep the hassle to a minimum.
- Create a new folder that you will share with BTSync.
- Drop to CLI and use "mklink /H" to hardlink the files you want to share into this new folder.
- Drink a beer and relax!
So here's the trick. If you aren't familiar with UNIX, a hardlink is simply another filename pointing to the same data on the disk. So let's say you have a file called MyBigFile located on your Desktop. Well, you don't want to share your whole Desktop, but just that one file. So we do this:
- Create new folder on Desktop called SharedFolder.
- Open a CLI window (Start -> "cmd" at the bottom -> click cmd.exe in the search results at the top).
- cd %userprofile%\Desktop\SharedFolder
- mklink /H MyBigFile ..\MyBigFile
That's it!
You can use this method to even change the name of the file, but not duplicate the data like a copy would. And with a hardlink, now both files are just pointers to the same data on the disk, so you can feel free to delete one without damaging/deleting the other. For instance, if you hardlink file1 to file2 and file3, you can delete 1 or 2 of those (even file1!) and the others will still work just peachy. And just to be clear, if you edit one of those files, the updates will appear in the others as well (because like I said, they're just different names but pointing to the same data on the disk).
Oh, and as usual, Microsoft implements features into Windows without providing any method of using them from the GUI. Some of these features even require third-party utilities! So sorry about the CLI if you're not comfortable with it, but this is the easiest way to go. There is also a third-party GUI utility called NTFSlink that was used prior to Windows 6.x to provide access to this function before Microsoft even had a CLI utility. I don't know if using it in Windows 6.x is supported, but if you install it you can just right-drag files into the new folder, and from the context menu select "Create Junction Here" (for directories) or "Create Hardlink Here" (for files).
http://sourceforge.n...ts/ntfslinkext/
EDIT: I guess NTFSlink is no longer under development. Here's an alternative that also works with Windows x64: http://schinagl.priv.at/nt/hardlinkshellext/hardlinkshellext.html
- Create a new folder that you will share with BTSync.
-
Hi DragonDutch,
Please read the guide. It's very well-written.
http://labs.bittorrent.com/experiments/sync/get-started.html
-
That should "just work."
Maybe your computers are having a hard time talking to the BT tracker on the Internet?
Try adding the IP address of each computer to the share definition:
- Preferences -> make note of the port number on each computer
- Shared Folders -> double click the share
- Hosts -> Add -> put in every other computer's IP address and port number
- Restart BTSync for good measure (might want to check the port number again in step #1 after doing this to make sure it didn't change?)
Any difference?
If not, probably need to turn on the debug log and send it in for analysis.
Also, even if this does work, there doesn't seem to be an option in the GUI version to not randomize the port on each start ... not sure if it does that by default or not ... I haven't checked.
- Preferences -> make note of the port number on each computer
-
Running Windows 7 here. My profile directory contains 52,125 files totalling 52.1 GB.
A large portion of those files are Java caches, web browser caches, temp files ... all things that would be stupid to sync.
The only subdirectories I'm interested in syncing (Documents, Desktop, Downloads, etc.) contain only 10,206 files totalling 29.6 GB--only 20% of the files and 57% of the data. There is an easy way to put all of these subdirectories into a single directory so that a single folder share can be made with btsync (instead of an individual share for each one of the interested subdirectories).
NTFS 3.0 and higher (Windows 2000) support Directory Junctions, which are very similar to symlinks (symbolic links) in UNIX. In Windows 6.x (Windows Vista and Windows "7") you can create Junctions from the command line with the "mklink" utility, included already with Windows.
Using this utility, I can create Directory Junctions, of only the subdirectories I'm interested in syncing, into a new subdirectory. I then create a single btsync share of that new subdirectory which now encompasses all of the other directories--thanks to Directory Junctions--without actually being a "copy" of the data that would need updated.
I wrote a little command script that does this for you. Copy this text into a new text file, then rename it "btsync-win7-profile.cmd", after which you can double-click it to run.
I also attached the file directly to this post to make it even easier.
@echo offREM mklink script for Win7
REM by Jacob McDonald, btsync@McDonaldTechnology.com
REM version 130424a-jmmcd: initial version
REM version 130425a-jmmcd: removed extraneous 'echo' in description
echo.
echo This script creates a new subdirectory called "BTSync" "in your
echo user profile subdirectory. This will be located here:
echo.
echo %userprofile%\BTSync
echo.
echo By default this script links in Contacts, Desktop, Documents,
echo Downloads, Dropbox, Favorites, Music, Pictures, Saved Games,
echo and Videos.
echo.
echo ******************************************************
echo *** ***
echo *** This probably only makes sense with Windows 7! ***
echo *** ***
echo ******************************************************
echo.
echo If you want to use it on a different Windows version, or if you
echo want any other subdirectories not listed above, please edit this
echo script first. It's pretty self-explanatory.
echo.
echo Ready to begin. Press Ctrl+C (then answer Yes) if you want to stop.
pause
cls
cd %userprofile%
mkdir BTSync
mklink /J "BTSync\Contacts" "Contacts"
mklink /J "BTSync\Desktop" "Desktop"
mklink /J "BTSync\Documents" "Documents"
mklink /J "BTSync\Downloads" "Downloads"
mklink /J "BTSync\Dropbox" "Dropbox"
mklink /J "BTSync\Favorites" "Favorites"
mklink /J "BTSync\Music" "Music"
mklink /J "BTSync\Pictures" "Pictures"
mklink /J "BTSync\Saved Games" "Saved Games"
mklink /J "BTSync\Videos" "Videos"
echo.
echo Finished! Now create a new BTSync share of the following location:
echo.
echo %userprofile%\BTSync
echo.
echo Please note: you can delete the entire directory above at any time
echo without affecting any of the data to which it links.
echo.
REM btsync will have CLI option support in a future version; then you can
REM hopefully launch it to share the subdirectory above automatically at
REM completion!
pause
Enjoy!
Bug: Syncing changed filenames and deleted directories
in Sync General Discussion
Posted
Okay. Well I agree that files shouldn't be deleted but there must be something done about the incompatible characters between filesystems. Without building and maintaining a compatibility mapping for these cases (which is a significant undertaking) I think you should expect either a rename or lack of sync for incompatible files.
The directory that went missing isn't by chance in the .SyncTrash subdirectory is it? At the very least it should be--otherwise that's an additional bug...