Advice Needed: Rasberry Pi Build, Btsync, External Hdd

[advhound]

 Hello all,

My brother and I have been using BTSync for a couple weeks now and have been really impressed by it. We are going to build a system to mirror our data to each other's house by using BTSync. Our goals are to keep it as low power, quite and reliable as possible while sticking to a reasonable budget, hopefully under $300 for each build. We are not expecting these to be extremely fast, but instead the qualities mentioned above. We leaning towards building off of the Raspberry Pi platform, and we need your input on the following:

-Is the WD20EZRX hard drive a good choice for a reliable, cool and quite hard drive? Again, we don't need a screaming fast hard drive for this setup. Cost effective and reliable.

-We each need 2TB of storage, so each Pi will need 4TB attached to it. Would you recommend to get 2 separate 2TB drives, or to go with (1) 4TB hard drive on each build?

-Which hard drive enclosure do you recommend to run the drive(s) 24/7? We would like to run the hard drives without a fan to keep the power usage as low as possible and to keep them quite. We are even open to a solution that doesn't enclose the hard drives and allows the heat to dissipate into the air instead of enclosing it. We could enclose the entire system, Raspberry Pi and all in a cage-type cover that ventilated extremely well.

-This is related to the enclosure, but should we be looking for enclosures that allow spin down, or is that harder on the hard drives than letting them spin 24/7? Is there an enclosure that you guys have successfully used with a Pi?

-What would the bottleneck in this system be? Any other input you may have, or caveats that we are overlooking would be helpful! 

-Is there a summary on how encryption could work in this arrangement across different OS's, including Android?

Edited November 3, 2014 by advhound

[ricklahaye]

Hi,

 

Please note PI is only 700mhz.

It will indeed spin down your HD when it is not in use. I would advise to take 1 harddisk with external power supply. 

 

Also note that your disk is a 3.5 SATA disk, PI has no sata ports only usb. Therefore you need a USB disk with a external power supply. 

If you don't have a external power supply there is a big chance it will not be usable as a PI is only powered by 5v USB.

 

Also i don't think the PI can handle files over 1gb. 

 

Bottleneck would be CPU > RAM > DISK

[freigeist]

I have a similar setup running well. In your case you would need extra power supply for the drives and this ruins your energy saving goal. Besides the USB limitation mentioned above.

In your case I would prefer a carefully designed pc with good idle power consumption. You can achieve 10W with the advantage of plenty power when needed like indexing etc.

[piotrnik]

Something like the celeron Intel NUC may fit what you're trying to achieve - low power, low footprint, (relatively) low cost, but with a bit more power than a pi

[RomanZ]

@advhound

Also note, that Sync requires around 1Kb of memory per file / folder being synchronized. As RPI is pretty limited in memory (only 512Mb), it means that it will definitely have the limitation of ~500K of files total (that is not counting space necessary for OS to run on your Pi).

[advhound]

@RomanZ, what sort of energy efficient setup would you recommend with the best bang for buck to run 24/7? I'd consider picking up something used as well. I do not need the ability to run RAID setups etc. at this time as my files will be backed up remotely.

@piotrnik Good suggestion. The NUC does look like a decent setup, although once fitting with memory and an SSD, you're going to be over $300.

@freigeist What setup in specific would you recommend for a PC with good idle power consumption?

@ricklahaye Is there a consensus on HDD wear to spin down a HDD compared to leaving it spinning all the time?

[duch]

@advhound

Also note, that Sync requires around 1Kb of memory per file / folder being synchronized. As RPI is pretty limited in memory (only 512Mb), it means that it will definitely have the limitation of ~500K of files total (that is not counting space necessary for OS to run on your Pi).

 

Even without accounting for the OS the limit will probably be way lower, on my Pi btsync needs 40 to 50 MB of memory to sync 29k files.

 

Furthermore the 512 MB of memory of the Pi B or B+ can't be wholy used by system and/or user processes, a fraction of it has to be dedicated to the GPU (which is required in the boot process). The minimum is supposed to be 16 MB but I had trouble with values lower than 32 MB.

[j1miroquai]

I've been using a raspberry pi wth a usb external disk  for almost 2 years now.

To manage the power efficiency I plugged them all through a powered usb hub.

 

Since then everything works just fine with no problems.

 

I thing it's a pretty solid solution if you want to plug usb drives that don't come out with their own power.

[RomanZ]

@advhound

I'm not proficient in building such setups. Though I could advise to be careful with passive HDD cooling: any HDD has a bunch of mechanical parts which is very temperature-sensitive and every degree matters against the term of HDD service.

[advhound]

I have 27,000 files now at about 67 GB. Do you think the Pi would be able to keep up and allow the number and size of files to grow in the future? Using the 1kb/file for memory usage, I'd be using just over half the memory to sync and then add the OS memory needs to that. That would allow some room to grow in the next couple years.

[j1miroquai]

I have 27,000 files now at about 67 GB. Do you think the Pi would be able to keep up and allow the number and size of files to grow in the future? Using the 1kb/file for memory usage, I'd be using just over half the memory to sync and then add the OS memory needs to that. That would allow some room to grow in the next couple years.

 

I have about 200 gb syncing across 2 raspberrys pi  , 2 pcs and a macbook.

I'm using it for nearly 2 years.

No disc crash , no file missing .

 

I have the raspberrys auto mount the disc and auto start btsync (even if the power fails at my home for some reason).

 

One raspberry is at home and the other at my office.

I use them as mini servers for btsync and the main pcs and macbook for every change I make.

 

You will not have any problem if you set it right at the beginning

[advhound]

@j1miroquai What advice would you recommend to "set it right"? I'd be curious to hear the details of your Pi builds? What HDD enclosure do you use etc.

[j1miroquai]

well not much to say here

 

1)just make sure you mount your harddisk (so you won't have to mount it every time it reboots)

2)change the permissions on your disk so that btsync can write delete etc

3)make btsync auto start .It can be done with a lot of ways but the one I choose is through crontab. Just made a crontab saying to start btsync on boot.

 

And you're set.

 

On my raspberry  I also installed minidlna (so I can stream music, pictures and videos across my home) and transmission (so I can download movies and view them through minidlna)

 

I'm telling you raspberry really solved a lot at my place.

 

If you need help I can direct you to some sites or tell you how exactly I did it

as for enclosure I use a cheap enclosure from a local store (I live in Greece so by telling you exactly won`t help).

But it's pretty chep so my guess is that any one you pick on ebay will do

[advhound]

Is it true that there is a 1 gigabyte file limit with a Raspberry Pi as mentioned earlier in this thread?

[j1miroquai]

I have some files bigger than 1 gb .

video files , iso's etc

they sync just fine

[advhound]

Hmm, you are making me think more optimistically about the Pi. Another issue I heard is that there pipe took an extremely long amount of time to index the files. How long did yours take to index? I wouldn't mind waiting overnight or 12 hours, even 24 hours on the long end, but a week plus seems almost unusable.

[j1miroquai]

well another thing I forgot to tell you

 

some mac files or some .picasa.ini (these are hidden you wont see them normally ) wont sync and raspberry seems like indexing forever.(it happened to me)

 

so you have to tweak a little your .syncignore file so it won't sync these things.

 

after the first sync that takes a lot of time if you have many files tha indexing takes about 20 minutes max (to tell you the truth I never measured it,  just guessing now).

[advhound]

@j1miroquai Thanks for all of the feedback, I'm thinking it through. I may take you up on your offer on more specifics! Thanks again.

[freigeist]

@freigeist What setup in specific would you recommend for a PC with good idle power consumption?

See this list for lots of efficient setups http://goo.gl/z8nt3A

[duch]

Is it true that there is a 1 gigabyte file limit with a Raspberry Pi as mentioned earlier in this thread?

 

No. Don't know where that came from.

[advhound]

Regardless of which solution I end up going with, we just hit the jackpot on a great deal on 3TB drives that we are going to sync between my brother, Dad and I. My first thought was to partition the drives right off the bat into 3 separate, 1 for each of us. Are there any downsides to this? Would you recommend another solution? The basic goal would be to have some form of separation in the case of a virus etc. and to a lesser extent to keep each other's data somewhat private, although I am syncing with family, so no big worries there. Penny for your thoughts!

[Moe]

Just make sure the drive is designed for 24/7 use.. a lot of reading and writing files to the drive at them same time on diferent partitions can decrease the lifetime of the harddrive by a lot.

 

Oh and having three partitions on a harddrive won't help if a "virus" is on the harddrive they will find a way to get to any part of the harddrive and spread as wide a possible. Just make sure all your sync peers have a current antivirus installed - and maybe make sure it's also a good one haha

[GieltjE]

We have been running on an old rpi b (256M variant, raspbian with tweaks), which has become stable with the most recent updates (before that sync would break alot).

It has been running a setup which currently holds about 800G with 100k files (maxing at about 1MB/s).

 

If the disk doesn't spindown install and configure spindown as most usb disks won't spindown otherwise.