ccorsani Posted April 24, 2013 Report Share Posted April 24, 2013 Hi all. Great Job.My name is Cristiano, I work for National Library of Florence. We have built a scalable storage (at the moment 32TB) for storing book digitalizations. We have 3 32TB node in 3 different geographic position.For local redundancy we use glusterfs file system(replica 2 on 64TB fs) but not for geographic replica because it does not satisfy our needs (it is like a backup, not a peer to peer system). We was thinking about a torrent-like system.And here bittorent sync arrives! I just wondering how it can works with huge number of files (immagine that our 32TB storage is subdivided in 200MB compressed files - that contains jpg, pdf, tiff and so on - and it will grow a lot in time)Thanks a lot for your job Quote Link to comment Share on other sites More sharing options...
upcboy Posted April 24, 2013 Report Share Posted April 24, 2013 I believe the current version supports 200k files if you have more files than that the program will crash. Quote Link to comment Share on other sites More sharing options...
kos13 Posted April 24, 2013 Report Share Posted April 24, 2013 If I do the math correctly you have around ± 150,000 files. Sync should perform good on this data set. Quote Link to comment Share on other sites More sharing options...
ccorsani Posted April 26, 2013 Author Report Share Posted April 26, 2013 Yes, but the problem is that this storage will grow up quickly! And I wonder if is correct as it is. I mean: our goal is tot maintain 3 geographic copy. Immagine, for example, one disk crash in one node. The system should replicate automatically from the other two (or more) nodes the files deleted when you bring up the new (void) disk. So you need a sort of "only add - no delete" sync. We can eventually modify source to adapt to our needs (that could be an extra option for all):1 - we need huge (growing) number of files to sync2 - whenever we "join" two or more peer to one "secret" the system sould never delete files but it must works to maintain all the files added. Every file should be propagated to every node every time one node loose or not have it.Sorry for my english. Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.