How To Reliably Check If File Is Uploaded On Server?

[j2b]

The intention of experiment, is to use BTsync as an efficient transfer mechanism, to copy archive files between servers. As according to this forum info, versioning is not reliably working, I had an idea, to catch, when particular file is completely transferred to server #1, and use additional tools on server #2 to move ready file from particular sync folder to archive folder. For example:

 

1. Server #1 prepares archive, and places it in sync folder, linked with BTsync to Server #2

 

2. BTsync start transfer (new file on Server #2 meanwhile has a name with .!Sync extention).

 

3. When the file transfer is completed, it becomes without .!Sync extention on Server #2

 

4. Check this status on Server #2, and invoke script to move the file to Archive folder, from Sync folder on the same server #2

 

5. By moving file to Archive folder, this would invoke delete procedure of file on Server #1 via BTsync principles, thus freeing up space on Server #1

 

 

There are a couple of complexity scenarios regarding interrupted transfers, versioning, changing the content of synced file, etc. But for now, this would be the simple scenario - when file is uploaded to server #2, move it to another Archive folder outside BTsync functionality, and invoke delete on Server #1.

 

The question is - how to reliably find out, if the file is completely transferred to server #2? Or, does server #2 has complete correct file copy, to start other operations on it?

 

Looking in log file, didn't give any reliably signs nor entries, that file xxx.zip transfer was completed successfully, which was the first place I was looking for confirmation from BTsync process. But there are no such.

 

May be somebody can throw some ideas, on how this might be achieved? Or may be other solutions can result in this scenario completion? The main idea is to move archives from server #1 to server #2, and delete them from server #1 after sync.

[RomanZ]

j2b,

 

There are at least 2 ways to achieve what you need.

 

1. Disable moving files to sync archive on server #1.

2. Wait until file appears on server #2. When !sync file is renamed to the real file name - it means that it was delivered, not damaged and it is okay.

3. Remove file from sync folder #2 on server. It will be automatically deleted on server #1.

 

Second way, use API.

When using API - call command "Get folder peers". When "upload" and "download" equal zero indicate that transfer is complete. You can use it to check if file was actually transferred.

[j2b]

Thank you RomanZ for your insight.

 

Yes, I was planning to initiate deletion of archive from server #1 via moving archive from sync folder on server #2 to Archive folder. Thus initiating delete procedure from server #1. In this scenario, I didn't get understanding on your 1.point - Disable moving files to sync archive on server #1.

 

The case is, that Server #1 has it's own scripts, building these archives, and I can not interact with these scripts, and preferably wold not want to. E.g. let Server #1 do his own things as formerly configured.

 

At current state of idea, file names on Server #1 are different each time, which cuts off some difficulties on file identifications. Atleast for this scenario.

 

Can you please give more details on your meaning for point 1.? The other parts are clear. The thing, which troubled my mind was, can it be reliable, to consider, if .!Sync extention is removed, should we make any other checks, to understand, if file is correct. From your reply I understand, that it is reliable.

 

The second way - kind of interesting. I'll try to play with API. If I get it right, when we query all folder peers (in case some of them have transferred all chunks of file, others - not), it still gives us complete picture of file status. E.g., if there would be some peers, that haven't completed sync, it will result in difference between "upload" / "download" figures, not having to bother about number and state of these peers. Am I correct?

 

Thanks again!

[RomanZ]

In this scenario, I didn't get understanding on your 1.point - Disable moving files to sync archive on server #1.

 

 

There is a setting in folder properties, see screenshot attached. Unchecking it will prevent deleted files to move to syncarchive, 

 

 

E.g., if there would be some peers, that haven't completed sync, it will result in difference between "upload" / "download" figures, not having to bother about number and state of these peers. Am I correct?

 

 

Sort of. In your sample, for server #1 if there still data to send - you'll see non-zero value in "Upload" section. Server #2 will see data in "Download" section.

[j2b]

1st issue:

Ah, ok, I got it. I was thinking, that this would be kind of solution, to temporary prevent Server #1 putting new archives in folders. OK, this would be to ensure, that actually deleted archives in Sync folder, in future do not take place, while they are in .SyncArchive folder according to set up TTL value.

 

2nd issue:

Thank you for clearing this up. This was another way, what I was thinking, to check whether differences in file upload/download, or even monitor file size changes with local script on server #2.

 

There will be no GUI on these servers, besides btsync installation web interface, if needed. But, as far as I know, it can be set by Web GUI too. Or at least - via config file.

 

Just noticed another problem - Server #1 is CentOS. Do you have any reliable info, on whether there is a distribution for CenOS?