Loading README.txt +28 −129 Original line number Diff line number Diff line Simple communication test that involes 5 docker containers: - client (container_name: client, files to run: 'vos_rest_client.py' and 'dataArchiver.py') Simple communication test that involves 5 docker containers: - client (container_name: client, commands available: 'vos_data') - server (container_name: transfer_service) - RabbitMQ (container_name: rabbitmq) - Redis (container_name: redis) Loading @@ -23,127 +23,13 @@ $ docker-compose up Once all the containers are up and running, open another shell and access the 'client' container: $ docker exec -it client /bin/bash At this point you can launch the 'vos_rest_client.py' within the 'client' container using the following syntax: $ python3.9 vos_rest_client.py QUEUE_NAME For example: $ python3.9 vos_rest_client.py start_job_queue The output should be something like this (you send and then you receive a response containing the info previously stored into the database): client@a89c0bb962f7:~$ python3.9 vos_rest_client.py start_job_queue Sending transfer request: { "jobId": "3ff92acedc9611eabf140242ac1f0007", "runId": null, "ownerId": "anonymous", "phase": "PENDING", "quote": null, "creationTime": null, "startTime": null, "endTime": null, "executionDuration": 0, "destruction": null, "parameters": null, "results": [], "errorSummary": null, "jobInfo": { "transfer": { "target": "vos://example.com!vospace/mydata1", "direction": "pullToVoSpace", "view": null, "protocol": [ { "endpoint": null, "param": [], "uri": "ia2:tape-recall" } ], "keepBytes": null, "version": null } }, "version": null } Response: { "job_id": "3ff92acedc9611eabf140242ac1f0007", "owner_id": "0000", "job_type": "pullToVoSpace", "phase": "PENDING", "start_time": null, "end_time": null, "job_info": { "transfer": { "view": null, "target": "vos://example.com!vospace/mydata1", "version": null, "protocol": [ { "uri": "ia2:tape-recall", "param": [], "endpoint": null } ], "direction": "pullToVoSpace", "keepBytes": null } }, "results": null } After processing the request, the server launches an internal thread delayed of 10 seconds which changes the state of the job from "PENDING" to "EXECUTING". You can easily verify this change by launching again the client in this other way: $ python vos_rest_client.py QUEUE_NAME JOB_ID For example, in our case: $ python vos_rest_client.py poll_job_queue 3ff92acedc9611eabf140242ac1f0007 The output should be something like this: client@a89c0bb962f7:~$ python vos_rest_client.py poll_job_queue 3ff92acedc9611eabf140242ac1f0007 Sending poll request: { "jobId": "3ff92acedc9611eabf140242ac1f0007" } Response: { "job_id": "3ff92acedc9611eabf140242ac1f0007", "owner_id": "0000", "job_type": "pullToVoSpace", "phase": "EXECUTING", "start_time": null, "end_time": null, "job_info": { "transfer": { "view": null, "target": "vos://example.com!vospace/mydata1", "version": null, "protocol": [ { "uri": "ia2:tape-recall", "param": [], "endpoint": null } ], "direction": "pullToVoSpace", "keepBytes": null } }, "results": null } --------------------------------------------------------------------------------------------------------------- Another thing you can do is to launch the 'dataArchiver.py' client. Now you can launch the 'vos_data' command. Launching the client without any argument will show you how to use it: client@28970a09202d:~$ python3.9 dataArchiverCli.py client@28970a09202d:~$ vos_data NAME dataArchiverCli.py vos_data SYNOPSYS python3.x dataArchiverCli.py COMMAND USERNAME Loading @@ -152,34 +38,42 @@ DESCRIPTION The purpose of this client application is to notify to the VOSpace backend that data is ready to be saved somewhere. The client accepts only one command at a time. This command is mandatory. The client accepts only one (mandatory) command at a time. A list of supported commands is shown here below: cstore performs a 'cold storage' request, data will be saved on the tape library performs a 'cold storage' request, data will be saved on tape hstore performs a 'hot storage' request, data will be saved on a standard server performs a 'hot storage' request, data will be saved to disk The client also needs to know the username associated to a storage request process. The username must be the same used for accessing the transfer node. For example, if we want to perform a 'cold storage' request for the 'curban' user, we do: client@28970a09202d:~$ python3.9 dataArchiverCli.py cstore curban client@28970a09202d:~$ vos_data cstore curban Sending CSTORE request... Choose one of the following storage locations: WARNING!!! WARNING!!! WARNING!!! WARNING!!! WARNING!!! ---------------------------------------------------------------------- [*] storage_id: 1 => hostname: tape-fe.ia2.inaf.it ---------------------------------------------------------------------- Please, insert a storage id: 1 !!!!!!!!!!!!!!!!!!!!!!!!!!WARNING!!!!!!!!!!!!!!!!!!!!!!!!!!! If you confirm, all your data on the transfer node will be available in read-only mode for all the time the archiving available in read-only mode for all the time the storage process is running. WARNING!!! WARNING!!! WARNING!!! WARNING!!! WARNING!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Are you sure to proceed? [yes/no]: yes JobID: c63697eafbf711eaa44d0242ac1c0008 Store process started successfully! Storage process started successfully! client@28970a09202d:~$ After receiving this request the application will: Loading @@ -191,7 +85,12 @@ After receiving this request the application will: 4) Re-scan the folder for the last time in order to obtain the final directory structure 5) Insert information about files and folders into the Node table of the file catalog, according to the VOSpace specification 6) Move the job from the 'pending' queue to a 'ready' queue in Redis, if all the previous steps succeeded. 6) Move the job from the 'write_pending' queue to a 'Write_ready' queue in Redis, if all the previous steps succeeded. 7) Obtain the physical paths from the VOSpace paths of the nodes and copy all the data to the right destination according to the information previously inserted by the user 8) Cleanup of the '/home/curban/store/' directory (remove data and set right permissions) and database update (async_trans flag is set to 'true'). ############################################################################################################### Loading Loading
README.txt +28 −129 Original line number Diff line number Diff line Simple communication test that involes 5 docker containers: - client (container_name: client, files to run: 'vos_rest_client.py' and 'dataArchiver.py') Simple communication test that involves 5 docker containers: - client (container_name: client, commands available: 'vos_data') - server (container_name: transfer_service) - RabbitMQ (container_name: rabbitmq) - Redis (container_name: redis) Loading @@ -23,127 +23,13 @@ $ docker-compose up Once all the containers are up and running, open another shell and access the 'client' container: $ docker exec -it client /bin/bash At this point you can launch the 'vos_rest_client.py' within the 'client' container using the following syntax: $ python3.9 vos_rest_client.py QUEUE_NAME For example: $ python3.9 vos_rest_client.py start_job_queue The output should be something like this (you send and then you receive a response containing the info previously stored into the database): client@a89c0bb962f7:~$ python3.9 vos_rest_client.py start_job_queue Sending transfer request: { "jobId": "3ff92acedc9611eabf140242ac1f0007", "runId": null, "ownerId": "anonymous", "phase": "PENDING", "quote": null, "creationTime": null, "startTime": null, "endTime": null, "executionDuration": 0, "destruction": null, "parameters": null, "results": [], "errorSummary": null, "jobInfo": { "transfer": { "target": "vos://example.com!vospace/mydata1", "direction": "pullToVoSpace", "view": null, "protocol": [ { "endpoint": null, "param": [], "uri": "ia2:tape-recall" } ], "keepBytes": null, "version": null } }, "version": null } Response: { "job_id": "3ff92acedc9611eabf140242ac1f0007", "owner_id": "0000", "job_type": "pullToVoSpace", "phase": "PENDING", "start_time": null, "end_time": null, "job_info": { "transfer": { "view": null, "target": "vos://example.com!vospace/mydata1", "version": null, "protocol": [ { "uri": "ia2:tape-recall", "param": [], "endpoint": null } ], "direction": "pullToVoSpace", "keepBytes": null } }, "results": null } After processing the request, the server launches an internal thread delayed of 10 seconds which changes the state of the job from "PENDING" to "EXECUTING". You can easily verify this change by launching again the client in this other way: $ python vos_rest_client.py QUEUE_NAME JOB_ID For example, in our case: $ python vos_rest_client.py poll_job_queue 3ff92acedc9611eabf140242ac1f0007 The output should be something like this: client@a89c0bb962f7:~$ python vos_rest_client.py poll_job_queue 3ff92acedc9611eabf140242ac1f0007 Sending poll request: { "jobId": "3ff92acedc9611eabf140242ac1f0007" } Response: { "job_id": "3ff92acedc9611eabf140242ac1f0007", "owner_id": "0000", "job_type": "pullToVoSpace", "phase": "EXECUTING", "start_time": null, "end_time": null, "job_info": { "transfer": { "view": null, "target": "vos://example.com!vospace/mydata1", "version": null, "protocol": [ { "uri": "ia2:tape-recall", "param": [], "endpoint": null } ], "direction": "pullToVoSpace", "keepBytes": null } }, "results": null } --------------------------------------------------------------------------------------------------------------- Another thing you can do is to launch the 'dataArchiver.py' client. Now you can launch the 'vos_data' command. Launching the client without any argument will show you how to use it: client@28970a09202d:~$ python3.9 dataArchiverCli.py client@28970a09202d:~$ vos_data NAME dataArchiverCli.py vos_data SYNOPSYS python3.x dataArchiverCli.py COMMAND USERNAME Loading @@ -152,34 +38,42 @@ DESCRIPTION The purpose of this client application is to notify to the VOSpace backend that data is ready to be saved somewhere. The client accepts only one command at a time. This command is mandatory. The client accepts only one (mandatory) command at a time. A list of supported commands is shown here below: cstore performs a 'cold storage' request, data will be saved on the tape library performs a 'cold storage' request, data will be saved on tape hstore performs a 'hot storage' request, data will be saved on a standard server performs a 'hot storage' request, data will be saved to disk The client also needs to know the username associated to a storage request process. The username must be the same used for accessing the transfer node. For example, if we want to perform a 'cold storage' request for the 'curban' user, we do: client@28970a09202d:~$ python3.9 dataArchiverCli.py cstore curban client@28970a09202d:~$ vos_data cstore curban Sending CSTORE request... Choose one of the following storage locations: WARNING!!! WARNING!!! WARNING!!! WARNING!!! WARNING!!! ---------------------------------------------------------------------- [*] storage_id: 1 => hostname: tape-fe.ia2.inaf.it ---------------------------------------------------------------------- Please, insert a storage id: 1 !!!!!!!!!!!!!!!!!!!!!!!!!!WARNING!!!!!!!!!!!!!!!!!!!!!!!!!!! If you confirm, all your data on the transfer node will be available in read-only mode for all the time the archiving available in read-only mode for all the time the storage process is running. WARNING!!! WARNING!!! WARNING!!! WARNING!!! WARNING!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Are you sure to proceed? [yes/no]: yes JobID: c63697eafbf711eaa44d0242ac1c0008 Store process started successfully! Storage process started successfully! client@28970a09202d:~$ After receiving this request the application will: Loading @@ -191,7 +85,12 @@ After receiving this request the application will: 4) Re-scan the folder for the last time in order to obtain the final directory structure 5) Insert information about files and folders into the Node table of the file catalog, according to the VOSpace specification 6) Move the job from the 'pending' queue to a 'ready' queue in Redis, if all the previous steps succeeded. 6) Move the job from the 'write_pending' queue to a 'Write_ready' queue in Redis, if all the previous steps succeeded. 7) Obtain the physical paths from the VOSpace paths of the nodes and copy all the data to the right destination according to the information previously inserted by the user 8) Cleanup of the '/home/curban/store/' directory (remove data and set right permissions) and database update (async_trans flag is set to 'true'). ############################################################################################################### Loading
Cristiano Urban <cristiano.urban@inaf.it>Cristiano Urban <cristiano.urban@inaf.it>