Modified README.txt.

- server (container_name: transfer_service)

- RabbitMQ (container_name: rabbitmq)

- Redis (container_name: redis)

- File catalog (container_name: file_catalog)

- File catalog (container_name: file_catalog), now available here: 

  https://www.ict.inaf.it/gitlab/ia2/vospace-file-catalog

In addition to these containers, Sonia Zorba modified 'docker-compose.yml' by adding the REST portion.

The images used for this purpose are:

- git.ia2.inaf.it:5050/bertocco/vospace-oats

- git.ia2.inaf.it:5050/ia2/vospace-file-service

- git.ia2.inaf.it:5050/ia2/vospace-ui

The web interface is available on your browser at http://localhost:8080/ when all the containers are up and 

running (read the section here below).

###############################################################################################################

$ docker exec -it client /bin/bash

At this point you can launch the 'vos_rest_client.py' within the 'client' container using the following syntax:

$ python vos_rest_client.py QUEUE_NAME

$ python3.9 vos_rest_client.py QUEUE_NAME

For example:

$ python vos_rest_client.py start_job_queue

$ python3.9 vos_rest_client.py start_job_queue

The output should be something like this:

     client@a89c0bb962f7:~$ python vos_rest_client.py start_job_queue

     client@a89c0bb962f7:~$ python3.9 vos_rest_client.py start_job_queue

     Sending transfer request:

     {

        "transfer": {

       The username must be the same used for accessing the transfer node.

For example, if we want to perform a 'cold storage' request for a the 'transfer_service' user, we do:

client@28970a09202d:~$ python3 dataArchiverCli.py cstore transfer_service

For example, if we want to perform a 'cold storage' request for a the 'curban' user, we do:

client@28970a09202d:~$ python3 dataArchiverCli.py cstore curban

Sending CSTORE request...

JobID: c63697eafbf711eaa44d0242ac1c0008

Store process started successfully!

After receiving this request the application will:

1) Create a job object, insert it into the job table of the file catalog database and push a copy into a 

   'pending' queue stored in Redis for scheduling purposes

2) Scan the content of '/home/curban/store/' to find crowded 'leaf' dirs and substitute these ones wit an

   uncompressed tar according to some constraints defined in the global configuration file

3) Re-scan the folder, move the content into a temporary folder if needed and perform recursive MD5 checksum

4) Re-scan the folder for the last time in order to obtain the final directory structure

5) Insert information about files and folders into the Node table of the file catalog, according to the VOSpace

   specification

6) Move the job from the 'pending' queue to a 'ready' queue in Redis, if all the previous steps succeeded.

###############################################################################################################

You can access the rabbitmq web interface via browser:

You can access the redis server from the 'client' container:

    1) Use redis-cli to connect to redis:

    $ redis-cli -h redis -n DB_INDEX

    NOTE: DB_INDEX is a non-negative number representing the db to work on:

    - 0: jobs that retrieve data (pullFromVOSpace) 

    - 1: jobs that store data (push)

    - 2: scheduling queues

    2) You can now perform a query based on the job ID, for example show the job object info stored on db = 1:

    get JOB_ID (if we consider the last example: "get  c63697eafbf711eaa44d0242ac1c0008")

    3) This will return all the information regarding the job

    $ redis-cli -h redis

    2) You can obtain some info about the jobs by searching them on the 'pending' and 'ready' queues 

       using the lrange command. For example, a few seconds after launching three jobs with 'dataArchiverCli.py',

       you should be able to see a output similar to the following one:

    redis:6379[2]> lrange ready 0 5

    1) "{\"jobId\": \"56577c8645da11ebbbfe356e379843eb\", \"jobType\": \"other\", \"ownerId\": \"2386\", \"phase\": \"PENDING\", 

    \"quote\": null, \"startTime\": null, \"endTime\": null, \"executionDuration\": null, \"destruction\": null, \"parameters\": null, 

    \"results\": null, \"jobInfo\": {\"requestType\": \"CSTORE\", \"userName\": \"szorba\"}}"

    2) "{\"jobId\": \"53d2f2a545da11ebb7bd356e379843eb\", \"jobType\": \"other\", \"ownerId\": \"2048\", \"phase\": \"PENDING\", 

    \"quote\": null, \"startTime\": null, \"endTime\": null, \"executionDuration\": null, \"destruction\": null, \"parameters\": null, 

    \"results\": null, \"jobInfo\": {\"requestType\": \"CSTORE\", \"userName\": \"sbertocco\"}}"

    3) "{\"jobId\": \"502afdca45da11eb9676356e379843eb\", \"jobType\": \"other\", \"ownerId\": \"3354\", \"phase\": \"PENDING\", 

    \"quote\": null, \"startTime\": null, \"endTime\": null, \"executionDuration\": null, \"destruction\": null, \"parameters\": null, 

    \"results\": null, \"jobInfo\": {\"requestType\": \"CSTORE\", \"userName\": \"curban\"}}"lrange ready 0 5

You can access the file catalog from the 'client' container:

    1) Access the db via psql client:

    $ psql -h file_catalog -U postgres -d vospace_testdb

    $ psql -h file_catalog -d vospace_testdb -U postgres

    2) You can now perform a query, for example show all the tuples of the Node table displaying some fields:

    vospace_testdb=# SELECT node_id, parent_path, path, name, type, owner_id, creator_id, content_MD5 FROM Node;

The default output of the query after the container initialization should be something like this:

vospace_testdb=# SELECT node_id, parent_path, path, name, type, owner_id, creator_id, content_MD5 FROM Node;

 node_id | parent_path |   path    |    name    |   type    | owner_id | creator_id | content_md5 

 ---------+-------------+------+--------+-----------+----------+------------+--------------

        1 |             |      |        | container | 3354     | 3354       |             

        2 |             | 2    | home   | container | 3354     | 3354       |             

        3 | 2           | 2.3  | curban | container | 3354     | 3354       |             

 (3 rows)

 Now, open a shell on the 'transfer_service' container and launch 'store_preprocessor.py', with:

 $ docker exec -it transfer_service /bin/bash

 $ python3 store_preprocessor.py 

 The new output of the query will be the following, because a new file called 'foo.txt' was placed in /home/curban/store:

---------+-------------+-----------+------------+-----------+----------+------------+-------------

       1 |             |           |            | container | 0        | 0          | 

       2 |             | 2         | curban     | container | 3354     | 3354       | 

       3 | 2           | 2.3       | store      | container | 3354     | 3354       | 

       4 |             | 4         | sbertocco  | container | 2048     | 2048       | 

       5 | 4           | 4.5       | store      | container | 2048     | 2048       | 

       6 |             | 6         | szorba     | container | 2386     | 2386       | 

       7 | 6           | 6.7       | store      | container | 2386     | 2386       | 

       8 |             | 8         | test       | container | 2386     | 2386       | 

       9 | 8           | 8.9       | f1         | container | 2386     | 2386       | 

      10 | 8.9         | 8.9.10    | f2_renamed | container | 2386     | 2386       | 

      11 | 8.9.10      | 8.9.10.11 | f3         | container | 2386     | 2386       | 

(11 rows)

A few seconds after launching three jobs with 'dataArchiverCli.py', the database will be ppulated and launching the previous

SQL query you will be able an output like the one here below:       

vospace_testdb=# SELECT node_id, parent_path, path, name, type, owner_id, creator_id, content_MD5 FROM Node;

 node_id | parent_path |     path     |        name         |   type    | owner_id | creator_id |           content_md5            

 ---------+-------------+-----------+---------------------+-----------+----------+------------+----------------------------------

        1 |             |           |                     | container | 3354     | 3354       | 

        2 |             | 2         | home                | container | 3354     | 3354       | 

        3 | 2           | 2.3       | curban              | container | 3354     | 3354       | 

        4 | 2.3         | 2.3.4     | store               | container | 3354     | 3354       | 

        5 | 2.3.4       | 2.3.4.5   | 2020_12_02-11_31_56 | container | 3354     | 3354       | 

        6 | 2.3.4.5     | 2.3.4.5.6 | foo.txt             | data      | 3354     | 3354       | d41d8cd98f00b204e9800998ecf8427e

 (6 rows)

---------+-------------+--------------+---------------------+-----------+----------+------------+----------------------------------

       1 |             |              |                     | container | 0        | 0          | 

       2 |             | 2            | curban              | container | 3354     | 3354       | 

       3 | 2           | 2.3          | store               | container | 3354     | 3354       | 

       4 |             | 4            | sbertocco           | container | 2048     | 2048       | 

       5 | 4           | 4.5          | store               | container | 2048     | 2048       | 

       6 |             | 6            | szorba              | container | 2386     | 2386       | 

       7 | 6           | 6.7          | store               | container | 2386     | 2386       | 

       8 |             | 8            | test                | container | 2386     | 2386       | 

       9 | 8           | 8.9          | f1                  | container | 2386     | 2386       | 

      10 | 8.9         | 8.9.10       | f2_renamed          | container | 2386     | 2386       | 

      11 | 8.9.10      | 8.9.10.11    | f3                  | container | 2386     | 2386       | 

      12 | 2.3         | 2.3.12       | 2020_12_24-11_22_47 | container | 3354     | 3354       | 

      13 | 2.3.12      | 2.3.12.13    | mydir               | container | 3354     | 3354       | 

      14 | 2.3.12      | 2.3.12.14    | foo2.txt            | data      | 3354     | 3354       | e07f37a6bfe96ad66e408380a5e3a899

      15 | 2.3.12.13   | 2.3.12.13.15 | another_foo2.txt    | data      | 3354     | 3354       | e048e5108d71191158b50052d531b0ca

      16 | 4.5         | 4.5.16       | 2020_12_24-11_23_02 | container | 2048     | 2048       | 

      17 | 4.5.16      | 4.5.16.17    | foo4.txt            | data      | 2048     | 2048       | d41d8cd98f00b204e9800998ecf8427e

      18 | 6.7         | 6.7.18       | 2020_12_24-11_23_17 | container | 2386     | 2386       | 

      19 | 6.7.18      | 6.7.18.19    | foo5.txt            | data      | 2386     | 2386       | d41d8cd98f00b204e9800998ecf8427e

 From the file catalog database you can also obtain information about jobs, according to the UWS specification. 

 Just try the following query:

 vospace_testdb=# SELECT * FROM job;

 ###############################################################################################################

 To stop the whole environment:

 Stop the whole environment:

 $ docker-compose down

 Cleanup: