Loading Dockerfile +8 −3 Original line number Diff line number Diff line FROM registry.fedoraproject.org/fedora:latest RUN dnf -y update &&\ dnf -y install httpd python3-mod_wsgi pip &&\ dnf -y install systemd httpd python3-mod_wsgi pip ps hostname &&\ dnf clean all &&\ pip install pandas tables Pyro4 pip install pandas tables scipy Pyro4 &&\ mkdir -p /var/www/wsgi-scripts /srv/sed-data COPY wsgi.conf /etc/httpd/conf.d/ COPY ./wsgi-scripts/*.py /var/www/wsgi-scripts/ COPY ./start-sed.sh /root EXPOSE 80 ENTRYPOINT /usr/sbin/httpd -DFOREGROUND CMD ["/root/start-sed.sh"] #ENTRYPOINT ["/usr/sbin/httpd","-DFOREGROUND"] #ENTRYPOINT /bin/bash Makefile 0 → 100644 +51 −0 Original line number Diff line number Diff line .PHONY: build build: docker build -t sedmods -f Dockerfile . # --name container's name (aso edded to /etc/hosts) # -d --detach run in background # -t --tty allocate pseudo-tty terminal # --rm automatically remove resources at exit .PHONY: run run: docker run --detach --tty \ --name sedmod-test \ --rm \ -v /srv/sed-data:/srv/sed-data:ro \ -p 9090:80 sedmods @echo ${PWD} # -v ${PWD}/new_dataset:/sed-data:ro \ # -v ${PWD}/wsgi-scripts:/var/www/wsgi-scripts:rw \ .PHONY: exec-bash exec-bash: docker exec -it sedmod-test bash # docker login git.ia2.inaf.it:5050 -u robert.butora # pwd szokasas regi: C*n .PHONY: publish-localy publish-localy: docker tag sedmods git.ia2.inaf.it:5050/vialactea/vlkb-sedmods/sedmods:0.1.2 docker push git.ia2.inaf.it:5050/vialactea/vlkb-sedmods/sedmods:0.1.2 docker image remove git.ia2.inaf.it:5050/vialactea/vlkb-sedmods/sedmods:0.1.2 # with podman build-podman: podman build --tag vlkb-sedmods -f ./Dockerfile run-podman: podman run -dt \ --name sedmod-test \ --rm \ -v ${PWD}/sed-data:/sed-data:z \ -v ${PWD}/wsgi-scripts:/var/www/wsgi-scripts:z \ -p 8080:80 vlkb-sedmods README.md +33 −62 Original line number Diff line number Diff line Loading @@ -11,99 +11,68 @@ container to help maintenance. ## Deploy SEDModS ### Pull repo When pulling this repo you'll end up with |-- Dockerfile (container description) |-- README.md (this text) |-- sed-data | `-- link.to.hdf5 (dummy data file, see later) |-- wsgi.conf (WSGI configuration for the http server) `-- wsgi-scripts (actual service business logic) |-- hdf_query.py |-- query-server_d.py |-- wsgid.py `-- wsgi.py ### Project content description The `Dockerfile` cotains instructions to build the container that includes: The `Dockerfile` depends on: - an httpd server (running on container's port 80) - python and WSGI packages to be able to run the service - python and WSGI packages It needs a properly configured `wsgi.conf` that will be loaded within the `http/conf.d` of the container. The base container is a _fedora:latest_ environment. Besides the instructions to build the container and the WSGI configuration, the container, to properly run, needs two further pieces, two folders: - the `sed-data` one that contains the SED Models HDF5 file - the `wsgi-scripts` that actually contains the business logic of the service itself (The actual HDF5 file is not kept in this repo, because it's too large and because it's not a wise choice to store byte-blobs in a versioning system). The `wsgi-scripts` is provided and maintained in this repo, the `sed-data` one can be anywhere else, in reach of the host system. If the location of these these pieces has to be changed, it suffices to update accordingly the run command of the container (see below). ### Build the container image To build the container image, simply run ### Build and run the container image To build the container image: podman build --tag vlkb-sedmods -f ./Dockerfile The tag can be one of your choice. It is suggested to have a dedicated user to run this in production. It is suggested to have a dedicated user to run in production. The service expects the SED models in **/srv/sed-data/sim_total.dat**. ### Run the container Once the podman image is ready and the two directories are in place, to run the container it suffices a command like Start the service: podman run -dt \ --name sedmod-test \ --rm \ -v $PWD/sed-data:/sed-data:z \ -v $PWD/sed-data:/srv/sed-data:z \ -v $PWD/wsgi-scripts:/var/www/wsgi-scripts:z \ -p 8080:80 vlkb-sedmods where the name is optional and of your choice and the left-hand side of the _:_ in the _-v_ arguments must point to the actual location of the two folders to be mounted in the container. In the example the _-v_ arguments are considered such as the command is In this example the _-v_ arguments are considered such as the command is run from the local working copy and the HDF5 SED Models file is actually within the `sed-data` folder. Also, _-p_ maps the 80 port of the container onto the 8080 port on the host server, this must be changed if the host's 8080 is already in use. ### Service(s) in the container Kubernetes manifests are available in [vlkb-k8s](https://ict.inaf.it/gitlab/ViaLactea/vlkb-k8s) project. Two flavours of the _SEDModS_ service can run with the repo content: - one that reads the HDF5 at each query - one that reads the HDF5 once and works as a _system daemon_ ### Service endpoints #### Single query service mode This mode is directly available when the cotainer is run. It uses the following code files: The service implementation is descripbed [here](README_implementation.md). wsgi.py hdf_query.py The service presents the **/searchd** endpoint. Arguments are separated by underscore and their meaning is indicated in this C++ code snipped: If you run on the host server ```cpp QString args = QString("'%1_%2_%3_%4_%5_%6_%7_0_%8_%9'") .arg(sedFitInputW) .arg(sedFitInputF) .arg(sedFitInputErrF) .arg(sedFitInputFflag) .arg(ui->distTheoLineEdit->text()) .arg(ui->prefilterLineEdit->text()) .arg(sedWeights) .arg(sedFitInputW) .arg(ui->delta_chi2_lineEdit->text()); ``` curl localhost:8080/sedsearch/?'clump_mass<10.005' > output.dat Access the service by: you should get the response in the `output.dat` file, or you can point the browser to curl localhost:8080/searchd/?arg0_arg1_...-_arg9 > output.json http://host.server:8080/sedsearch/?'clump_mass<10.005' Response is in JSON format. and see the response directly. <<<<<<< HEAD #### Daemon service mode This mode uses the code in: Loading Loading @@ -155,3 +124,5 @@ Within the container (i.e. provided in the build): - pandas - Pyro4 (deamon mode) - (py)tables ======= >>>>>>> modifs-on-vlkb README_implementation.md 0 → 100644 +74 −0 Original line number Diff line number Diff line ### Service(s) in the container Two flavours of the _SEDModS_ service can run with the repo content: - one that reads the HDF5 at each query - one that reads the HDF5 once and works as a _system daemon_ #### Single query service mode This mode is directly available when the cotainer is run. It uses the following code files: wsgi.py hdf_query.py If you run on the host server curl localhost:8080/sedsearch/?'clump_mass<10.005' > output.dat you should get the response in the `output.dat` file, or you can point the browser to http://host.server:8080/sedsearch/?'clump_mass<10.005' and see the response directly. #### Daemon service mode This mode uses the code in: wsgid.py query_server_d.py It requires a couple of processes to run before the deamonised service can work. These processes run within the container, so, after running it, one can launch them attaching to the running container with podman exec -it sedmod-test /bin/bash and within it run python -m Pyro4.naming & python query-server_d.py & After that, on can exit the shell and the daemon-based service should be reachable at http://host.server:8080/seddaemon with the same usage of the single query one. ## SED Models HDF5 file This is preserved, currently, on the INAF ICT Owncloud instance. ### Network Proxy The service can be made visible on a specific context path in the host server's http using the httpd _ProxyPass_ directive, like <Location "/sedmods"> ProxyPass "http://localhost:8080" </Location> where _/sedmods_ is an example and the _8080_ port depends on the passed parameters to the podman run command (see above). ## Dependencies On the host: - podman - httpd Within the container (i.e. provided in the build): - httpd with python3-mod\_wsgi - python 3.x - pandas - Pyro4 (deamon mode) - (py)tables start-sed.sh 0 → 100755 +16 −0 Original line number Diff line number Diff line #!/bin/bash set -eux { date env python3 -m Pyro4.naming & python3 /var/www/wsgi-scripts/query-server_d.py & date } 1> /tmp/start-sed.log 2>&1 exec /usr/sbin/httpd -DFOREGROUND Loading
Dockerfile +8 −3 Original line number Diff line number Diff line FROM registry.fedoraproject.org/fedora:latest RUN dnf -y update &&\ dnf -y install httpd python3-mod_wsgi pip &&\ dnf -y install systemd httpd python3-mod_wsgi pip ps hostname &&\ dnf clean all &&\ pip install pandas tables Pyro4 pip install pandas tables scipy Pyro4 &&\ mkdir -p /var/www/wsgi-scripts /srv/sed-data COPY wsgi.conf /etc/httpd/conf.d/ COPY ./wsgi-scripts/*.py /var/www/wsgi-scripts/ COPY ./start-sed.sh /root EXPOSE 80 ENTRYPOINT /usr/sbin/httpd -DFOREGROUND CMD ["/root/start-sed.sh"] #ENTRYPOINT ["/usr/sbin/httpd","-DFOREGROUND"] #ENTRYPOINT /bin/bash
Makefile 0 → 100644 +51 −0 Original line number Diff line number Diff line .PHONY: build build: docker build -t sedmods -f Dockerfile . # --name container's name (aso edded to /etc/hosts) # -d --detach run in background # -t --tty allocate pseudo-tty terminal # --rm automatically remove resources at exit .PHONY: run run: docker run --detach --tty \ --name sedmod-test \ --rm \ -v /srv/sed-data:/srv/sed-data:ro \ -p 9090:80 sedmods @echo ${PWD} # -v ${PWD}/new_dataset:/sed-data:ro \ # -v ${PWD}/wsgi-scripts:/var/www/wsgi-scripts:rw \ .PHONY: exec-bash exec-bash: docker exec -it sedmod-test bash # docker login git.ia2.inaf.it:5050 -u robert.butora # pwd szokasas regi: C*n .PHONY: publish-localy publish-localy: docker tag sedmods git.ia2.inaf.it:5050/vialactea/vlkb-sedmods/sedmods:0.1.2 docker push git.ia2.inaf.it:5050/vialactea/vlkb-sedmods/sedmods:0.1.2 docker image remove git.ia2.inaf.it:5050/vialactea/vlkb-sedmods/sedmods:0.1.2 # with podman build-podman: podman build --tag vlkb-sedmods -f ./Dockerfile run-podman: podman run -dt \ --name sedmod-test \ --rm \ -v ${PWD}/sed-data:/sed-data:z \ -v ${PWD}/wsgi-scripts:/var/www/wsgi-scripts:z \ -p 8080:80 vlkb-sedmods
README.md +33 −62 Original line number Diff line number Diff line Loading @@ -11,99 +11,68 @@ container to help maintenance. ## Deploy SEDModS ### Pull repo When pulling this repo you'll end up with |-- Dockerfile (container description) |-- README.md (this text) |-- sed-data | `-- link.to.hdf5 (dummy data file, see later) |-- wsgi.conf (WSGI configuration for the http server) `-- wsgi-scripts (actual service business logic) |-- hdf_query.py |-- query-server_d.py |-- wsgid.py `-- wsgi.py ### Project content description The `Dockerfile` cotains instructions to build the container that includes: The `Dockerfile` depends on: - an httpd server (running on container's port 80) - python and WSGI packages to be able to run the service - python and WSGI packages It needs a properly configured `wsgi.conf` that will be loaded within the `http/conf.d` of the container. The base container is a _fedora:latest_ environment. Besides the instructions to build the container and the WSGI configuration, the container, to properly run, needs two further pieces, two folders: - the `sed-data` one that contains the SED Models HDF5 file - the `wsgi-scripts` that actually contains the business logic of the service itself (The actual HDF5 file is not kept in this repo, because it's too large and because it's not a wise choice to store byte-blobs in a versioning system). The `wsgi-scripts` is provided and maintained in this repo, the `sed-data` one can be anywhere else, in reach of the host system. If the location of these these pieces has to be changed, it suffices to update accordingly the run command of the container (see below). ### Build the container image To build the container image, simply run ### Build and run the container image To build the container image: podman build --tag vlkb-sedmods -f ./Dockerfile The tag can be one of your choice. It is suggested to have a dedicated user to run this in production. It is suggested to have a dedicated user to run in production. The service expects the SED models in **/srv/sed-data/sim_total.dat**. ### Run the container Once the podman image is ready and the two directories are in place, to run the container it suffices a command like Start the service: podman run -dt \ --name sedmod-test \ --rm \ -v $PWD/sed-data:/sed-data:z \ -v $PWD/sed-data:/srv/sed-data:z \ -v $PWD/wsgi-scripts:/var/www/wsgi-scripts:z \ -p 8080:80 vlkb-sedmods where the name is optional and of your choice and the left-hand side of the _:_ in the _-v_ arguments must point to the actual location of the two folders to be mounted in the container. In the example the _-v_ arguments are considered such as the command is In this example the _-v_ arguments are considered such as the command is run from the local working copy and the HDF5 SED Models file is actually within the `sed-data` folder. Also, _-p_ maps the 80 port of the container onto the 8080 port on the host server, this must be changed if the host's 8080 is already in use. ### Service(s) in the container Kubernetes manifests are available in [vlkb-k8s](https://ict.inaf.it/gitlab/ViaLactea/vlkb-k8s) project. Two flavours of the _SEDModS_ service can run with the repo content: - one that reads the HDF5 at each query - one that reads the HDF5 once and works as a _system daemon_ ### Service endpoints #### Single query service mode This mode is directly available when the cotainer is run. It uses the following code files: The service implementation is descripbed [here](README_implementation.md). wsgi.py hdf_query.py The service presents the **/searchd** endpoint. Arguments are separated by underscore and their meaning is indicated in this C++ code snipped: If you run on the host server ```cpp QString args = QString("'%1_%2_%3_%4_%5_%6_%7_0_%8_%9'") .arg(sedFitInputW) .arg(sedFitInputF) .arg(sedFitInputErrF) .arg(sedFitInputFflag) .arg(ui->distTheoLineEdit->text()) .arg(ui->prefilterLineEdit->text()) .arg(sedWeights) .arg(sedFitInputW) .arg(ui->delta_chi2_lineEdit->text()); ``` curl localhost:8080/sedsearch/?'clump_mass<10.005' > output.dat Access the service by: you should get the response in the `output.dat` file, or you can point the browser to curl localhost:8080/searchd/?arg0_arg1_...-_arg9 > output.json http://host.server:8080/sedsearch/?'clump_mass<10.005' Response is in JSON format. and see the response directly. <<<<<<< HEAD #### Daemon service mode This mode uses the code in: Loading Loading @@ -155,3 +124,5 @@ Within the container (i.e. provided in the build): - pandas - Pyro4 (deamon mode) - (py)tables ======= >>>>>>> modifs-on-vlkb
README_implementation.md 0 → 100644 +74 −0 Original line number Diff line number Diff line ### Service(s) in the container Two flavours of the _SEDModS_ service can run with the repo content: - one that reads the HDF5 at each query - one that reads the HDF5 once and works as a _system daemon_ #### Single query service mode This mode is directly available when the cotainer is run. It uses the following code files: wsgi.py hdf_query.py If you run on the host server curl localhost:8080/sedsearch/?'clump_mass<10.005' > output.dat you should get the response in the `output.dat` file, or you can point the browser to http://host.server:8080/sedsearch/?'clump_mass<10.005' and see the response directly. #### Daemon service mode This mode uses the code in: wsgid.py query_server_d.py It requires a couple of processes to run before the deamonised service can work. These processes run within the container, so, after running it, one can launch them attaching to the running container with podman exec -it sedmod-test /bin/bash and within it run python -m Pyro4.naming & python query-server_d.py & After that, on can exit the shell and the daemon-based service should be reachable at http://host.server:8080/seddaemon with the same usage of the single query one. ## SED Models HDF5 file This is preserved, currently, on the INAF ICT Owncloud instance. ### Network Proxy The service can be made visible on a specific context path in the host server's http using the httpd _ProxyPass_ directive, like <Location "/sedmods"> ProxyPass "http://localhost:8080" </Location> where _/sedmods_ is an example and the _8080_ port depends on the passed parameters to the podman run command (see above). ## Dependencies On the host: - podman - httpd Within the container (i.e. provided in the build): - httpd with python3-mod\_wsgi - python 3.x - pandas - Pyro4 (deamon mode) - (py)tables
start-sed.sh 0 → 100755 +16 −0 Original line number Diff line number Diff line #!/bin/bash set -eux { date env python3 -m Pyro4.naming & python3 /var/www/wsgi-scripts/query-server_d.py & date } 1> /tmp/start-sed.log 2>&1 exec /usr/sbin/httpd -DFOREGROUND