Skip to content
Commits on Source (46)
Hide whitespace changes
Inline Side-by-side
README.md
View file @ 7c962dfb
......@@ -43,30 +43,40 @@ Clean
### Configuration
Example Webapp configuraion:
Webapp service configuraion parameters and their defaults:
- SAFEMODE=False
- DJANGO_DEV_SERVER=True
- DJANGO_DEBUG=True
- SAFEMODE=false
- DJANGO_DEV_SERVER=true
- DJANGO_DEBUG=true
- DJANGO_LOG_LEVEL=ERROR
- ROSETTA_LOG_LEVEL=ERROR
- ROSETTA_TUNNEL_HOST=localhost # Not http or https
- ROSETTA_WEBAPP_HOST=
- ROSETTA_HOST=localhost
- ROSETTA_TUNNEL_HOST=localhost
- ROSETTA_WEBAPP_HOST=""
- ROSETTA_WEBAPP_PORT=8080
- LOCAL_DOCKER_REGISTRY_HOST=
- LOCAL_DOCKER_REGISTRY_PORT=5000
- ROSETTA_REGISTRY_HOST=proxy
- ROSETTA_REGISTRY_PORT=5000
- DJANGO_EMAIL_SERVICE=Sendgrid
- DJANGO_EMAIL_APIKEY=
- DJANGO_EMAIL_APIKEY=""
- DJANGO_EMAIL_FROM="Rosetta <notifications@rosetta.local>"
- DJANGO_PUBLIC_HTTP_HOST=http://localhost # Public facing, with http or https
- INVITATION_CODE=""
- OIDC_RP_CLIENT_ID=""
- OIDC_RP_CLIENT_SECRET=""
- OIDC_OP_AUTHORIZATION_ENDPOINT=""
- OIDC_OP_TOKEN_ENDPOINT=""
- OIDC_OP_JWKS_ENDPOINT=""
- DISABLE_LOCAL_AUTH=False
In Rosetta, only power users can:
- DISABLE_LOCAL_AUTH=false
Notes:
- `ROSETTA_TUNNEL_HOST` must not include http:// or https://
- `ROSETTA_REGISTRY_HOST` should be set to the same value as `ROSETTA_HOST` for production scenarios, in order to be secured unders SSL. The `standaloneworker` is configured to treat the following hosts (and ports) as unsecure registies, where it can connect without a valid certificate: `proxy:5000`,`dregistry:5000` and `rosetta.platform:5000`.
- `ROSETTA_WEBAPP_HOST` is used for let the agent know where to connect, and it is differentiated from `ROSETTA_HOST` as it can be on an internal Docker network. It is indeed defaulted to the `webapp` container IP address.
### User types
In Rosetta there are two user types: standard users and power users. Their type is set in their user profile, and only power users can:
- set custom task passwords
- choose task access methods other than the default one (bypassing HTTP proxy + auth)
......@@ -121,6 +131,15 @@ Run Web App unit tests (with Rosetta running)
$ rosetta/logs webapp server
$ rosetta/test
### Computing resources requirements
Ensure that computing resource have:
- a container engine or wms available (of course);
- Python installed and callable with the "python" executable or the agent will fail;
- Bash as default shell for ssh-based computign resources.
## Known issues
......
docker-compose-dev.yml
View file @ 7c962dfb
......@@ -60,15 +60,15 @@ services:
- ROSETTA_LOG_LEVEL=DEBUG
#- ROSETTA_WEBAPP_HOST=localhost # Internal, for the agent
#- ROSETTA_WEBAPP_PORT=8080 # Internal, for the agent
#- LOCAL_DOCKER_REGISTRY_HOST=
#- LOCAL_DOCKER_REGISTRY_PORT=5000
#- ROSETTA_REGISTRY_HOST=
#- ROSETTA_REGISTRY_PORT=5000
#- DJANGO_EMAIL_APIKEY=""
#- DJANGO_EMAIL_FROM="Rosetta Platform <notifications@rosetta.platform>"
#- DJANGO_SECRET_KEY=""
- TASK_PROXY_HOST=localhost
- TASK_TUNNEL_HOST=localhost
- ROSETTA_HOST=localhost
- REGISTRY_HOST=proxy # Use same value as ROSETTA_HOST for production or to use "real" computing resurces
- REGISTRY_HOST=proxy:5000 # Use same value as ROSETTA_HOST for production or to use "real" computing resurces
ports:
- "8080:8080"
- "7000-7020:7000-7020"
......
services/base/Dockerfile
View file @ 7c962dfb
......@@ -25,7 +25,7 @@ RUN apt-get install net-tools iproute2 iputils-ping -y
#------------------------
# Scienceuser user
# Rosetta user
#------------------------
# Add group. We chose GID 65527 to try avoiding conflicts.
......@@ -90,47 +90,6 @@ RUN mkdir /prestartup
COPY prestartup.py /
#----------------------
# Singularity
#----------------------
# Dependencies
RUN apt-get update && apt-get install -y \
build-essential \
libssl-dev \
uuid-dev \
libgpgme11-dev \
squashfs-tools \
libseccomp-dev \
pkg-config \
cryptsetup-bin \
wget
# Install GO
RUN cd /tmp && wget https://dl.google.com/go/go1.11.linux-amd64.tar.gz
RUN cd /tmp && tar -zxf go1.11.linux-amd64.tar.gz && mv go /usr/local
ENV GOROOT=/usr/local/go
ENV GOPATH=/root/go
ENV PATH=$PATH:/usr/local/go/bin:$GOPATH/bin
COPY singularity-3.4.1.tar.gz /tmp
# Install Singularity
RUN mkdir -p /usr/local/var/singularity/mnt && \
mkdir -p $GOPATH/src/github.com/sylabs && \
cd $GOPATH/src/github.com/sylabs && \
mv /tmp/singularity-3.4.1.tar.gz ./ && \
tar -xzvf singularity-3.4.1.tar.gz
RUN cd $GOPATH/src/github.com/sylabs/singularity && \
./mconfig -p /usr/local && \
make -C builddir && \
make -C builddir install
# Build test image
RUN mkdir /singularity_images && chmod 777 /singularity_images
COPY testimage.def /singularity_images/testimage.def
RUN singularity build /singularity_images/testimage.simg /singularity_images/testimage.def
#----------------------
# Entrypoint
......
services/dregistry/Dockerfile
View file @ 7c962dfb
FROM registry:2
RUN set -ex && apk --no-cache add sudo bash
#------------------------
# Rosetta user
#------------------------
# Add group. We chose GID 65527 to try avoiding conflicts.
RUN addgroup -g 65527 rosetta
# Add user. We chose UID 65527 to try avoiding conflicts.
RUN adduser rosetta -D -h /rosetta -u 65527 -G rosetta -s /bin/bash
# Add rosetta user to sudoers
RUN adduser rosetta wheel
# Passwordless sudo
RUN sed -e 's;^# \(%wheel.*NOPASSWD.*\);\1;g' -i /etc/sudoers
\ No newline at end of file
services/proxy/proxy-global.conf
View file @ 7c962dfb
......@@ -13,6 +13,7 @@
ProxyPass / http://webapp:8080/
ProxyPassReverse / http://webapp:8080/
AllowEncodedSlashes NoDecode
</VirtualHost>
......@@ -32,9 +33,9 @@
ServerName ${ROSETTA_HOST}
ProxyPass / http://webapp:8080/
ProxyPassReverse / http://webapp:8080/
AllowEncodedSlashes NoDecode
SSLEngine on
SSLCertificateFile /root/certificates/rosetta_platform/rosetta_platform.crt
SSLCertificateKeyFile /root/certificates/rosetta_platform/rosetta_platform.key
SSLCACertificateFile /root/certificates/rosetta_platform/rosetta_platform.ca-bundle
......
services/slurmbase/Dockerfile
View file @ 7c962dfb
FROM rosetta/base
MAINTAINER Stefano Alberto Russo <stefano.russo@gmail.com>
#----------------------
# Singularity
#----------------------
# Dependencies
RUN apt-get update && apt-get install -y \
build-essential \
libssl-dev \
uuid-dev \
libgpgme11-dev \
squashfs-tools \
libseccomp-dev \
pkg-config \
cryptsetup-bin \
wget
# Install GO
RUN cd /tmp && wget https://dl.google.com/go/go1.11.linux-amd64.tar.gz
RUN cd /tmp && tar -zxf go1.11.linux-amd64.tar.gz && mv go /usr/local
ENV GOROOT=/usr/local/go
ENV GOPATH=/root/go
ENV PATH=$PATH:/usr/local/go/bin:$GOPATH/bin
COPY singularity-3.4.1.tar.gz /tmp
# Install Singularity
RUN mkdir -p /usr/local/var/singularity/mnt && \
mkdir -p $GOPATH/src/github.com/sylabs && \
cd $GOPATH/src/github.com/sylabs && \
mv /tmp/singularity-3.4.1.tar.gz ./ && \
tar -xzvf singularity-3.4.1.tar.gz
RUN cd $GOPATH/src/github.com/sylabs/singularity && \
./mconfig -p /usr/local && \
make -C builddir && \
make -C builddir install
# Build test image
RUN mkdir /singularity_images && chmod 777 /singularity_images
COPY testimage.def /singularity_images/testimage.def
RUN singularity build /singularity_images/testimage.simg /singularity_images/testimage.def
#----------------------
# Slurm
#----------------------
# Install Slurm
RUN apt-get -y install slurm-wlm
......@@ -23,6 +69,12 @@ COPY slurm.conf /etc/slurm-llnl/slurm.conf
RUN ln -s /var/lib/slurm-llnl /var/lib/slurm-wlm
RUN ln -s /var/log/slurm-llnl /var/log/slurm-wlm
#----------------------
# Test user and
# prestartup
#----------------------
# Add testuser user
RUN useradd testuser
RUN mkdir -p /home/testuser/.ssh
......
services/standaloneworker/Dockerfile
View file @ 7c962dfb
FROM quay.io/podman/stable:v3.2.3
#RUN dnf repolist
#RUN dnf update --refresh
# This is necessary due to some base image permission errors.
RUN chown -R podman:podman /home/podman
# Change user
RUN usermod -l testuser podman
RUN usermod -d /home/testuser testuser
RUN ln -s /home/podman /home/testuser
RUN groupmod -n testuser podman
# Change user, from podman to rosetta
RUN usermod -l rosetta podman
RUN usermod -d /rosetta rosetta
RUN ln -s /home/podman /rosetta
RUN groupmod -n rosetta podman
# Replace uid/gid mapping from podman to testuser user
# Replace uid/gid mapping from podman to rosetta user
COPY subuid /etc/subuid
COPY subgid /etc/subgid
#RUN dnf repolist
#RUN dnf update --refresh
RUN dnf install -y docker singularity openssh-server
RUN ssh-keygen -A
# Authorized keys for rosetta
RUN mkdir /rosetta/.ssh
COPY keys/id_rsa.pub /rosetta/.ssh/authorized_keys
# Add rosetta user to sudoers
RUN usermod -aG wheel rosetta
# Passwordless sudo (for everyone)
RUN sed -e 's;^# \(%wheel.*NOPASSWD.*\);\1;g' -i /etc/sudoers
# Add testuser user
RUN groupadd -g 1001 testuser
RUN useradd testuser -d /home/testuser -u 1001 -g 1001 -m -s /bin/bash
# Authorized keys for testuser
RUN mkdir /home/testuser/.ssh
COPY keys/id_rsa.pub /home/testuser/.ssh/authorized_keys
RUN dnf install -y python wget
# Install iputils (ping)
RUN dnf install -y iputils
# Install Docker, Singularity, various utilities including iputils (for ping) and openssh-clients (for scp)
RUN dnf install -y docker singularity openssh-server python wget iputils openssh-clients
# Copy registries.conf to allow insecure access to dregistry
COPY registries.conf /etc/containers/registries.conf
# Generate host keys
RUN ssh-keygen -A
#----------------------
# Entrypoint
#----------------------
# Copy registries.conf to allow insecure access to internal/dev registries
COPY registries.conf /etc/containers/registries.conf
# Copy entrypoint
COPY entrypoint.sh /
......
services/standaloneworker/entrypoint.sh
View file @ 7c962dfb
......@@ -12,6 +12,20 @@ chmod 777 /dev/net/tun
#PROXY_IP=$(ping proxy -c1 | head -n1 | cut -d '(' -f2 | cut -d')' -f1)
#echo "$PROXY_IP rosetta.platform" >> /etc/hosts
# Create shared data directories
mkdir -p /shared/scratch
chmod 777 /shared/scratch
mkdir -p /shared/data/shared
chmod 777 /shared/data/shared
mkdir -p /shared/data/users
chown rosetta:rosetta /shared/data/users/
mkdir -p /shared/data/users/testuser
chown testuser:testuser /shared/data/users/testuser
#---------------------
# Entrypoint command
#---------------------
......
services/standaloneworker/registries.conf
View file @ 7c962dfb
......@@ -85,6 +85,10 @@ short-name-mode="enforcing"
location = "dregistry:5000"
insecure = true
[[registry]]
location = "proxy:5000"
insecure = true
[[registry]]
location = "rosetta.platform:5000"
insecure = true
......
services/standaloneworker/subgid
View file @ 7c962dfb
testuser:10000:5000
\ No newline at end of file
rosetta:10000:5000
\ No newline at end of file
services/standaloneworker/subuid
View file @ 7c962dfb
testuser:10000:5000
\ No newline at end of file
rosetta:10000:5000
\ No newline at end of file
services/webapp/code/rosetta/auth.py
View file @ 7c962dfb
from mozilla_django_oidc.auth import OIDCAuthenticationBackend
from mozilla_django_oidc.views import OIDCAuthenticationCallbackView
from .core_app.utils import finalize_user_creation
from django.http import HttpResponseRedirect
# Setup logging
import logging
......@@ -18,9 +20,27 @@ class RosettaOIDCAuthenticationBackend(OIDCAuthenticationBackend):
return user
def get_userinfo(self, access_token, id_token, payload):
# Payload must contain the "email" key
return payload
class RosettaOIDCAuthenticationCallbackView(OIDCAuthenticationCallbackView):
def login_success(self):
# Call parent login_success but do not return
super(RosettaOIDCAuthenticationCallbackView, self).login_success()
logger.debug('Trying to get cookie-based post login redirect')
post_login_page = self.request.COOKIES.get('post_login_redirect')
if post_login_page:
logger.debug('Got "%s" and redirecting', post_login_page )
response = HttpResponseRedirect(post_login_page)
response.delete_cookie('post_login_redirect')
return response
else:
logger.debug('No cookie-based post login redirect found, redirecting to "%s"', self.success_url)
return HttpResponseRedirect(self.success_url)
services/webapp/code/rosetta/core_app/api.py
View file @ 7c962dfb
......@@ -11,7 +11,7 @@ from django.conf import settings
from rest_framework.response import Response
from rest_framework import status, serializers, viewsets
from rest_framework.views import APIView
from .utils import format_exception, send_email, os_shell, now_t, get_ssh_access_mode_credentials
from .utils import format_exception, send_email, os_shell, now_t, get_ssh_access_mode_credentials, get_or_create_container_from_repository
from .models import Profile, Task, TaskStatuses, Computing, Storage, KeyPair
from .exceptions import ConsistencyException
import json
......@@ -268,6 +268,8 @@ logger.info('Reporting for task uuid: "{}"'.format(task_uuid))
# Get IP
ip = socket.gethostbyname(hostname)
if ip == '127.0.1.1':
ip = socket.gethostbyname(hostname+'.local')
logger.info(' - ip: "{}"'.format(ip))
# Get port
......@@ -321,7 +323,7 @@ print(port)
return HttpResponse('Port not valid (got "{}")'.format(task_interface_port))
# Set fields
logger.info('Setting task "{}" to ip "{}" and port "{}"'.format(task.uuid, task_interface_ip, task_interface_port))
logger.info('Agent API setting task "{}" to ip "{}" and port "{}"'.format(task.uuid, task_interface_ip, task_interface_port))
task.status = TaskStatuses.running
task.interface_ip = task_interface_ip
......@@ -349,7 +351,7 @@ print(port)
# Notify the user that the task called back home if using a WMS
if task.computing.wms:
if settings.DJANGO_EMAIL_APIKEY:
logger.info('Sending task ready mail notification to "{}"'.format(task.user.email))
logger.info('Agent API sending task ready mail notification to "{}"'.format(task.user.email))
mail_subject = 'Your Task "{}" is now starting up'.format(task.container.name)
mail_text = 'Hello,\n\nyour Task "{}" on {} is now starting up. Check logs or connect here: https://{}/tasks/?uuid={}\n\nThe Rosetta notifications bot.'.format(task.container.name, task.computing, settings.ROSETTA_HOST, task.uuid)
try:
......@@ -380,10 +382,10 @@ class FileManagerAPI(PrivateGETAPI, PrivatePOSTAPI):
authentication_classes = (CsrfExemptSessionAuthentication, BasicAuthentication)
def scp_command(self, source, dest, user, computing, mode='get'):
def prepare_scp_command(self, source, dest, user, computing, mode='get'):
# Prepare paths for scp. They have been already made shell-ready, but we need to triple-escape
# spaces on remote source or destination: My\ Folder mut become My\\\ Folder.
# spaces on remote source or destination: My\ Folder must become My\\\ Folder.
if mode=='get':
source = source.replace('\ ', '\\\\\\ ')
......@@ -404,14 +406,53 @@ class FileManagerAPI(PrivateGETAPI, PrivatePOSTAPI):
return command
def ssh_command(self, command, user, computing):
# Get credentials
computing_user, computing_host, computing_keys = get_ssh_access_mode_credentials(computing, user)
# Command
command = 'ssh -o LogLevel=ERROR -i {} -4 -o StrictHostKeyChecking=no {}@{} "{}"'.format(computing_keys.private_key_file, computing_user, computing_host, command)
def prepare_sh_command(self, command, user, storage):
if storage.access_mode == 'ssh+cli':
if storage.access_through_computing:
# Get credentials
computing_user, computing_host, computing_keys = get_ssh_access_mode_credentials(storage.computing, user)
# Command
command = 'ssh -o LogLevel=ERROR -i {} -4 -o StrictHostKeyChecking=no {}@{} "{}"'.format(computing_keys.private_key_file, computing_user, computing_host, command)
else:
raise NotImplementedError('Not accessing through computing is not implemented for storage type "{}"'.format(storage.type))
elif storage.access_mode == 'cli':
try:
as_user = storage.conf['as_user']
# Is "as_user" a UID?
try:
uid = int(as_user)
except:
pass
else:
# What is the user for this uid?
out = os_shell('sudo getent passwd "1000" | cut -d: -f1', capture=True)
if out.exit_code != 0:
raise Exception(out.sterr)
else:
if not out.stdout.strip():
# No user found, create it
os_shell('sudo groupadd -g {0} group_{0}'.format(uid), capture=True)
if out.exit_code != 0:
raise Exception(out.sterr)
os_shell('sudo useradd user_{0} -d /home_{0} -u {0} -g {0} -m -s /bin/bash'.format(uid), capture=True)
if out.exit_code != 0:
raise Exception(out.sterr)
as_user = 'user_' + str(uid)
else:
as_user = out.stdout.strip()
except (KeyError, TypeError):
as_user = None
if as_user:
command = 'sudo -i -u {} /bin/bash -c "{}"'.format(as_user, command)
else:
command = '/bin/bash -c "{}"'.format(command)
else:
raise NotImplementedError('Access mode "{}" not implemented for storage type "{}"'.format(storage.access_mode, storage.type))
return command
@staticmethod
......@@ -428,11 +469,13 @@ class FileManagerAPI(PrivateGETAPI, PrivatePOSTAPI):
return path
@staticmethod
def sanitize_and_prepare_shell_path(path, storage, user):
path = path.replace(' ', '\ ')
cleaner = re.compile('(?:\\\)+')
path = re.sub(cleaner,r"\\",path)
def sanitize_and_prepare_shell_path(path, user, storage, escapes=True):
if escapes:
path = path.replace(' ', '\ ')
cleaner = re.compile('(?:\\\)+')
path = re.sub(cleaner,r"\\",path)
# Prepare the base path (expand it with variables substitution)
base_path_expanded = storage.base_path
if '$SSH_USER' in base_path_expanded:
......@@ -441,7 +484,7 @@ class FileManagerAPI(PrivateGETAPI, PrivatePOSTAPI):
if computing.auth_mode == 'user_keys':
computing_user = user.profile.get_extra_conf('computing_user', storage.computing)
if not computing_user:
raise Exception('Computing resource \'{}\' user is not configured'.format(storage.computing.name))
raise ValueError('No \'computing_user\' parameter found for computing resource \'{}\' in user profile'.format(storage.computing.name))
base_path_expanded = base_path_expanded.replace('$SSH_USER', computing_user)
else:
base_path_expanded = base_path_expanded.replace('$SSH_USER', computing.conf.get('user'))
......@@ -449,7 +492,7 @@ class FileManagerAPI(PrivateGETAPI, PrivatePOSTAPI):
else:
raise NotImplementedError('Accessing a storage with ssh+cli without going through its computing resource is not implemented')
if '$USER' in base_path_expanded:
base_path_expanded = base_path_expanded.replace('$USER', user.name)
base_path_expanded = base_path_expanded.replace('$USER', user.username)
# If the path is not starting with the base path, do it
if not path.startswith(base_path_expanded):
......@@ -503,200 +546,252 @@ class FileManagerAPI(PrivateGETAPI, PrivatePOSTAPI):
# Data container
data = []
shell_path = self.sanitize_and_prepare_shell_path(path, storage, user)
# Prepare command
# https://askubuntu.com/questions/1116634/ls-command-show-time-only-in-iso-format
# https://www.howtogeek.com/451022/how-to-use-the-stat-command-on-linux/
command = self.ssh_command('cd {} && stat --printf=\'%F/%s/%Y/%n\\n\' * .*'.format(shell_path), user, storage.computing)
# Execute_command
out = os_shell(command, capture=True)
if out.exit_code != 0:
if storage.type == 'generic_posix':
shell_path = self.sanitize_and_prepare_shell_path(path, user, storage)
# Did we just get a "cannot stat - No such file or directory error?
if 'No such file or directory' in out.stderr:
pass
else:
raise Exception(out.stderr)
# Log
#logger.debug('Shell exec output: "{}"'.format(out))
out_lines = out.stdout.split('\n')
for line in out_lines:
# Prepare command
# https://askubuntu.com/questions/1116634/ls-command-show-time-only-in-iso-format
# https://www.howtogeek.com/451022/how-to-use-the-stat-command-on-linux/
# Example line: directory/My folder/68/1617030350
command = self.prepare_sh_command('cd {} && stat --printf=\'%F/%s/%Y/%n\\n\' * .*'.format(shell_path), user, storage)
# Set name
line_pieces = line.split('/')
type = line_pieces[0]
size = line_pieces[1]
timestamp = line_pieces[2]
name = line_pieces[3]
# Define and clean listing path:
listing_path = '/{}/{}/{}/'.format(storage.id, path, name)
listing_path = self.clean_path(listing_path)
# File or directory?
if type == 'directory':
if name not in ['.', '..']:
# Execute_command
out = os_shell(command, capture=True)
if out.exit_code != 0:
# Did we just get a "cannot stat - No such file or directory" (bash) or a "can't cd to" (sh) error?
if 'No such file or directory' in out.stderr or 'can\'t cd to' in out.stderr :
# Create the folder if this was the root for the user (storage base path)
# Note: if the folder is completely empty, this gets execute as well.
# TODO: fix me (e.g. check for "cannot stat" or similar)
if path == '/':
self.mkdir(self.sanitize_and_prepare_shell_path('/', user, storage), user, storage, force=True)
# Return (empty) data
return data
else:
raise Exception(out.stderr)
# Log
#logger.debug('Shell exec output: "{}"'.format(out))
out_lines = out.stdout.split('\n')
for line in out_lines:
# Example line: directory/My folder/68/1617030350
# Set name
line_pieces = line.split('/')
type = line_pieces[0]
size = line_pieces[1]
timestamp = line_pieces[2]
name = line_pieces[3]
# Define and clean listing path:
listing_path = '/{}/{}/{}/'.format(storage.id, path, name)
listing_path = self.clean_path(listing_path)
# File or directory?
if type == 'directory':
if name not in ['.', '..']:
data.append({
'id': listing_path,
'type': 'folder',
'attributes':{
'modified': timestamp,
'name': name,
'readable': 1,
'writable': 1,
'path': listing_path
}
})
else:
data.append({
'id': listing_path,
'type': 'folder',
'id': listing_path[:-1], # Remove trailing slash
'type': 'file',
'attributes':{
'modified': timestamp,
'name': name,
'readable': 1,
'writable': 1,
'path': listing_path
"size": size,
'path': listing_path[:-1] # Remove trailing slash
}
})
else:
data.append({
'id': listing_path[:-1], # Remove trailing slash
'type': 'file',
'attributes':{
'modified': timestamp,
'name': name,
'readable': 1,
'writable': 1,
"size": size,
'path': listing_path[:-1] # Remove trailing slash
}
})
})
else:
raise NotImplementedError('Storage type "{}" not implemented'.format(storage.type))
return data
def stat(self, path, user, storage):
path = self.sanitize_and_prepare_shell_path(path, storage, user)
# Prepare command. See the ls function above for some more info
command = self.ssh_command('stat --printf=\'%F/%s/%Y/%n\\n\' {}'.format(path), user, storage.computing)
# Execute_command
out = os_shell(command, capture=True)
if out.exit_code != 0:
# Data container
data = []
if storage.type == 'generic_posix':
shell_path = self.sanitize_and_prepare_shell_path(path, user, storage)
# Did we just get a "cannot stat - No such file or directory error?
if 'No such file or directory' in out.stderr:
pass
else:
raise Exception(out.stderr)
# Log
#logger.debug('Shell exec output: "{}"'.format(out))
# Prepare command. See the ls function above for some more info
command = self.prepare_sh_command('stat --printf=\'%F/%s/%Y/%n\\n\' {}'.format(shell_path), user, storage)
# Execute_command
out = os_shell(command, capture=True)
if out.exit_code != 0:
# Did we just get a "cannot stat - No such file or directory error?
if 'No such file or directory' in out.stderr:
pass
else:
raise Exception(out.stderr)
# Log
#logger.debug('Shell exec output: "{}"'.format(out))
out_lines = out.stdout.split('\n')
if len(out_lines) > 1:
raise Exception('Internal error on stat: more than one ouput line')
out_line = out_lines[0]
out_lines = out.stdout.split('\n')
if len(out_lines) > 1:
raise Exception('Internal error on stat: more than one ouput line')
out_line = out_lines[0]
# Example output line: directory:My folder:68/1617030350
# In this context, we also might get the following output:
# directory/68/1617030350//My folder/
# ..so, use the clean path to remove all extra slashes.
# The only uncovered case is to rename the root folder...
out_line = self.clean_path(out_line)
# Example output line: directory:My folder:68/1617030350
# In this context, we also might get the following output:
# directory/68/1617030350//My folder/
# ..so, use the clean path to remove all extra slashes.
# The only uncovered case is to rename the root folder...
out_line = self.clean_path(out_line)
# Set names
line_pieces = out_line.split('/')
type = line_pieces[0]
size = line_pieces[1]
timestamp = line_pieces[2]
name = '/'.join(line_pieces[3:])
# Set names
line_pieces = out_line.split('/')
type = line_pieces[0]
size = line_pieces[1]
timestamp = line_pieces[2]
name = '/'.join(line_pieces[3:])
data = {'type': type, 'name': name, 'size': size, 'timestamp': timestamp}
else:
raise NotImplementedError('Storage type "{}" not implemented'.format(storage.type))
return {'type': type, 'name': name, 'size': size, 'timestamp': timestamp}
return data
def delete(self, path, user, storage):
path = self.sanitize_and_prepare_shell_path(path, storage, user)
if storage.type == 'generic_posix':
# Prepare command
command = self.ssh_command('rm -rf {}'.format(path), user, storage.computing)
shell_path = self.sanitize_and_prepare_shell_path(path, user, storage)
# Prepare command
command = self.prepare_sh_command('rm -rf {}'.format(shell_path), user, storage)
# Execute_command
out = os_shell(command, capture=True)
if out.exit_code != 0:
raise Exception(out.stderr)
# Execute_command
out = os_shell(command, capture=True)
if out.exit_code != 0:
raise Exception(out.stderr)
return out.stdout
else:
raise NotImplementedError('Storage type "{}" not implemented'.format(storage.type))
def mkdir(self, path, user, storage, force=False):
def mkdir(self, path, user, storage):
path = self.sanitize_and_prepare_shell_path(path, storage, user)
# Prepare command
command = self.ssh_command('mkdir {}'.format(path), user, storage.computing)
path = self.sanitize_and_prepare_shell_path(path, user, storage)
if storage.type == 'generic_posix':
# Prepare command
if force:
command = self.prepare_sh_command('mkdir -p {}'.format(path), user, storage)
else:
command = self.prepare_sh_command('mkdir {}'.format(path), user, storage)
# Execute_command
out = os_shell(command, capture=True)
if out.exit_code != 0:
raise Exception(out.stderr)
# Execute_command
out = os_shell(command, capture=True)
if out.exit_code != 0:
raise Exception(out.stderr)
return out.stdout
else:
raise NotImplementedError('Storage type "{}" not implemented'.format(storage.type))
def cat(self, path, user, storage):
path = self.sanitize_and_prepare_shell_path(path, storage, user)
# Prepare command
command = self.ssh_command('cat {}'.format(path), user, storage.computing)
# Execute_command
out = os_shell(command, capture=True)
if out.exit_code != 0:
raise Exception(out.stderr)
return out.stdout
# Data container
data = []
if storage.type == 'generic_posix':
shell_path = self.sanitize_and_prepare_shell_path(path, user, storage)
# Prepare command
command = self.prepare_sh_command('cat {}'.format(shell_path), user, storage)
# Execute_command
out = os_shell(command, capture=True)
if out.exit_code != 0:
raise Exception(out.stderr)
data = out.stdout
def rename(self, old, new, user, storage):
old = self.sanitize_and_prepare_shell_path(old, storage, user)
new = self.sanitize_and_prepare_shell_path(new, storage, user)
else:
raise NotImplementedError('Storage type "{}" not implemented'.format(storage.type))
return data
# Prepare command
command = self.ssh_command('mv {} {}'.format(old, new), user, storage.computing)
logger.critical(command)
def rename(self, old, new, user, storage):
if storage.type == 'generic_posix':
old = self.sanitize_and_prepare_shell_path(old, user, storage)
new = self.sanitize_and_prepare_shell_path(new, user, storage)
# Prepare command
command = self.prepare_sh_command('mv {} {}'.format(old, new), user, storage)
logger.critical(command)
# Execute_command
out = os_shell(command, capture=True)
if out.exit_code != 0:
raise Exception(out.stderr)
# Execute_command
out = os_shell(command, capture=True)
if out.exit_code != 0:
raise Exception(out.stderr)
return out.stdout
else:
raise NotImplementedError('Storage type "{}" not implemented'.format(storage.type))
def copy(self, source, target, user, storage):
source = self.sanitize_and_prepare_shell_path(source, storage, user)
target = self.sanitize_and_prepare_shell_path(target, storage, user)
if storage.type == 'generic_posix':
# Prepare command
command = self.ssh_command('cp -a {} {}'.format(source, target), user, storage.computing)
# Execute_command
out = os_shell(command, capture=True)
if out.exit_code != 0:
raise Exception(out.stderr)
return out.stdout
source = self.sanitize_and_prepare_shell_path(source, user, storage)
target = self.sanitize_and_prepare_shell_path(target, user, storage)
# Prepare command
command = self.prepare_sh_command('cp -a {} {}'.format(source, target), user, storage)
# Execute_command
out = os_shell(command, capture=True)
if out.exit_code != 0:
raise Exception(out.stderr)
else:
raise NotImplementedError('Storage type "{}" not implemented'.format(storage.type))
def scp_from(self, source, target, user, storage, mode='get'):
source = self.sanitize_and_prepare_shell_path(source, storage, user)
source = self.sanitize_and_prepare_shell_path(source, user, storage)
target = self.sanitize_shell_path(target) # This is a folder on Rosetta (/tmp)
# Prepare command
command = self.scp_command(source, target, user, storage.computing, mode)
command = self.prepare_scp_command(source, target, user, storage.computing, mode)
# Execute_command
out = os_shell(command, capture=True)
......@@ -707,10 +802,10 @@ class FileManagerAPI(PrivateGETAPI, PrivatePOSTAPI):
def scp_to(self, source, target, user, storage, mode='get'):
source = self.sanitize_shell_path(source) # This is a folder on Rosetta (/tmp)
target = self.sanitize_and_prepare_shell_path(target, storage, user)
target = self.sanitize_and_prepare_shell_path(target, user, storage)
# Prepare command
command = self.scp_command(source, target, user, storage.computing, mode)
command = self.prepare_scp_command(source, target, user, storage.computing, mode)
# Execute_command
out = os_shell(command, capture=True)
......@@ -745,7 +840,7 @@ class FileManagerAPI(PrivateGETAPI, PrivatePOSTAPI):
data = {'data':[]}
# Get storages
storages = list(Storage.objects.filter(group=None)) + list(Storage.objects.filter(group__user=request.user))
storages = list(Storage.objects.filter(group=None,browsable=True)) + list(Storage.objects.filter(group__user=request.user,browsable=True))
# Oder storages (re-orderded in the file manager anyway)
storages.sort(key=lambda storage: storage.id)
......@@ -757,6 +852,9 @@ class FileManagerAPI(PrivateGETAPI, PrivatePOSTAPI):
if not storage.type=='generic_posix' and storage.access_mode=='ssh+cli':
continue
if storage.access_through_computing and not storage.computing.manager.is_configured_for(user=request.user):
continue
data['data'].append({
'id': '/{}/'.format(storage.id),
'type': 'folder',
......@@ -779,37 +877,60 @@ class FileManagerAPI(PrivateGETAPI, PrivatePOSTAPI):
elif mode in ['download', 'getimage']:
logger.debug('Downloading "{}"'.format(path))
# Set support vars
storage = self.get_storage_from_path(path, request)
file_path = '/'+'/'.join(path.split('/')[2:])
if path.endswith('/'):
return error400('Downloading a folder is not supported')
# TOOD: here we are not handling ajax request, Maybe they have been deperacted?
if storage.type != 'generic_posix':
raise NotImplementedError('Storage type "{}" not implemented'.format(storage.type))
# TODO: here we are not handling the ajax request, Maybe they have been deperacted?
# The download process consists of 2 requests:
# - Ajax GET request. Perform all checks and validation. Should return file/folder object in the response data to proceed.
# - Regular GET request. Response headers should be properly configured to output contents to the browser and start download.
# See here: https://github.com/psolom/RichFilemanager/wiki/API
# Set support vars
storage = self.get_storage_from_path(path, request)
file_path = '/'+'/'.join(path.split('/')[2:])
target_path = '/tmp/{}'.format(uuid.uuid4())
# Get the file
self.scp_from(file_path, target_path, request.user, storage, mode='get')
if storage.access_mode == 'ssh+cli':
target_path = '/tmp/{}'.format(uuid.uuid4())
# Get the file
self.scp_from(file_path, target_path, request.user, storage, mode='get')
# Detect content type
try:
content_type = str(magic.from_file(target_path, mime=True))
except:
content_type = None
# Read file data
with open(target_path, 'rb') as f:
data = f.read()
# Remove temporary file
os.remove(target_path)
elif storage.access_mode == 'cli':
# Detect content type
try:
content_type = str(magic.from_file(target_path, mime=True))
except:
content_type = None
# Define storage internal source path
storage_source_path = self.sanitize_and_prepare_shell_path(file_path, request.user, storage, escapes=False)
# Read file data
with open(target_path, 'rb') as f:
data = f.read()
# Remove file
os.remove(target_path)
# Detect content type
try:
content_type = str(magic.from_file(storage_source_path, mime=True))
except:
content_type = None
# Read file data
with open(storage_source_path, 'rb') as f:
data = f.read()
else:
raise NotImplementedError('Storage access mode "{}" not implemented'.format(storage.access_mode))
# Return file data
response = HttpResponse(data, status=status.HTTP_200_OK, content_type=content_type)
response['Content-Disposition'] = 'attachment; filename="{}"'.format(file_path.split('/')[-1])
......@@ -842,8 +963,8 @@ class FileManagerAPI(PrivateGETAPI, PrivatePOSTAPI):
else:
is_folder=False
# Get file contents
data = self.delete(path, request.user, storage)
# Delete
self.delete(path, request.user, storage)
# Response data
data = { 'data': {
......@@ -874,8 +995,8 @@ class FileManagerAPI(PrivateGETAPI, PrivatePOSTAPI):
storage = self.get_storage_from_path(path, request)
path = '/'+'/'.join(path.split('/')[2:]) + name
# Get file contents
data = self.mkdir(path, request.user, storage)
# Create the directory
self.mkdir(path, request.user, storage)
# Response data
data = { 'data': {
......@@ -891,7 +1012,6 @@ class FileManagerAPI(PrivateGETAPI, PrivatePOSTAPI):
}
}
# Return file contents
return Response(data, status=status.HTTP_200_OK)
......@@ -1023,12 +1143,10 @@ class FileManagerAPI(PrivateGETAPI, PrivatePOSTAPI):
# Return file contents
return Response(data, status=status.HTTP_200_OK)
else:
return error400('Operation "{}" not supported'.format(mode))
return Response(data, status=status.HTTP_200_OK)
......@@ -1042,7 +1160,6 @@ class FileManagerAPI(PrivateGETAPI, PrivatePOSTAPI):
path = request.POST.get('path', None)
_ = request.GET.get('_', None)
if mode == 'savefile':
return error400('Operation "{}" not supported'.format(mode))
......@@ -1051,48 +1168,168 @@ class FileManagerAPI(PrivateGETAPI, PrivatePOSTAPI):
# Set support vars
storage = self.get_storage_from_path(path, request)
path = '/'+'/'.join(path.split('/')[2:])
# Bug workaround?
if not path.endswith('/'):
path += '/'
# Get the file upload
file_upload = request.FILES['files']
# generate temporary UUID
file_uuid = uuid.uuid4()
with open('/tmp/{}'.format(file_uuid), 'wb') as temp_file:
temp_file.write(file_upload.read())
logger.debug('Wrote "/tmp/{}" for "{}"'.format(file_uuid, file_upload.name))
# Now copy with scp
self.scp_to('/tmp/{}'.format(file_uuid), path + file_upload.name , request.user, storage, mode='put')
# Response data
data = { 'data': [{
'id': '/{}{}{}'.format(storage.id, path, file_upload.name),
'type': 'file',
'attributes':{
'modified': now_t(), # This is an approximation!
'name': file_upload.name,
'readable': 1,
'size': os.path.getsize('/tmp/{}'.format(file_uuid)), # This is kind of an approximation!
'writable': 1,
'path': '/{}{}{}'.format(storage.id, path, file_upload.name)
}
}]
}
if storage.access_mode == 'ssh+cli':
# Generate temporary UUID
file_uuid = uuid.uuid4()
with open('/tmp/{}'.format(file_uuid), 'wb') as temp_file:
temp_file.write(file_upload.read())
logger.debug('Wrote "/tmp/{}" for "{}"'.format(file_uuid, file_upload.name))
# Now copy with scp
self.scp_to('/tmp/{}'.format(file_uuid), path + file_upload.name , request.user, storage, mode='put')
# Remove file
os.remove('/tmp/{}'.format(file_uuid))
# Response data
data = { 'data': [{
'id': '/{}{}{}'.format(storage.id, path, file_upload.name),
'type': 'file',
'attributes':{
'modified': now_t(), # This is an approximation!
'name': file_upload.name,
'readable': 1,
'size': os.path.getsize('/tmp/{}'.format(file_uuid)), # This is kind of an approximation!
'writable': 1,
'path': '/{}{}{}'.format(storage.id, path, file_upload.name)
}
}]
}
# Remove file
os.remove('/tmp/{}'.format(file_uuid))
if storage.access_mode == 'cli':
try:
as_user = storage.conf['as_user']
# Is "as_user" a UID?
try:
uid = int(as_user)
except:
pass
else:
# What is the user for this uid?
out = os_shell('sudo getent passwd "1000" | cut -d: -f1', capture=True)
if out.exit_code != 0:
raise Exception(out.sterr)
else:
if not out.stdout.strip():
# No user found, create it
os_shell('sudo groupadd -g {0} group_{0}'.format(uid), capture=True)
if out.exit_code != 0:
raise Exception(out.sterr)
os_shell('sudo useradd user_{0} -d /home_{0} -u {0} -g {0} -m -s /bin/bash'.format(uid), capture=True)
if out.exit_code != 0:
raise Exception(out.sterr)
as_user = 'user_' + str(uid)
else:
as_user = out.stdout.strip()
except (KeyError, TypeError):
as_user = None
# Define storage internal dest path
storage_dest_path = self.sanitize_and_prepare_shell_path(path + file_upload.name, request.user, storage, escapes=False)
storage_dest_path_escaped = self.sanitize_and_prepare_shell_path(path + file_upload.name, request.user, storage, escapes=True)
logger.debug('Writing "{}" for "{}"'.format(storage_dest_path, file_upload.name))
if as_user:
# Generate temporary UUID
file_uuid = uuid.uuid4()
with open('/tmp/{}'.format(file_uuid), 'wb') as temp_file:
temp_file.write(file_upload.read())
file_size = os.path.getsize('/tmp/{}'.format(file_uuid))
# Change ownership and move
os_shell('sudo -i -u {0} chown {0}:{0} /tmp/{1}'.format(as_user, file_uuid), capture=True)
if out.exit_code != 0:
raise Exception(out.sterr)
os_shell('sudo -i -u {0} mv /tmp/{1} {2}'.format(as_user, file_uuid, storage_dest_path_escaped), capture=True)
if out.exit_code != 0:
raise Exception(out.sterr)
else:
with open(storage_dest_path, 'wb') as upload_file:
upload_file.write(file_upload.read())
file_size = os.path.getsize(storage_dest_path)
logger.debug('Wrote "{}" for "{}"'.format(storage_dest_path, file_upload.name))
# Response data
data = { 'data': [{
'id': '/{}{}{}'.format(storage.id, path, file_upload.name),
'type': 'file',
'attributes':{
'modified': now_t(), # This is an approximation!
'name': file_upload.name,
'readable': 1,
'size': file_size,
'writable': 1,
'path': '/{}{}{}'.format(storage.id, path, file_upload.name)
}
}]
}
else:
raise NotImplementedError('Storage access mode "{}" not implemented'.format(storage.access_mode))
# Return
return Response(data, status=status.HTTP_200_OK)
else:
return error400('Operation "{}" not supported'.format(mode))
return ok200('ok')
#==============================
# Import repository APIs
#==============================
class ImportRepositoryAPI(PrivateGETAPI):
"""
get:
Import a repository as a container and get the container uuid.
"""
def _get(self, request):
repository_url = request.GET.get('repository_url', None)
repository_tag = request.GET.get('repository_tag', None)
container_name = request.GET.get('container_name', None)
container_description = request.GET.get('container_description', None)
if not repository_url:
return error400('Missing "repository_url"')
if not repository_tag:
return error400('Missing "repository_tag"')
logger.debug('Importing repository "%s" with tag "%s"', repository_url, repository_tag)
results = {}
try:
container = get_or_create_container_from_repository(request.user, repository_url, repository_tag, container_name, container_description)
except Exception as e:
results['import_succeded'] = False
results['error_message'] = str(e)
else:
results['import_succeded'] = True
results['container_uuid'] = str(container.uuid)
return ok200(results)
......
services/webapp/code/rosetta/core_app/computing_managers.py
View file @ 7c962dfb
......@@ -71,6 +71,9 @@ class ComputingManager(object):
# Call actual get task log logic
return self._get_task_log(task, **kwargs)
def is_configured_for(self, user):
return True
class StandaloneComputingManager(ComputingManager):
......@@ -82,8 +85,14 @@ class ClusterComputingManager(ComputingManager):
class SSHComputingManager(ComputingManager):
# SSH-f + keys utils here
pass
def is_configured_for(self, user):
try:
get_ssh_access_mode_credentials(self.computing, user)
except:
return False
else:
return True
......@@ -240,7 +249,7 @@ class SSHStandaloneComputingManager(StandaloneComputingManager, SSHComputingMana
else:
raise NotImplementedError('Accessing a storage with ssh+cli without going through its computing resource is not implemented')
if '$USER' in expanded_base_path:
expanded_base_path = expanded_base_path.replace('$USER', self.task.user.name)
expanded_base_path = expanded_base_path.replace('$USER', task.user.username)
# Expand the bind_path
expanded_bind_path = storage.bind_path
......@@ -250,7 +259,7 @@ class SSHStandaloneComputingManager(StandaloneComputingManager, SSHComputingMana
else:
raise NotImplementedError('Accessing a storage with ssh+cli without going through its computing resource is not implemented')
if '$USER' in expanded_bind_path:
expanded_bind_path = expanded_bind_path.replace('$USER', self.task.user.name)
expanded_bind_path = expanded_bind_path.replace('$USER', task.user.username)
# Add the bind
if not binds:
......@@ -300,7 +309,7 @@ class SSHStandaloneComputingManager(StandaloneComputingManager, SSHComputingMana
else:
raise NotImplementedError('Accessing a storage with ssh+cli without going through its computing resource is not implemented')
if '$USER' in expanded_base_path:
expanded_base_path = expanded_base_path.replace('$USER', self.task.user.name)
expanded_base_path = expanded_base_path.replace('$USER', task.user.username)
# Expand the bind_path
expanded_bind_path = storage.bind_path
......@@ -310,7 +319,7 @@ class SSHStandaloneComputingManager(StandaloneComputingManager, SSHComputingMana
else:
raise NotImplementedError('Accessing a storage with ssh+cli without going through its computing resource is not implemented')
if '$USER' in expanded_bind_path:
expanded_bind_path = expanded_bind_path.replace('$USER', self.task.user.name)
expanded_bind_path = expanded_bind_path.replace('$USER', task.user.username)
# Add the bind
if not binds:
......@@ -324,12 +333,12 @@ class SSHStandaloneComputingManager(StandaloneComputingManager, SSHComputingMana
run_command = 'ssh -o LogLevel=ERROR -i {} -4 -o StrictHostKeyChecking=no {}@{} '.format(computing_keys.private_key_file, computing_user, computing_host)
run_command += '/bin/bash -c \'"rm -rf /tmp/{}_data && mkdir /tmp/{}_data && chmod 700 /tmp/{}_data && '.format(task.uuid, task.uuid, task.uuid)
run_command += 'wget {}/api/v1/base/agent/?task_uuid={} -O /tmp/{}_data/agent.py &> /dev/null && export TASK_PORT=\$(python /tmp/{}_data/agent.py 2> /tmp/{}_data/task.log) && '.format(webapp_conn_string, task.uuid, task.uuid, task.uuid, task.uuid)
run_command += '{} {} run -p \$TASK_PORT:{} {} {} {} '.format(prefix, container_engine, task.container.interface_port, authstring, varsstring, binds)
run_command += 'exec nohup {} {} run -p \$TASK_PORT:{} {} {} {} '.format(prefix, container_engine, task.container.interface_port, authstring, varsstring, binds)
if container_engine == 'podman':
run_command += '--network=private --uts=private '
run_command += '--network=private --uts=private --userns=keep-id '
#run_command += '-d -t {}/{}:{}'.format(task.container.registry, task.container.image_name, task.container.image_tag)
run_command += '-h task-{} -d -t {}/{}:{}'.format(task.short_uuid, task.container.registry, task.container.image_name, task.container.image_tag)
run_command += '"\''
run_command += '-h task-{} -t {}/{}:{}'.format(task.short_uuid, task.container.registry, task.container.image_name, task.container.image_tag)
run_command += '&>> /tmp/{}_data/task.log & echo \$!"\''.format(task.uuid)
else:
raise NotImplementedError('Container engine {} not supported'.format(container_engine))
......@@ -376,7 +385,7 @@ class SSHStandaloneComputingManager(StandaloneComputingManager, SSHComputingMana
stop_command = 'ssh -o LogLevel=ERROR -i {} -4 -o StrictHostKeyChecking=no {}@{} \'/bin/bash -c "{}"\''.format(computing_keys.private_key_file, computing_user, computing_host, internal_stop_command)
out = os_shell(stop_command, capture=True)
if out.exit_code != 0:
if ('No such process' in out.stderr) or ('No such container' in out.stderr):
if ('No such process' in out.stderr) or ('No such container' in out.stderr) or ('no container' in out.stderr) or ('missing' in out.stderr):
pass
else:
raise Exception(out.stderr)
......@@ -402,8 +411,9 @@ class SSHStandaloneComputingManager(StandaloneComputingManager, SSHComputingMana
internal_view_log_command = 'cat /tmp/{}_data/task.log'.format(task.uuid)
elif container_engine in ['docker','podman']:
# TODO: remove this hardcoding
prefix = 'sudo' if (computing_host == 'slurmclusterworker' and container_engine=='docker') else ''
internal_view_log_command = '{} {} logs {}'.format(prefix,container_engine,task.id)
#prefix = 'sudo' if (computing_host == 'slurmclusterworker' and container_engine=='docker') else ''
#internal_view_log_command = '{} {} logs {}'.format(prefix,container_engine,task.id)
internal_view_log_command = 'cat /tmp/{}_data/task.log'.format(task.uuid)
else:
raise NotImplementedError('Container engine {} not supported'.format(container_engine))
......@@ -493,7 +503,7 @@ class SlurmSSHClusterComputingManager(ClusterComputingManager, SSHComputingManag
else:
raise NotImplementedError('Accessing a storage with ssh+cli without going through its computing resource is not implemented')
if '$USER' in expanded_base_path:
expanded_base_path = expanded_base_path.replace('$USER', self.task.user.name)
expanded_base_path = expanded_base_path.replace('$USER', task.user.username)
# Expand the bind_path
expanded_bind_path = storage.bind_path
......@@ -503,7 +513,7 @@ class SlurmSSHClusterComputingManager(ClusterComputingManager, SSHComputingManag
else:
raise NotImplementedError('Accessing a storage with ssh+cli without going through its computing resource is not implemented')
if '$USER' in expanded_bind_path:
expanded_bind_path = expanded_bind_path.replace('$USER', self.task.user.name)
expanded_bind_path = expanded_bind_path.replace('$USER', task.user.username)
# Add the bind
if not binds:
......
services/webapp/code/rosetta/core_app/decorators.py
View file @ 7c962dfb
......@@ -132,5 +132,8 @@ def private_view(wrapped_view):
else:
log_user_activity("DEBUG", "Redirecting to login since not authenticated", request)
return HttpResponseRedirect('/login')
logger.debug('Setting cookie-based post login redirect to "%s"', request.build_absolute_uri())
response = HttpResponseRedirect('/login')
response.set_cookie('post_login_redirect', request.build_absolute_uri())
return response
return private_view_wrapper
services/webapp/code/rosetta/core_app/management/commands/core_app_populate.py
View file @ 7c962dfb
......@@ -280,8 +280,20 @@ to provide help, news and informations on your deployment. Or you can just ignor
auth_mode = 'user_keys',
wms = None,
conf = {'host': 'standaloneworker'},
container_engines = ['singularity','podman'])
container_engines = ['podman','singularity'])
# Demo standalone platform computing plus conf
demo_singlenode_computing = Computing.objects.create(name = 'Demo Standalone Platform',
description = 'A demo standalone computing resource access as platform.',
type = 'standalone',
arch = 'amd64',
supported_archs = ['386'],
access_mode = 'ssh+cli',
auth_mode = 'platform_keys',
wms = None,
conf = {'host': 'standaloneworker', 'user': 'rosetta'},
container_engines = ['podman','singularity'])
# Add testuser extra conf for this computing resource
testuser.profile.add_extra_conf(conf_type = 'computing_user', object=demo_singlenode_computing, value= 'testuser')
......@@ -324,7 +336,7 @@ to provide help, news and informations on your deployment. Or you can just ignor
for computing in demo_computing_resources:
# Demo shared computing plus conf
# Demo shared storage
Storage.objects.create(computing = computing,
access_through_computing = True,
name = 'Shared',
......@@ -334,7 +346,7 @@ to provide help, news and informations on your deployment. Or you can just ignor
base_path = '/shared/data/shared',
bind_path = '/storages/shared')
# Demo shared computing plus conf
# Demo personal storage
Storage.objects.create(computing = computing,
access_through_computing = True,
name = 'Personal',
......@@ -343,7 +355,24 @@ to provide help, news and informations on your deployment. Or you can just ignor
auth_mode = 'user_keys',
base_path = '/shared/data/users/$SSH_USER',
bind_path = '/storages/personal')
try:
demo_standalone_computing = Computing.objects.get(name='Demo Standalone Platform')
demo_computing_resources.append(demo_standalone_computing)
# Demo personal storage
Storage.objects.create(computing = computing,
access_through_computing = True,
name = 'Personal',
type = 'generic_posix',
access_mode = 'ssh+cli',
auth_mode = 'user_keys',
base_path = '/shared/data/users/$SSH_USER',
bind_path = '/storages/personal')
except:
pass
......
services/webapp/code/rosetta/core_app/migrations/0032_auto_20220409_1813.py 0 → 100644
View file @ 7c962dfb
# Generated by Django 2.2.1 on 2022-04-09 18:13
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
('core_app', '0031_container_env_vars'),
]
operations = [
migrations.AddField(
model_name='storage',
name='browsable',
field=models.BooleanField(default=False, verbose_name='Browsable in the file manager?'),
),
migrations.AlterField(
model_name='storage',
name='bind_path',
field=models.CharField(blank=True, max_length=4096, null=True, verbose_name='Bind path'),
),
]
services/webapp/code/rosetta/core_app/migrations/0033_auto_20220410_1531.py 0 → 100644
View file @ 7c962dfb
# Generated by Django 2.2.1 on 2022-04-10 15:31
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
('core_app', '0032_auto_20220409_1813'),
]
operations = [
migrations.AlterField(
model_name='storage',
name='browsable',
field=models.BooleanField(default=True, verbose_name='Browsable in the file manager?'),
),
]