4. Charliecloud command reference¶
This section is a comprehensive description of the usage and arguments of the Charliecloud commands. Its content is identical to the commands’ man pages.
4.1. ch-build¶
Build an image and place it in the builder’s back-end storage.
4.1.1. Synopsis¶
$ ch-build [-b BUILDER] [--builder-info] -t TAG [ARGS ...] CONTEXT
4.1.2. Description¶
Build an image named TAG
described by a Dockerfile. Place the result
into the builder’s back-end storage.
Using this script is not required for a working Charliecloud image. You can also use any builder that can produce a Linux filesystem tree directly, whether or not it is in the list below. However, this script hides the vagaries of making the supported builders work smoothly with Charliecloud and adds some conveniences (e.g., pass HTTP proxy environment variables to the build environment if the builder doesn’t do this by default).
Supported builders, unprivileged:
buildah
: Buildah in “rootless” mode with no setuid helpers, usingch-run
(viach-run-oci
) forRUN
instructions. This currently requires a patched Buildah; see the install instructions.
ch-grow
: Our internal builder.
Supported builders, privileged:
buildah-runc
: Buildah in “rootless” mode with setuid helpers, using the defaultrunc
forRUN
instructions.
buildah-setuid
: Buildah in “rootless” mode with setuid helpers, usingch-run
(viach-run-oci
) forRUN
instructions.
docker
: Docker.
Specifying the builder, in descending order of priority:
-b
,--builder BUILDER
Command line option.
$CH_BUILDER
Environment variable
- Default
docker
if Docker is installed; otherwise,ch-grow
.
Other arguments:
--builder-info
Print the builder to be used and its version, then exit.
-f
,--file DOCKERFILE
Dockerfile to use (default:
$CONTEXT/Dockerfile
)-t TAG
Name (tag) of Docker image to build.
--help
Print help and exit.
--version
Print version and exit.
Additional arguments are accepted and passed unchanged to the underlying builder.
4.1.3. Bugs¶
The tag suffix :latest
is somewhat misleading, as by default neither
ch-build
nor bare builders will notice if the base FROM
image
has been updated. Use --pull
to make sure you have the latest base
image.
4.1.4. Examples¶
Create an image tagged foo
and specified by the file
Dockerfile
located in the context directory. Use /bar
as the
Docker context directory. Use the default builder.
$ ch-build -t foo /bar
Equivalent to above:
$ ch-build -t foo --file=/bar/Dockerfile /bar
Instead, use /bar/Dockerfile.baz
:
$ ch-build -t foo --file=/bar/Dockerfile.baz /bar
Equivalent to the first example, but use ch-grow
even if Docker is
installed:
$ ch-build -b ch-grow -t foo /bar
Equivalent to above:
$ export CH_BUILDER=ch-grow
$ ch-build -t foo /bar
4.2. ch-build2dir¶
Build a Charliecloud image from Dockerfile and unpack it into a directory.
4.2.1. Synopsis¶
$ ch-build2dir -t TAG [ARGS ...] CONTEXT OUTDIR
4.2.2. Description¶
Build a Docker image named TAG
described by a Dockerfile (default
$CONTEXT/Dockerfile
) and unpack it into OUTDIR/TAG
. This is a
wrapper for ch-build
, ch-builder2tar
, and ch-tar2dir
;
see also those man pages.
Arguments:
ARGS
additional arguments passed to
ch-build
CONTEXT
Docker context directory
OUTDIR
directory in which to place image directory (named
TAG
) and temporary tarball-t TAG
name (tag) of Docker image to build
--help
print help and exit
--version
print version and exit
4.2.3. Examples¶
To build using ./Dockerfile
and create image directory
/var/tmp/foo
:
$ ch-build2dir -t foo . /var/tmp
Same as above, but build with a different Dockerfile:
$ ch-build2dir -t foo -f ./Dockerfile.foo . /var/tmp
4.3. ch-builder2tar¶
Flatten a builder image into a Charliecloud image tarball.
4.3.1. Synopsis¶
$ ch-builder2tar [-b BUILDER] [--nocompress] IMAGE OUTDIR
4.3.2. Description¶
Flatten the builder image tagged IMAGE
into a Charliecloud tarball in
directory OUTDIR
.
The builder-specified environment (e.g., ENV
statements) is placed in
a file in the tarball at $IMAGE/ch/environment
, in a form suitable for
ch-run --set-env
.
See ch-build(1)
for details on specifying the builder.
Additional arguments:
-b
,--builder BUILDER
Use specified builder; if not given, use
$CH_BUILDER
or default.--nocompress
Do not compress tarball.
--help
Print help and exit.
--version
Print version and exit.
4.3.3. Example¶
$ ch-builder2tar hello /var/tmp
57M /var/tmp/hello.tar.gz
$ ls -lh /var/tmp
-rw-r----- 1 reidpr reidpr 57M Feb 13 16:14 hello.tar.gz
4.4. ch-checkns¶
Check ch-run
prerequisites, e.g., namespaces and pivot_root(2)
.
4.4.1. Synopsis¶
$ ch-checkns
4.4.2. Description¶
Check ch-run
prerequisites, e.g., namespaces and pivot_root(2)
.
4.4.3. Example¶
$ ch-checkns
ok
4.5. ch-dir2squash¶
Create a SquashFS file from an image directory.
4.5.1. Synopsis¶
$ ch-dir2squash IMGDIR OUTDIR [ARGS ...]
4.5.2. Description¶
Create Charliecloud SquashFS file from image directory IMGDIR
under
directory OUTDIR
, named as last component of IMGDIR
plus
suffix .sqfs
.
Optional ARGS
will passed to mksquashfs
unchanged.
Additional arguments:
--help
print help and exit
--version
print version and exit
4.5.3. Example¶
$ ch-dir2squash /var/tmp/debian /var/tmp
Parallel mksquashfs: Using 6 processors
Creating 4.0 filesystem on /var/tmp/debian.sqfs, block size 131072.
[...]
-rw-r--r-- 1 charlie charlie 41M Apr 23 14:41 /var/tmp/debian.sqfs
4.6. ch-docker2squash¶
Flatten a Docker image into a Charliecloud SquashFS file.
4.6.1. Synopsis¶
$ ch-docker2squash IMAGE OUTDIR [ARGS ...]
4.6.2. Description¶
Flattens the Docker image tagged IMAGE
into a SquashFS file in
OUTDIR
.
Wrapper for ch-docker2tar --nocompress
and ch-tar2sqfs
.
Intermediate files and directories are removed.
Sudo privileges are required to run docker export
.
Optional ARGS
passed to mksquashfs
unchanged.
Additional arguments:
--help
print help and exit
--version
print version and exit
4.6.3. Example¶
$ docker image list | fgrep debian
REPOSITORY TAG IMAGE ID CREATED SIZE
debian stretch 2d337f242f07 3 weeks ago 101MB
$ ch-docker2squash debian /var/tmp
Parallel mksquashfs: Using 6 processors
Creating 4.0 filesystem on /var/tmp/debian.sqfs, block size 131072.
[...]
squashed /var/tmp/debian.sqfs OK
$ ls -lh /var/tmp/debian*
-rw-r--r-- 1 charlie charlie 41M Apr 23 14:37 debian.sqfs
4.7. ch-fromhost¶
Inject files from the host into an image directory.
4.7.1. Synopsis¶
$ ch-fromhost [OPTION ...] [FILE_OPTION ...] IMGDIR
4.7.2. Description¶
Note
This command is experimental. Features may be incomplete and/or buggy. Please report any issues you find, so we can fix them!
Inject files from the host into the Charliecloud image directory
IMGDIR
.
The purpose of this command is to provide host-specific files, such as GPU
libraries, to a container. It should be run after ch-tar2dir
and
before ch-run
. After invocation, the image is no longer portable to
other hosts.
Injection is not atomic; if an error occurs partway through injection, the image is left in an undefined state. Injection is currently implemented using a simple file copy, but that may change in the future.
By default, file paths that contain the strings /bin
or /sbin
are assumed to be executables and placed in /usr/bin
within the
container. File paths that contain the strings /lib
or .so
are
assumed to be shared libraries and are placed in the first-priority directory
reported by ldconfig
(see --lib-path
below). Other files are
placed in the directory specified by --dest
.
If any shared libraries are injected, run ldconfig
inside the
container (using ch-run -w
) after injection.
4.7.3. Options¶
4.7.3.1. To specify which files to inject¶
-c
,--cmd CMD
Inject files listed in the standard output of command
CMD
.-f
,--file FILE
Inject files listed in the file
FILE
.-p
,--path PATH
Inject the file at
PATH
.--cray-mpi
Cray-enable an MPICH installed inside the image. See important details below.
--nvidia
Use
nvidia-container-cli list
(fromlibnvidia-container
) to find executables and libraries to inject.
These can be repeated, and at least one must be specified.
4.7.3.2. To specify the destination within the image¶
-d
,--dest DST
Place files specified later in directory
IMGDIR/DST
, overriding the inferred destination, if any. If a file’s destination cannot be inferred and--dest
has not been specified, exit with an error. This can be repeated to place files in varying destinations.
4.7.3.3. Additional arguments¶
--lib-path
Print the guest destination path for shared libraries inferred as described above.
--no-ldconfig
Don’t run
ldconfig
even if we appear to have injected shared libraries.-h
,--help
Print help and exit.
-v
,--verbose
List the injected files.
--version
Print version and exit.
4.7.4. --cray-mpi
dependencies and quirks¶
The implementation of --cray-mpi
for MPICH is messy, foul smelling,
and brittle. It replaces or overrides the open source MPICH libraries
installed in the container. Users should be aware of the following.
Containers must have the following software installed:
Open source MPICH.
PatchELF with our patches. Use the
shrink-soname
branch.libgfortran.so.3
, because Cray’slibmpi.so.12
links to it.
Applications must be linked to
libmpi.so.12
(not e.g.libmpich.so.12
). How to configure MPICH to accomplish this is not yet clear to us;test/Dockerfile.mpich
does it, while the Debian packages do not.One of the
cray-mpich-abi
modules must be loaded whench-fromhost
is invoked.Tested only for C programs compiled with GCC, and it probably won’t work otherwise. If you’d like to use another compiler or another programming language, please get in touch so we can implement the necessary support.
Please file a bug if we missed anything above or if you know how to make the code better.
4.7.5. Notes¶
Symbolic links are dereferenced, i.e., the files pointed to are injected, not the links themselves.
As a corollary, do not include symlinks to shared libraries. These will be
re-created by ldconfig
.
There are two alternate approaches for nVidia GPU libraries:
Link
libnvidia-containers
intoch-run
and call the library functions directly. However, this would mean that Charliecloud would either (a) need to be compiled differently on machines with and without nVidia GPUs or (b) havelibnvidia-containers
available even on machines without nVidia GPUs. Neither of these is consistent with Charliecloud’s philosophies of simplicity and minimal dependencies.Use
nvidia-container-cli configure
to do the injecting. This would require that containers have a half-started state, where the namespaces are active and everything is mounted butpivot_root(2)
has not been performed. This is not feasible because Charliecloud has no notion of a half-started container.
Further, while these alternate approaches would simplify or eliminate this script for nVidia GPUs, they would not solve the problem for other situations.
4.7.6. Bugs¶
File paths may not contain colons or newlines.
4.7.7. Examples¶
Place shared library /usr/lib64/libfoo.so
at path
/usr/lib/libfoo.so
(assuming /usr/lib
is the first directory
searched by the dynamic loader in the image), within the image
/var/tmp/baz
and executable /bin/bar
at path
/usr/bin/bar
. Then, create appropriate symlinks to libfoo
and
update the ld.so
cache.
$ cat qux.txt
/bin/bar
/usr/lib64/libfoo.so
$ ch-fromhost --file qux.txt /var/tmp/baz
Same as above:
$ ch-fromhost --cmd 'cat qux.txt' /var/tmp/baz
Same as above:
$ ch-fromhost --path /bin/bar --path /usr/lib64/libfoo.so /var/tmp/baz
Same as above, but place the files into /corge
instead (and the shared
library will not be found by ldconfig
):
$ ch-fromhost --dest /corge --file qux.txt /var/tmp/baz
Same as above, and also place file /etc/quux
at /etc/quux
within the container:
$ ch-fromhost --file qux.txt --dest /etc --path /etc/quux /var/tmp/baz
Inject the executables and libraries recommended by nVidia into the image, and
then run ldconfig
:
$ ch-fromhost --nvidia /var/tmp/baz
4.7.8. Acknowledgements¶
This command was inspired by the similar Shifter feature
that allows Shifter containers to use the Cray Aires network. We particularly
appreciate the help provided by Shane Canon and Doug Jacobsen during our
implementation of --cray-mpi
.
We appreciate the advice of Ryan Olson at nVidia on implementing
--nvidia
.
4.8. ch-grow¶
Build an image from a Dockerfile; completely unprivileged.
4.8.1. Synopsis¶
$ ch-grow [OPTIONS] [-t TAG] [-f DOCKERFILE] CONTEXT
4.8.2. Description¶
Warning
This script is experimental. Please report the bugs you find so we can fix them!
Build an image named TAG
as specified in DOCKERFILE
; use
ch-run(1)
to execute RUN
instructions. This builder is
completely unprivileged, with no setuid/setgid/setcap helpers.
ch-grow
maintains state and temporary images using normal files and
directories. This storage directory can reside on any filesystem, and its
location is configurable. In descending order of priority:
-s
,--storage DIR
Command line option.
$CH_GROW_STORAGE
Environment variable.
/var/tmp/ch-grow
Default.
Note
Images are stored unpacked, so place your storage directory on a filesystem that can handle the metadata traffic for large numbers of small files. For example, the Charliecloud test suite uses approximately 400,000 files and directories.
Other arguments:
CONTEXT
Context directory; this is the root of
COPY
andADD
instructions in the Dockerfile.-f
,--file DOCKERFILE
Use
DOCKERFILE
instead ofCONTEXT/Dockerfile
.-h
,--help
Print help and exit.
-n
,--dry-run
Do not actually excute any Dockerfile instructions.
--parse-only
Stop after parsing the Dockerfile.
--print-storage
Print the storage directory path and exit.
-t
,-tag TAG
Name of image to create. Append
:latest
if no colon present.--verbose
Print lots of debugging chatter.
--version
Print version number and exit.
4.8.3. Bugs¶
This script executes RUN
instructions with host EUID and EGID both
mapped to zero in the container, i.e., with ch-run --uid=0 gid=0
. This
confuses many programs that appear in RUN
, which see EUID 0 and/or
EGID 0 and assume they can actually do privileged things, which then fail with
“permission denied” and related errors. For example, chgrp(1)
often
appears in Debian package post-install scripts. We have worked around some of
these problems, but many remain; please report any you find as bugs.
COPY
and ADD
source paths are not restricted to the context
directory. However, because ch-grow
is completely unprivileged, this
cannot be used to add files not normally readable by the user to the
image.
4.9. ch-mount¶
Mount a SquashFS image file using FUSE.
4.9.1. Synopsis¶
$ ch-mount SQFS PARENTDIR
4.9.2. Description¶
Create new empty directory named SQFS
with suffix (e.g.,
.sqfs
) removed, then mount SQFS
on this new directory. This
new directory must not already exist.
Additional arguments:
--help
print help and exit
--version
print version and exit
4.9.3. Example¶
$ ch-mount /var/tmp/debian.sqfs /var/tmp
$ ls /var/tmp/debian
bin dev home lib64 mnt proc run srv tmp var
boot etc lib media opt root sbin sys usr WEIRD_AL_YANKOVIC
4.10. ch-pull2dir¶
Pull image from a Docker Hub and unpack into directory.
4.10.1. Synopsis¶
$ ch-pull2dir IMAGE[:TAG] DIR
4.10.2. Description¶
Pull Docker image named IMAGE[:TAG]
from Docker Hub and extract it
into a subdirectory of DIR
. A temporary tarball is stored in
DIR
.
Sudo privileges are required to run the docker pull
command.
This runs the following command sequence: ch-pull2tar
,
ch-tar2dir
. See warning in the documentation for ch-tar2dir
.
Additional arguments:
--help
print help and exit
--version
print version and exit
4.10.3. Examples¶
$ ch-pull2dir alpine /var/tmp
Using default tag: latest
latest: Pulling from library/alpine
Digest: sha256:621c2f39f8133acb8e64023a94dbdf0d5ca81896102b9e57c0dc184cadaf5528
Status: Image is up to date for alpine:latest
-rw-r--r--. 1 charlie charlie 2.1M Oct 5 19:52 /var/tmp/alpine.tar.gz
creating new image /var/tmp/alpine
/var/tmp/alpine unpacked ok
removed '/var/tmp/alpine.tar.gz'
Same as above, except optional TAG
is specified:
$ ch-pull2dir alpine:3.6 /var/tmp
3.6: Pulling from library/alpine
Digest: sha256:cc24af836d1377e092ecb4e8f0a4324c3b1aa2b5295c2239edcc7bbc86a9cbc6
Status: Image is up to date for alpine:3.6
-rw-r--r--. 1 charlie charlie 2.1M Oct 5 19:54 /var/tmp/alpine:3.6.tar.gz
creating new image /var/tmp/alpine:3.6
/var/tmp/alpine:3.6 unpacked ok
removed '/var/tmp/alpine:3.6.tar.gz'
4.11. ch-pull2tar¶
Pull image from a Docker Hub and flatten into tarball.
4.11.1. Synopsis¶
$ ch-pull2tar IMAGE[:TAG] OUTDIR
4.11.2. Description¶
Pull a Docker image named IMAGE[:TAG]
from Docker Hub and flatten it
into a Charliecloud tarball in directory OUTDIR
.
This runs the following command sequence: docker pull
,
ch-builder2tar
but provides less flexibility than the individual
commands.
Sudo privileges are required for docker pull
.
Additional arguments:
--help
print help and exit
--version
print version and exit
4.11.3. Examples¶
$ ch-pull2tar alpine /var/tmp
Using default tag: latest
latest: Pulling from library/alpine
Digest: sha256:621c2f39f8133acb8e64023a94dbdf0d5ca81896102b9e57c0dc184cadaf5528
Status: Image is up to date for alpine:latest
-rw-r--r--. 1 charlie charlie 2.1M Oct 5 19:52 /var/tmp/alpine.tar.gz
Same as above, except optional TAG
is specified:
$ ch-pull2tar alpine:3.6
3.6: Pulling from library/alpine
Digest: sha256:cc24af836d1377e092ecb4e8f0a4324c3b1aa2b5295c2239edcc7bbc86a9cbc6
Status: Image is up to date for alpine:3.6
-rw-r--r--. 1 charlie charlie 2.1M Oct 5 19:54 /var/tmp/alpine:3.6.tar.gz
4.12. ch-run¶
Run a command in a Charliecloud container.
4.12.1. Synopsis¶
$ ch-run [OPTION...] NEWROOT CMD [ARG...]
4.12.2. Description¶
Run command CMD
in a Charliecloud container using the flattened and
unpacked image directory located at NEWROOT
.
-b
,--bind=SRC[:DST]
mount
SRC
at guestDST
(default/mnt/0
,/mnt/1
, etc.)-c
,--cd=DIR
initial working directory in container
--ch-ssh
bind
ch-ssh(1)
into container at/usr/bin/ch-ssh
-g
,--gid=GID
run as group
GID
within container-j
,--join
use the same container (namespaces) as peer
ch-run
invocations--join-pid=PID
join the namespaces of an existing process
--join-ct=N
number of
ch-run
peers (implies--join
; default: see below)--join-tag=TAG
label for
ch-run
peer group (implies--join
; default: see below)--no-home
do not bind-mount your home directory (by default, your home directory is mounted at
/home/$USER
in the container)-t
,--private-tmp
use container-private
/tmp
(by default,/tmp
is shared with the host)--set-env=FILE
set environment variables as specified in host path
FILE
-u
,--uid=UID
run as user
UID
within container--unset-env=GLOB
unset environment variables whose names match
GLOB
-v
,--verbose
be more verbose (debug if repeated)
-w
,--write
mount image read-write (by default, the image is mounted read-only)
-?
,--help
print help and exit
--usage
print a short usage message and exit
-V
,--version
print version and exit
4.12.3. Host files and directories available in container via bind mounts¶
In addition to any directories specified by the user with --bind
,
ch-run
has standard host files and directories that are bind-mounted
in as well.
The following host files and directories are bind-mounted at the same location in the container. These cannot be disabled.
/dev
/etc/passwd
/etc/group
/etc/hosts
/etc/resolv.conf
/proc
/sys
Three additional bind mounts can be disabled by the user:
Your home directory (i.e.,
$HOME
) is mounted at guest/home/$USER
by default. This is accomplished by mounting a newtmpfs
at/home
, which hides any image content under that path. If--no-home
is specified, neither of these things happens and the image’s/home
is exposed unaltered.
/tmp
is shared with the host by default. If--private-tmp
is specified, a newtmpfs
is mounted on the guest’s/tmp
instead.If file
/usr/bin/ch-ssh
is present in the image, it is over-mounted with thech-ssh
binary in the same directory asch-run
.
4.12.4. Multiple processes in the same container with --join
¶
By default, different ch-run
invocations use different user and mount
namespaces (i.e., different containers). While this has no impact on sharing
most resources between invocations, there are a few important exceptions.
These include:
ptrace(2)
, used by debuggers and related tools. One can attach a debugger to processes in descendant namespaces, but not sibling namespaces. The practical effect of this is that (without--join
), you can’t run a command withch-run
and then attach to it with a debugger also run withch-run
.Cross-memory attach (CMA) is used by cooperating processes to communicate by simply reading and writing one another’s memory. This is also not permitted between sibling namespaces. This affects various MPI implementations that use CMA to pass messages between ranks on the same node, because it’s faster than traditional shared memory.
--join
is designed to address this by placing related ch-run
commands (the “peer group”) in the same container. This is done by one of the
peers creating the namespaces with unshare(2)
and the others joining
with setns(2)
.
To do so, we need to know the number of peers and a name for the group. These are specified by additional arguments that can (hopefully) be left at default values in most cases:
--join-ct
sets the number of peers. The default is the value of the first of the following environment variables that is defined:OMPI_COMM_WORLD_LOCAL_SIZE
,SLURM_STEP_TASKS_PER_NODE
,SLURM_CPUS_ON_NODE
.--join-tag
sets the tag that names the peer group. The default is environment variableSLURM_STEP_ID
, if defined; otherwise, the PID ofch-run
’s parent. Tags can be re-used for peer groups that start at different times, i.e., once all peerch-run
have replaced themselves with the user command, the tag can be re-used.
Caveats:
One cannot currently add peers after the fact, for example, if one decides to start a debugger after the fact. (This is only required for code with bugs and is thus an unusual use case.)
ch-run
instances race. The winner of this race sets up the namespaces, and the other peers use the winner to find the namespaces to join. Therefore, if the user command of the winner exits, any remaining peers will not be able to join the namespaces, even if they are still active. There is currently no general way to specify whichch-run
should be the winner.If
--join-ct
is too high, the winningch-run
’s user command exits before all peers join, orch-run
itself crashes, IPC resources such as semaphores and shared memory segments will be leaked. These appear as files in/dev/shm/
and can be removed withrm(1)
.Many of the arguments given to the race losers, such as the image path and
--bind
, will be ignored in favor of what was given to the winner.
4.12.5. Environment variables¶
ch-run
leaves environment variables unchanged, i.e. the host
environment is passed through unaltered, except:
limited tweaks to avoid significant guest breakage;
user-set variables via
--set-env
; anduser-unset variables via
--unset-env
.
This section describes these features.
The default tweaks happen first, and then --set-env
and
--unset-env
in the order specified on the command line. The latter two
can be repeated arbitrarily many times, e.g. to add/remove multiple variable
sets or add only some variables in a file.
4.12.5.1. Default behavior¶
By default, ch-run
makes the following environment variable changes:
$HOME
: If the path to your home directory is not/home/$USER
on the host, then an inherited$HOME
will be incorrect inside the guest. This confuses some software, such as Spack.Thus, we change
$HOME
to/home/$USER
, unless--no-home
is specified, in which case it is left unchanged.$PATH
: Newer Linux distributions replace some root-level directories, such as/bin
, with symlinks to their counterparts in/usr
.Some of these distributions (e.g., Fedora 24) have also dropped
/bin
from the default$PATH
. This is a problem when the guest OS does not have a merged/usr
(e.g., Debian 8 “Jessie”). Thus, we add/bin
to$PATH
if it’s not already present.Further reading:
4.12.5.2. Setting variables with --set-env
¶
The purpose of --set-env=FILE
is to set environment variables that
cannot be inherited from the host shell, e.g. Dockerfile ENV
directives or other build-time configuration. FILE
is a host path to
provide the greatest flexibility; guest paths can be specified by prepending
the image path.
ch-builder2tar(1)
lists variables specified at build time in
Dockerfiles in the image in file /ch/environment
. To set these
variables: --set-env=$IMG/ch/environment
.
Variable values in FILE
replace any already set. If a variable is
repeated, the last value wins.
The syntax of FILE
is key-value pairs separated by the first equals
character (=
, ASCII 61), one per line, with optional single straight
quotes ('
, ASCII 39) around the value. Empty lines are ignored.
Newlines (ASCII 10) are not permitted in either key or value. No variable
expansion, comments, etc. are provided. The value may be empty, but not the
key. (This syntax is designed to accept the output of printenv
and be
easily produced by other simple mechanisms.) Examples of valid lines:
Line |
Key |
Value |
---|---|---|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
(empty string) |
|
|
(empty string) |
|
|
|
Example invalid lines:
Line |
Problem |
---|---|
|
no separator |
|
key cannot be empty |
Example valid lines that are probably not what you want:
Line |
Key |
Value |
Problem |
---|---|---|---|
|
|
|
double quotes aren’t stripped |
|
|
|
comments not supported |
|
|
|
variables not expanded |
|
|
|
leading space in key |
|
|
|
leading space in value |
4.12.5.3. Removing variables with --unset-env
¶
The purpose of --unset-env=GLOB
is to remove unwanted environment
variables. The argument GLOB
is a glob pattern (dialect fnmatch(3)
with no flags); all variables with matching names are removed from the
environment.
Warning
Because the shell also interprets glob patterns, if any wildcard characters
are in GLOB
, it is important to put it in single quotes to avoid
surprises.
GLOB
must be a non-empty string.
Example 1: Remove the single environment variable FOO
:
$ export FOO=bar
$ env | fgrep FOO
FOO=bar
$ ch-run --unset-env=FOO $CH_TEST_IMGDIR/chtest -- env | fgrep FOO
$
Example 2: Hide from a container the fact that it’s running in a Slurm
allocation, by removing all variables beginning with SLURM
. You might
want to do this to test an MPI program with one rank and no launcher:
$ salloc -N1
$ env | egrep '^SLURM' | wc
44 44 1092
$ ch-run $CH_TEST_IMGDIR/mpihello-openmpi -- /hello/hello
[... long error message ...]
$ ch-run --unset-env='SLURM*' $CH_TEST_IMGDIR/mpihello-openmpi -- /hello/hello
0: MPI version:
Open MPI v3.1.3, package: Open MPI root@c897a83f6f92 Distribution, ident: 3.1.3, repo rev: v3.1.3, Oct 29, 2018
0: init ok cn001.localdomain, 1 ranks, userns 4026532530
0: send/receive ok
0: finalize ok
Example 3: Clear the environment completely (remove all variables):
$ ch-run --unset-env='*' $CH_TEST_IMGDIR/chtest -- env
$
Note that some programs, such as shells, set some environment variables even if started with no init files:
$ ch-run --unset-env='*' $CH_TEST_IMGDIR/debian9 -- bash --noprofile --norc -c env
SHLVL=1
PWD=/
_=/usr/bin/env
$
4.12.6. Examples¶
Run the command echo hello
inside a Charliecloud container using the
unpacked image at /data/foo
:
$ ch-run /data/foo -- echo hello
hello
Run an MPI job that can use CMA to communicate:
$ srun ch-run --join /data/foo -- bar
4.13. ch-run-oci¶
OCI wrapper for ch-run
.
4.13.1. Synopsis¶
$ ch-run-oci OPERATION [ARG ...]
4.13.2. Description¶
Note
This command is experimental. Features may be incomplete and/or buggy. The quality of code is not yet up to the usual Charliecloud standards, and error handling is poor. Please report any issues you find, so we can fix them!
Open Containers Initiative (OCI) wrapper for ch-run(1)
. You probably
don’t want to run this command directly; it is intended to interface with
other software that expects an OCI runtime. The current goal is to support
completely unprivileged image building (e.g. buildah
--runtime=ch-run-oci
) rather than general OCI container running.
Support of the OCI runtime specification is only partial. This is for two reasons. First, it’s an experimental and incomplete feature. More importantly, the philosophy and goals of OCI differ significantly from those of Charliecloud. Key differences include:
OCI is designed to run services, while Charliecloud is designed to run scientific applications.
OCI containers are persistent things with a complex lifecycle, while Charliecloud containers are simply UNIX processes.
OCI expects support for a variety of namespaces, while Charliecloud supports user and mount, no more and no less.
OCI expects runtimes to maintain a supervisor process in addition to user processes; Charliecloud has no need for this.
OCI expects runtimes to maintain state throughout the container lifecycle in a location independent from the caller.
For these reasons, ch-run-oci
is a bit of a kludge, and much of what
it does is provide scaffolding to satisfy OCI requirements.
Which OCI features are and are not supported is provided in the rest of this man page, and technical analysis and discussion are in the Contributor’s Guide.
This command supports OCI version 1.0.0 only and fails with an error if other versions are offered.
4.13.3. Operations¶
All OCI operations are accepted, but some are no-ops or merely scaffolding to satisfy the caller. For comparison, see also:
4.13.3.1. create
¶
$ ch-run-oci create --bundle DIR --pid-file FILE [--no-new-keyring] CONTAINER_ID
Create a container. Charliecloud does not have separate create and start phases, so this operation only sets up OCI-related scaffolding.
Arguments:
--bundle DIR
Directory containing the OCI bundle. This must be
/tmp/buildahYYY
, whereYYY
matchesCONTAINER_ID
below.--pid-file FILE
Filename to write the “container” process PID to. Note that for Charliecloud, the process given is fake; see above. This must be
DIR/pid
, whereDIR
is given by--bundle
.--no-new-keyring
Ignored. (Charliecloud does not implement session keyrings.)
CONTAINER_ID
String to use as the container ID. This must be
buildah-buildahYYY
, whereYYY
matchesDIR
above.
Unsupported arguments:
--console-socket PATH
UNIX socket to pass pseudoterminal file descriptor. Charliecloud does not support pseudoterminals; fail with an error if this argument is given. For Buildah, redirect its input from
/dev/null
to prevent it from requesting a pseudoterminal.
4.13.3.2. delete
¶
$ ch-run-oci delete CONTAINER_ID
Clean up the OCI-related scaffolding for specified container.
4.13.3.4. start
¶
$ ch-run-oci start CONTAINER_ID
Eexecute the user command specified at create time in a Charliecloud container.
4.13.3.5. state
¶
$ ch-run-oci state CONTAINER_ID
Print the state of the given container on standard output as an OCI compliant JSON document.
4.13.4. Unsupported OCI features¶
As noted above, various OCI features are not supported by Charliecloud. We
have tried to guess which features would be essential to callers;
ch-run-oci
fails with an error if these are requested. Otherwise, the
request is simply ignored.
We are interested in hearing about scientific-computing use cases for unsupported features, so we can add support for things that are needed.
Our goal is for this man page to be comprehensive: every OCI runtime feature should either work or be listed as unsupported.
Unsupported features that are an error:
Pseudoterminals
Hooks (prestart, poststart, and prestop)
Annotations
Joining existing namespaces
Intel Resource Director Technology (RDT)
Unsupported features that are ignored:
Mounts other than the root filesystem (we do use
--no-home
)User/group mappings beyond one user mapped to EUID and one group mapped to EGID
Disabling
prctl(PR_SET_NO_NEW_PRIVS)
Root filesystem propagation mode
sysctl
directivesmasked and read-only paths (remaining unprivileged protects you)
Capabilities
rlimits
Devices (all devices are inherited from the host)
cgroups
seccomp
SELinux
AppArmor
Container hostname setting
4.13.5. Environment variables¶
CH_RUN_OCI_LOGFILE
If set, send log chatter to this file. We use a side channel because standard error and standard output may be arbitrarily messed up by the caller.
CH_RUN_OCI_HANG
If set to the name of a command (e.g.,
create
), sleep indefinitely when that command is invoked. The purpose here is to halt a build so it can be examined and debugged.
4.14. ch-ssh¶
Run a remote command in a Charliecloud container.
4.14.1. Synopsis¶
$ CH_RUN_ARGS="NEWROOT [ARG...]"
$ ch-ssh [OPTION...] HOST CMD [ARG...]
4.14.2. Description¶
Runs command CMD
in a Charliecloud container on remote host
HOST
. Use the content of environment variable CH_RUN_ARGS
as
the arguments to ch-run
on the remote host.
Note
Words in CH_RUN_ARGS
are delimited by spaces only; it is not shell
syntax.
4.14.3. Example¶
On host bar.example.com, run the command echo hello
inside a
Charliecloud container using the unpacked image at /data/foo
with
starting directory /baz
:
$ hostname
foo
$ export CH_RUN_ARGS='--cd /baz /data/foo'
$ ch-ssh bar.example.com -- hostname
bar
4.15. ch-tar2dir¶
Unpack an image tarball into a directory.
4.15.1. Synopsis¶
$ ch-tar2dir TARBALL DIR
4.15.2. Description¶
Extract the tarball TARBALL
into a subdirectory of DIR
.
TARBALL
must contain a Linux filesystem image, e.g. as created by
ch-builder2tar
, and be compressed with gzip
or xz
. If
TARBALL
has no extension, try appending .tar.gz
and
.tar.xz
.
Inside DIR
, a subdirectory will be created whose name corresponds to
the name of the tarball with .tar.gz
or other suffix removed. If such
a directory exists already and appears to be a Charliecloud container image,
it is removed and replaced. If the existing directory doesn’t appear to be a
container image, the script aborts with an error.
Additional arguments:
--help
print help and exit
--version
print version and exit
Warning
Placing DIR
on a shared file system can cause significant metadata
load on the file system servers. This can result in poor performance for
you and all your colleagues who use the same file system. Please consult
your site admin for a suitable location.
4.15.3. Example¶
$ ls -lh /var/tmp
total 57M
-rw-r----- 1 reidpr reidpr 57M Feb 13 16:14 hello.tar.gz
$ ch-tar2dir /var/tmp/hello.tar.gz /var/tmp
creating new image /var/tmp/hello
/var/tmp/hello unpacked ok
$ ls -lh /var/tmp
total 57M
drwxr-x--- 22 reidpr reidpr 4.0K Feb 13 16:29 hello
-rw-r----- 1 reidpr reidpr 57M Feb 13 16:14 hello.tar.gz
4.16. ch-tar2squash¶
Create a SquashFS file from a tarball image.
4.16.1. Synopsis¶
$ ch-tar2squash TARBALL OUTDIR [ARGS ...]
4.16.2. Description¶
Create Charliecloud SquashFS file from TARBALL
in directory
OUTDIR
, named as TARBALL
with extension .sqfs
.
Wrapper for ch-tar2dir
and ch-dir2sqfs
.
Additional arguments:
--help
print help and exit
--version
print version and exit
4.16.3. Example¶
$ ch-tar2squash /var/tmp/debian.tar.gz /var/tmp
Parallel mksquashfs: Using 6 processors
Creating 4.0 filesystem on /var/tmp/debian.sqfs, block size 131072.
[...]
-rw-r--r-- 1 charlie charlie 41M Apr 23 14:50 debian.sqfs
$ ls -lh /var/tmp/debian*
total 83M
-rw-r--r-- 1 charlie charlie 41M Apr 23 14:50 debian.sqfs
-rw-rw-r-- 1 charlie charlie 43M Apr 23 14:49 debian.tar.gz
4.17. ch-umount¶
Unmount a FUSE mounted squash filesystem and remove the mount point.
4.17.1. Synopsis¶
$ ch-umount MOUNTDIR
4.17.2. Description¶
Unmount Charliecloud SquashFS file at target directory MOUNTDIR
.
Remove empty MOUNTDIR
after successful unmounting.
Additional arguments:
--help
print help and exit
--version
print version and exit
4.17.3. Example¶
$ ls /var/tmp/debian
bin dev home lib64 mnt proc run srv tmp var
boot etc lib media opt root sbin sys usr WEIRD_AL_YANKOVIC
$ ch-umount /var/tmp/debian
unmounted and removed /var/tmp/debian
$ ls /var/tmp/debian
ls: cannot access /var/tmp/debian: No such file or directory