Primer is a flexible OS initialisation automator in pure POSIX shell. The main goal is to bring up barebone minimum OS installations to a minimal set of features for running containerised applications. Primer automates a number of installation steps, each of which aiming at the installation of an OS level feature with minimum dependencies. New steps can easily be written if necessary, as they interact with primer using a well-defined interface. As such, primer can also be used in other domains than cloud first applications. Being written in POSIX compatible shell makes it suitable within the embedded space, for example.
The main target OSes of primer are minimal linux OSes such as Alpine Linux, Ubuntu cloud or server images, Fedora or the now defunct Clear Linux*. Primer is probably best fitted for initialising "on the metal", rather than VMs where cloud-init often fills such a role.
Primer will prep your system automatically with configuration descriptions that can be put under revision control. However, primer has no support for dependencies between the various steps, nor has a DSL: it rather express machine configuration through a set of environment variables. Primer tries to fill the gap between home-made initialisation scripts and larger solutions such as Ansible, puppet or cloud-init.
[!NOTE] Primer is still under development and a little bit of a moving target. But it has reached enough stability to start being used on real (or virtual) hosts, rather that test containers. Thanks to LLMs, there is now an extensive test suite. Please raise issues, or even better, fix bugs through PRs.
Primer is perhaps best explained through an example, scroll past this section for documentation. As you probably do not want to pollute your host system just for testing primer, the example is a little bit more convoluted than necessary to confine primer to a Docker container and not harm the main host. Let's simulate the installation of the Docker daemon and client on an Alpine host by running primer in an Alpine container. To do this, run the following command from the main directory of the project:
docker run \
-it \
--rm \
-v $(pwd):/primer:ro \
--entrypoint /primer/primer \
alpine \
-s "docker"This works by creating a transient alpine Docker container,
in which the main directory of the project is bind-mounted onto the /primer
directory in the container (the -v option to docker).
In that container, we make primer the entrypoint (given the bind-mount,
primer is located at /primer/primer in the container).
All options following the name of the image alpine are then passed to
primer.
In this example, we specify a single step, called docker.
Its implementation is part of the default set of
steps.
The command should output something similar to the following. In your terminal, this is most likely going to be pretty printed with a few colours instead:
[20200428-200338] [primer] [notice] Installing steps: docker
[20200428-200338] [primer] [ info ] Loading docker implementation from /primer/libexec/steps/docker.sh
[20200428-200338] [primer] [ info ] Primer step install>>> docker
[20200428-200338] [primer] [ info ] Installing packages: docker
[20200428-200338] [primer] [ info ] Updating OS package indices (if relevant)
[20200428-200354] [primer] [ info ] Starting Docker daemon
[20200428-200354] [primer] [notice] Service docker start is not relevant in a container
[20200428-200354] [primer] [ info ] Enabling docker daemon at start
[20200428-200354] [primer] [notice] Service docker enable is not relevant in a containerPrimer finds out the implementation of the step called docker at
/primer/libexec/steps/docker.sh and
start requesting the implementation to install.
The implementation at docker.sh is loaded into the main process on demand.
It needs to have a function called with the same name,
prefixed with primer_step_, i.e. primer_step_docker.
The function might be called a few times under the installation procedure.
By default, the current implementation installs the Docker daemon and client,
and arranges for the daemon to start right now and autostart with the OS.
This is the only feature that actually do not work when running in a container.
Finally, the implementation makes sure that the user calling the script,
if not root is made part of the group docker so that it will be able to call
the client with elevating privileges.
All other regular users in the system can also be made part of that group if
necessary.
This is a security risk, but is optional and
relevant when operating on unmanned servers.
As the container automatically dies and disappear once primer has finished,
it is hard to verify what happened.
You can however start the installation of another step called forever after
the step called docker.
forever is a dummy step that will sleep forever when requested to
be installed.
It is implemented here.
Modify the command example above by giving the following option to primer:
-s "docker forever"primer will executes both steps in sequence and
forever will have the side effect that the container remains up and running,
doing mostly nothing.
You should then be able to docker exec -it into it from another terminal and
poke around.
The docker installation step can do much more.
It is for example able to register at a Docker registry for one (the caller) or
some seleted users, and
would install Docker directly using the official installation
script on other distributions.
The script, as it is downloaded from the Internet,
will be checked for integrity and
on Debian derivatives the GPG signature of the official Docker repository that
it adds to the system will be validated.
Provided you have an account at gitlab.com,
the following would install Docker in an ubuntu container instead, and
login the main user (again root in the case of a container) at gitlab (you
will have to provide your credentials!).
Note the default is to verify the commit signature of the Docker installation
script against the one hard-coded into the docker.sh implementation.
If the official Docker script changes, the docker step in primer will fail.
docker run \
-it \
--rm \
-v $(pwd):/primer:ro \
--entrypoint /primer/primer \
ubuntu \
-s "docker" \
--docker:registry youruser:XXXXX@registry.gitlab.comIn the command above, note the specially formatted option --docker:registry.
This option is parsed by primer so that what appears before the separating :
(colon sign) is the name of a step to be looked up (in our case,
the same step as above, i.e. docker).
The remaining forms an option that will be blindly passed to the
primer_step_docker function in the implementation.
So, in this case, the function called primer_step_docker,
implemented as part of docker.sh will be called as follows.
Note that registry automatically became a double-dashed option --registry.
primer_step_docker option --registry youruser:XXXXX@registry.gitlab.comMost steps can be communicated with this way.
Modifying their behaviour can also be done through setting environment variables
that start with the same name as the step, but in uppercase,
i.e. the docker step can be controlled by a series of environment variables
that are called PRIMER_STEP_DOCKER_.
The value of command line options have precedence over these variables.
To finish your exploration of the possibilities of primer,
run the following command instead.
It creates a few users on the system,
they will be made part of the docker group automatically as part of the
docker step.
In this example,
primer takes the list of steps to perform from the PRIMER_STEPS environment
variable instead.
PRIMER_STEPS="users docker" \
docker run \
-it \
--rm \
-e PRIMER_STEPS \
-v $(pwd):/primer:ro \
--entrypoint /primer/primer \
ubuntu \
--users:db /primer/test/data/users.db \
--docker:access ".*"Primer sports a number of options to control its behaviour, but
also recognises environment variables constructed the same way,
i.e. all starting with PRIMER_.
Of interest might be the option called --config which should point to a .env
formatted file.
By default, and if it exists,
primer will read the file called primer.env from the current directory when it
is called.
As this file can contain any number of environment variables,
including PRIMER_STEPS to control which steps to execute and in which order,
the file forms a "contract" describing what to install on a particular host.
Placing relevant .env formatted files under revision control,
combined with the ability for primer not only to install, but
also clean a system,
makes it possible to operate on existing OS installations in a reproducible way.
Cleaning only removes what primer can prove it made, e.g. firewall rules carrying
its marker, or files still equal to what it would install, so it never removes
what existed before. Clean with the same options as when installing.
At the time of writing, primer already have the following steps implemented. This list will grow with personal needs, or with the help of the community. PRs are welcome! Some inspiration can be taken from the config implementation in cloud-init. How primer drives steps, and the rules steps follow, is described in the step lifecycle.
packagesupgrades the system to the latest and installs additional packages.userstakes a/etc/passwdinspired colon separated file to create a number of users on the host system.- Relevant groups will be created and all users can be made members of
additional groups (e.g.
sudo?). - The module is able to generate strong passwords for all these users if necessary.
- This module is also able to fetch public keys from GitHub handles to automatically allow for SSH access.
- Relevant groups will be created and all users can be made members of
additional groups (e.g.
gitinstalls git on the system, with LFS support.announceinstalls and enables network announcement services so the host is discoverable by hostname on the local network, via mDNS and/or NetBIOS.dynufwinstalls ufw and a simplification wrapper on top of ufw.- The wrapper is able to track host names that would change their pointed IP with time (dynamic DNS).
firewallopens ports in the firewall of the host, detecting ufw, firewalld, awall or nftables and falling back to iptables.timezoneplaces the host at a given location.localesets the locale of the system, generating it when needed.sshkeysautomatically generates strong SSH keys for the calling user.- The target of this module are deploy keys when interacting with automated CI systems.
dockerinstalls the Docker daemon and client, together with the Docker compose and buildx plugins, it has been described above.composeis deprecated, Docker compose is now a CLI plugin installed by thedockerstep.machineinstalls Docker machine at the latest or a specific version, together with bash completion.- The binary integrity can be verified before installation.
machineryinstalls machinery from a given branch (defaults tomaster).lazydockerinstalls lazydocker at the latest or a specific version.- Lazydocker is a CUI for Docker.
- Binary integrity is verified before installation.
microinstalls micro at the latest or a specific version.- micro is a modern editor for the terminal.
githubinstalls binaries from project releases (assets) at github.- Downloads are verified against checksums, given directly or published with the release.
diskformats and mounts attached disks.wificonnects the host to a wifi network at boot, using ifupdown.installinstalls (recursively) remote or local files onto the host.installbenefits from the--curloption for authenticated access to remote resources, when necessary.
freeformlooks up and executes binaries in alphabetical order to perform free-form installation/removal operations.
Primer recognises both single-dash led short options and double-dash led long options. Long options can be separated from their values using both a space or an equal sign. After all options come a command instructing Primer what to do, the default being to perform installation of all steps, no questions asked. As an example of setting options, all styles, consider the following invocations that are all equivalent and set verbosity to debug before printing the help:
./primer -v debug help
./primer --verbose debug help
./primer --verbose=debug helpColon separated list of directories where to look for the implementation of
steps.
The default is the steps directory in the libexec directory.
Note that when amalgamated,
the default is empty as all standard steps are already part of the amalgamation.
Space separated list of paths to configuration files to read in order,
in .env format.
These files should contain a number of environment variables to specify the
steps (and their order) that primer will perform,
together with variables to change the options of these steps.
In those files, all empty lines or
lines starting with a # (hash mark) will be ignored.
Variables should be separated from their values by an equal sign = and
values should not be quoted.
The default is to substitute the value of existing (or set through
the file) variables by their value.
Substitution uses an almost eval-safe implementation for security and
supports all POSIX expansion formats.
The default for this option is to read:
- The file
primer.envin the current working directory, if present. - The file
primer_<mac>.envin the current working directory, if present. - It will be read on "top" of the previous file.
- In that filename, at run time,
<mac>will be the MAC address of the main Ethernet interface of the host, in lower-case and without any field separators.
Primer is not only able to read files, but also remote resources, over
https://.
Plain http:// is refused: anyone on the way could change the configuration,
and thus the commands that primer runs as root.
To access protected resources,
you can combine with using the --curl option.
Variables that change how commands are run cannot be set from configuration
files, and are ignored with a warning:
PATH, IFS, ENV, BASH_ENV, CDPATH, PS4, SHELLOPTS,
LD_* and PRIMER_OS_*.
In path or resource specifications passed to --config,
a number of tokens will automatically be replaced by their value at run time.
These are:
%mac%will be replaced with the MAC address of the host, in lower case, without any separator signs.%host%and%hostname%will be replaced with the hostname of the host, as reported by the commandhostname.
These defaults and
tokens allow to use common primer settings across several (similar) host
installations and in networked settings.
Generic options for all hosts would be placed in the main primer.env file,
while host-specific options would be present in the resources using the %mac%
token.
To discover the MAC address that primer believes is the one of the main Ethernet
interface, you can run the following command:
./primer -v notice env|grep -E '\s+mac:'|awk '{print $2}'|tr -d ':'When this flag is present,
references to environment variables in .env files will not be expanded to
their value at run time.
The value of this option should be the space separated list of steps to
consider.
When installing or cleaning, steps will be run in that order, and
the value of this option would override any value that would come from an
environment variable, e.g. read as a .env file.
The value of this option is the path to a file matching URLs to their curl
options.
In this file, blank lines and
lines starting with a # (hash mark) will be ignored.
Otherwise,
the first item of the line should be an extended regular expression to match
against a URL, and
the remaining options that will be passed to curl when it is used to
download URLs matching the expression.
Only the first matching line is used.
The expression itself can be URL encoded to arrange for all special characters,
including possible spaces.
Options are quoted as in the shell, e.g.
-H "Authorization: Bearer xxx", without variable or command substitution.
They are given to curl as a configuration file on its standard input, so
they do not show in the list of processes.
This feature can be used, for example, for debugging purposes, or
to provide authentication details to curl to access URLs:
^https://api%5C.github%5C.com/ -H "Authorization: Bearer ghp_xxx"
^https://files%5C.example%5C.com/ -u "user:pass word" --retry 3
Any long option that looks like the above,
where <step> is the name of an existing step, and
<opt> is one of the options supported by the implementation of the step will
be passed further as --<opt> to the main function in the implementation of the
step.
These values have precedence over any value that would have been set through
environment variables, e.g. read through an .env file.
Change verbosity. Available levels are, from most to least verbose, and as of the yu.sh implementation:
tracedebuginfo(default)noticewarnerror
Do not colour the log messages. --no-color is an alias.
Print the version and exit.
Print help and exit.
Apart from the options, Primer takes a trailing command.
The trailing command blindly defaults to install,
meaning that the default behaviour will be to install as specified per the
--steps command-line option or PRIMER_STEPS environment variable.
Primer accepts a number of aliases to those commands:
install,up: This is the default command, it will perform all the installation steps specified through thePRIMER_STEPSenvironment variable or the--stepsoption.clean,remove,down: Uninstall the steps specified as for theinstallcommand.env,environment: List all known and accessible steps, together with the options and environment variables that can be used to modify their behaviour.- This also prints out host information, e.g. hostname, MAC address (of the primary ethernet interface), name of distribution, version, etc.
info: List all options and environment variables that can be used to modify the behaviour of the steps to be run.- The list of steps can be specified as for the
installcommand help: Print help and exit.
Primer recognises a number of environment variables,
some of which associated to its command-line options.
Command-line options always have precedence over the value coming from
environment variables.
All variables start with PRIMER_.
They are:
PRIMER_PATHis the same as the--pathoption.PRIMER_EXTSis a space-separated list of dot-led extensions to add to the name of the step when looking for their implementations withinPRIMER_PATH.- It defaults to
.sh. PRIMER_STEPSis the same as the--stepsoption.PRIMER_CONFIGis the same as the--configoption.PRIMER_LOCALis the root directory to use as/usr/localwhen manuall installing contrib packages of various sorts.- It defaults to
/usr/local. PRIMER_BINDIRis thebindirectory underPRIMER_LOCAL.PRIMER_LIBDIRis thelibdirectory underPRIMER_LOCAL.PRIMER_OPTDIRis theoptdirectory underPRIMER_LOCAL.PRIMER_CURL_OPTIONSis the same as the--curloption.
In addition, steps will also recognise environment variables,
these all start with PRIMER_STEP_.
More details are available in the conventions document.
Provided a release version, you can download and
run primer directly using curl, as long as it is installed.
For example, the following command would download release v0.2.0 and
show the help message.
curl -sSL \
https://github.com/efrecon/primer/releases/download/v0.2.0/primer-v0.2.0.sh |
sh -s - --helpThe current implementation depends on a number of internal modules, on the utility library yu.sh and a predefined set of steps, even though it is possible to create and integrate more steps. To make it easier to ship the script to "raw" servers, primer supports amalgamation. To create a single binary that can easily be copied to a target machine, run the following commands from the root directory of the project:
./libexec/yu.sh/bin/amalgamation.sh primer > primer.sh
chmod a+x ./primer.shPrimer comes with a test suite written in plain POSIX shell, so it needs no
other tool than docker or podman. All tests run in throwaway containers, so
they can modify the system freely without touching your host. The suite uses the
first of docker and podman that is installed and working. Set
PRIMER_TEST_RUNTIME, or use -r, to pick one explicitly. Run it from the
main directory of the project:
./test/run.shThe suite contains:
- Unit tests for every function of the library modules in libexec,
in
test/unit_<module>.test.sh. They source primer in library mode, call the functions directly in a container and replace external commands with mocks when needed, e.g. to emulate firewalls or services. In library mode, i.e. when sourced withPRIMER_LIBRARY=1, primer loads all its modules and stops before parsing options;PRIMER_ROOT_DIRcan point to its installation directory. Both variables are only for sourcing: when executed withPRIMER_LIBRARY=1, primer refuses to run, and it always ignoresPRIMER_ROOT_DIR. - Tests for every installation step, in
test/step_<step>.test.sh. Most run primer for real in Alpine and Debian containers, some also in Ubuntu and Fedora ones, and verify the resulting state. Steps that cannot fully work in a container, e.g.wifi, are tested with mocks. Some tests, e.g.dockerandfirewall, need privileged containers; they are skipped, with a reason, with rootless podman or docker. - Tests of the command-line interface and of the functions of
primeritself, intest/primer.test.sh. - Tests of the test framework, in
test/lib.test.sh.
Tests are named file:function. Any argument to run.sh is an extended regular
expression that selects tests by name, and -l lists them. When run without
patterns, the suite also checks that every function of primer and of the
library modules, and every step, has tests. Options come before the patterns.
Run ./test/run.sh -h for all options:
./test/run.sh -l # List all tests
./test/run.sh '^unit_' # Only unit tests
./test/run.sh -j 8 step_docker # Docker step tests, 8 in parallel
./test/run.sh -t # TAP output, as in CIOnly the files known to git, and new files that git does not ignore, are copied
into the containers: a primer.env of yours, which primer would read, never
is. Tests running longer than 20 minutes fail (-T to change). Containers are
labelled with primer-test, and removed when the suite is interrupted.
On GitHub Actions, detected through GITHUB_ACTIONS, a summary is also written
to the page of the run (GITHUB_STEP_SUMMARY): the number of passed, failed and
skipped tests, the failed tests with the end of their logs, and the reasons of
the skipped ones.
Some steps download releases from GitHub. Set GITHUB_TOKEN to make these
calls authenticated and avoid anonymous rate limits.
Most of the time goes to installing packages, which is CPU-bound. By default,
the suite runs as many tests in parallel as the container runtime has CPUs.
Debian, Ubuntu and Fedora tests run in warm copies of the official images,
built once a day from test/assets: their package lists are
fetched, apt keeps its caches, common packages are downloaded (not installed)
and documentation is not installed. Primer still has to install everything it
needs. Set PRIMER_TEST_WARM=0 to use the official images.
You are more than welcome to contribute through PRs. New steps will not be accepted without minimal test rules or documentation. There are a few coding conventions to follow for all contributors.
The original idea for this utility comes from a number of projects at Yanzi, including some of the code. The code has since then, undergone large modifications up to a point where little of the original is left.
Most of the design, refactoring and rewriting has been done on my free-time. A few enhancements have been sponsored by Lindborg Systems AB and GpsGate.