X Tutup
Skip to content

Commit b4376e9

Browse files
dweomerdweomer
committed
Update Vagrantfile for testing SELinux
`vagrant up` will build and install containerd and all dependencies, setting up proper SELinux contexts on the runc and containerd binaries. The VM is configured to be SELinux Enforcing by default but this gets changed during various CI passes via a matrix param to Disabled and Permissive before running tests. I have an open PR to fix the container-selinux policy for containerd at containers/container-selinux#98 which once accepted we will want to update the CI matrix to use Enforcing mode instead of Permissive. All tests currently pass in SELinux permissive mode with containerd configured with `enable_selinux=true`. To see which tests are failing with SELinux enforcing and an already spun up VM: `SELINUX=Enforcing vagrant up --provision-with=selinux,test-cri` To test SELinux enforcing in a new VM: `vagrant destroy -force; SELINUX=Enforcing vagrant up --provision-with=shell,selinux,test-cri` The `selinux` shell provisioner, parameterized by the SELINUX envvar, will configure the system as you would expect, with the side effect that containerd is configured with `enable_selinux=true` via `/etc/containerd/config.toml` for Permissive or Enforcing modes and `enable_selinux=false` when SELINUX=Disabled. Provided that virtualization is suported, this Vagrantfile and provisioners make it easy to test containerd/cri for conformance under SELinux on non-SELinux systems. Signed-off-by: Jacob Blain Christen <jacob@rancher.com>
1 parent 23934e8 commit b4376e9

File tree

6 files changed

+304
-68
lines changed

6 files changed

+304
-68
lines changed

‎.github/workflows/ci.yml‎

Lines changed: 13 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -423,7 +423,7 @@ jobs:
423423
test $TEST_RC -eq 0 || /bin/false
424424
425425
cgroup2:
426-
name: CGroups v2 Integration Test
426+
name: CGroupsV2 and SELinux Integration
427427
# nested virtualization is only available on macOS hosts
428428
runs-on: macos-10.15
429429
timeout-minutes: 40
@@ -436,12 +436,20 @@ jobs:
436436
uses: actions/checkout@v2
437437

438438
- name: Start vagrant
439-
env:
440-
RUNC_FLAVOR: ${{ matrix.runc }}
441439
run: vagrant up
442440

443441
- name: Integration
444-
run: vagrant ssh default -- sudo -i /integration.sh
442+
env:
443+
RUNC_FLAVOR: ${{ matrix.runc }}
444+
# SELinux: replace Permissive with Enforcing after https://github.com/containers/container-selinux/pull/98
445+
# is merged and the package becomes generally available.
446+
SELINUX: Permissive
447+
run: vagrant up --provision-with=selinux,install-runc,test-integration
445448

446449
- name: CRI test
447-
run: vagrant ssh default -- sudo -i /critest.sh
450+
env:
451+
RUNC_FLAVOR: ${{ matrix.runc }}
452+
# SELinux: replace Permissive with Enforcing after https://github.com/containers/container-selinux/pull/98
453+
# is merged and the package becomes generally available.
454+
SELINUX: Permissive
455+
run: vagrant up --provision-with=selinux,install-runc,test-cri

‎Vagrantfile‎

Lines changed: 203 additions & 62 deletions
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,7 @@
1515
# See the License for the specific language governing permissions and
1616
# limitations under the License.
1717

18-
# Vagrantfile for cgroup2
18+
# Vagrantfile for cgroup2 and SELinux
1919
Vagrant.configure("2") do |config|
2020
config.vm.box = "fedora/32-cloud-base"
2121
config.vm.provider :virtualbox do |v|
@@ -26,78 +26,219 @@ Vagrant.configure("2") do |config|
2626
v.memory = 2048
2727
v.cpus = 2
2828
end
29-
config.vm.provision "shell", env: {"RUNC_FLAVOR"=>ENV["RUNC_FLAVOR"]}, inline: <<-SHELL
30-
set -eux -o pipefail
31-
# configuration
32-
GO_VERSION="1.13.15"
3329

34-
# install dnf deps
35-
dnf install -y container-selinux gcc git iptables libseccomp-devel lsof make
30+
# Disabled by default. To run:
31+
# vagrant up --provision-with=upgrade-packages
32+
# To upgrade only specific packages:
33+
# UPGRADE_PACKAGES=selinux vagrant up --provision-with=upgrade-packages
34+
#
35+
config.vm.provision "upgrade-packages", type: "shell", run: "never" do |sh|
36+
sh.upload_path = "/tmp/vagrant-upgrade-packages"
37+
sh.env = {
38+
'UPGRADE_PACKAGES': ENV['UPGRADE_PACKAGES'],
39+
}
40+
sh.inline = <<~SHELL
41+
#!/usr/bin/env bash
42+
set -eux -o pipefail
43+
dnf -y upgrade ${UPGRADE_PACKAGES}
44+
SHELL
45+
end
3646

37-
# install Go
38-
curl -fsSL "https://dl.google.com/go/go${GO_VERSION}.linux-amd64.tar.gz" | tar Cxz /usr/local
47+
# To re-run, installing CNI from RPM:
48+
# INSTALL_PACKAGES="containernetworking-plugins" vagrant up --provision-with=install-packages
49+
#
50+
config.vm.provision "install-packages", type: "shell", run: "once" do |sh|
51+
sh.upload_path = "/tmp/vagrant-install-packages"
52+
sh.env = {
53+
'INSTALL_PACKAGES': ENV['INSTALL_PACKAGES'],
54+
}
55+
sh.inline = <<~SHELL
56+
#!/usr/bin/env bash
57+
set -eux -o pipefail
58+
dnf -y install \
59+
container-selinux \
60+
curl \
61+
gcc \
62+
git \
63+
iptables \
64+
libseccomp-devel \
65+
libselinux-devel \
66+
lsof \
67+
make \
68+
${INSTALL_PACKAGES}
69+
SHELL
70+
end
3971

40-
# setup env vars
41-
cat >> /etc/environment <<EOF
72+
# To re-run this provisioner, installing a different version of go:
73+
# GO_VERSION="1.14.6" vagrant up --provision-with=install-golang
74+
#
75+
config.vm.provision "install-golang", type: "shell", run: "once" do |sh|
76+
sh.upload_path = "/tmp/vagrant-install-golang"
77+
sh.env = {
78+
'GO_VERSION': ENV['GO_VERSION'] || "1.13.15",
79+
}
80+
sh.inline = <<~SHELL
81+
#!/usr/bin/env bash
82+
set -eux -o pipefail
83+
curl -fsSL "https://dl.google.com/go/go${GO_VERSION}.linux-amd64.tar.gz" | tar Cxz /usr/local
84+
cat >> /etc/environment <<EOF
4285
PATH=/usr/local/go/bin:$PATH
4386
GO111MODULE=off
4487
EOF
45-
source /etc/environment
46-
cat >> /etc/profile.d/sh.local <<EOF
88+
source /etc/environment
89+
cat >> /etc/profile.d/sh.local <<EOF
4790
GOPATH=\\$HOME/go
4891
PATH=\\$GOPATH/bin:\\$PATH
4992
export GOPATH PATH
5093
EOF
5194
source /etc/profile.d/sh.local
95+
SHELL
96+
end
5297

53-
# enter /root/go/src/github.com/containerd/containerd
54-
mkdir -p /root/go/src/github.com/containerd
55-
ln -s /vagrant /root/go/src/github.com/containerd/containerd
56-
cd /root/go/src/github.com/containerd/containerd
57-
58-
# install runc (or crun) and other components
59-
./script/setup/install-runc
60-
./script/setup/install-cni
61-
./script/setup/install-critools
62-
63-
# install containerd
64-
make BUILDTAGS="seccomp selinux no_aufs no_btrfs no_devmapper no_zfs" binaries install
65-
66-
# FIXME: enable SELinux
67-
setenforce 0
68-
umount /sys/fs/selinux
69-
70-
# create the daemon config
71-
mkdir -p /etc/containerd
72-
cat > /etc/containerd/config.toml <<EOF
73-
version = 2
74-
[plugins]
75-
[plugins."io.containerd.grpc.v1.cri"]
76-
# FIXME: enable SELinux
77-
enable_selinux = false
78-
EOF
98+
config.vm.provision "setup-gopath", type: "shell", run: "once" do |sh|
99+
sh.upload_path = "/tmp/vagrant-setup-gopath"
100+
sh.inline = <<~SHELL
101+
#!/usr/bin/env bash
102+
source /etc/environment
103+
source /etc/profile.d/sh.local
104+
set -eux -o pipefail
105+
mkdir -p ${GOPATH}/src/github.com/containerd
106+
ln -fnsv /vagrant ${GOPATH}/src/github.com/containerd/containerd
107+
SHELL
108+
end
109+
110+
config.vm.provision "install-runc", type: "shell", run: "once" do |sh|
111+
sh.upload_path = "/tmp/vagrant-install-runc"
112+
sh.env = {
113+
'RUNC_FLAVOR': ENV['RUNC_FLAVOR'] || "runc",
114+
}
115+
sh.inline = <<~SHELL
116+
#!/usr/bin/env bash
117+
source /etc/environment
118+
source /etc/profile.d/sh.local
119+
set -eux -o pipefail
120+
${GOPATH}/src/github.com/containerd/containerd/script/setup/install-runc
121+
type runc
122+
runc --version
123+
chcon -v -t container_runtime_exec_t $(type -ap runc)
124+
SHELL
125+
end
126+
127+
config.vm.provision "install-cni", type: "shell", run: "once" do |sh|
128+
sh.upload_path = "/tmp/vagrant-install-cni"
129+
sh.env = {
130+
'CNI_BINARIES': 'bridge dhcp flannel host-device host-local ipvlan loopback macvlan portmap ptp tuning vlan',
131+
}
132+
sh.inline = <<~SHELL
133+
#!/usr/bin/env bash
134+
source /etc/environment
135+
source /etc/profile.d/sh.local
136+
set -eux -o pipefail
137+
${GOPATH}/src/github.com/containerd/containerd/script/setup/install-cni
138+
PATH=/opt/cni/bin:$PATH type ${CNI_BINARIES} || true
139+
SHELL
140+
end
141+
142+
config.vm.provision "install-cri-tools", type: "shell", run: "once" do |sh|
143+
sh.upload_path = "/tmp/vagrant-install-cri-tools"
144+
sh.env = {
145+
'CRI_TOOLS_VERSION': ENV['CRI_TOOLS_VERSION'] || '16911795a3c33833fa0ec83dac1ade3172f6989e',
146+
'GOBIN': '/usr/local/bin',
147+
}
148+
sh.inline = <<~SHELL
149+
#!/usr/bin/env bash
150+
source /etc/environment
151+
source /etc/profile.d/sh.local
152+
set -eux -o pipefail
153+
${GOPATH}/src/github.com/containerd/containerd/script/setup/install-critools
154+
type crictl critest
155+
critest --version
156+
SHELL
157+
end
158+
159+
config.vm.provision "install-containerd", type: "shell", run: "once" do |sh|
160+
sh.upload_path = "/tmp/vagrant-install-containerd"
161+
sh.inline = <<~SHELL
162+
#!/usr/bin/env bash
163+
source /etc/environment
164+
source /etc/profile.d/sh.local
165+
set -eux -o pipefail
166+
cd ${GOPATH}/src/github.com/containerd/containerd
167+
make BUILDTAGS="seccomp selinux no_aufs no_btrfs no_devmapper no_zfs" binaries install
168+
type containerd
169+
containerd --version
170+
chcon -v -t container_runtime_exec_t /usr/local/bin/{containerd,containerd-shim*}
171+
./script/setup/config-containerd
172+
SHELL
173+
end
174+
175+
# SELinux is Enforcing by default.
176+
# To set SELinux as Disabled on a VM that has already been provisioned:
177+
# SELINUX=Disabled vagrant up --provision-with=selinux
178+
# To set SELinux as Permissive on a VM that has already been provsioned
179+
# SELINUX=Permissive vagrant up --provision-with=selinux
180+
config.vm.provision "selinux", type: "shell", run: "never" do |sh|
181+
sh.upload_path = "/tmp/vagrant-selinux"
182+
sh.env = {
183+
'SELINUX': ENV['SELINUX'] || "Enforcing"
184+
}
185+
sh.inline = <<~SHELL
186+
/vagrant/script/setup/config-selinux
187+
/vagrant/script/setup/config-containerd
188+
SHELL
189+
end
190+
191+
# SELinux is permissive by default (via provisioning) in this VM. To re-run with SELinux enforcing:
192+
# vagrant up --provision-with=selinux-enforcing,test-integration
193+
#
194+
config.vm.provision "test-integration", type: "shell", run: "never" do |sh|
195+
sh.upload_path = "/tmp/test-integration"
196+
sh.env = {
197+
'RUNC_FLAVOR': ENV['RUNC_FLAVOR'] || "runc",
198+
}
199+
sh.inline = <<~SHELL
200+
#!/usr/bin/env bash
201+
source /etc/environment
202+
source /etc/profile.d/sh.local
203+
set -eux -o pipefail
204+
rm -rf /var/lib/containerd-test /run/containerd-test
205+
cd ${GOPATH}/src/github.com/containerd/containerd
206+
make integration EXTRA_TESTFLAGS="-no-criu -test.v" TEST_RUNTIME=io.containerd.runc.v2 RUNC_FLAVOR=$RUNC_FLAVOR
207+
SHELL
208+
end
209+
210+
# SELinux is permissive by default (via provisioning) in this VM. To re-run with SELinux enforcing:
211+
# vagrant up --provision-with=selinux-enforcing,test-cri
212+
#
213+
config.vm.provision "test-cri", type: "shell", run: "never" do |sh|
214+
sh.upload_path = "/tmp/test-cri"
215+
sh.env = {
216+
'CRITEST_ARGS': ENV['CRITEST_ARGS'],
217+
}
218+
sh.inline = <<~SHELL
219+
#!/usr/bin/env bash
220+
source /etc/environment
221+
source /etc/profile.d/sh.local
222+
set -eux -o pipefail
223+
systemctl disable --now containerd || true
224+
rm -rf /var/lib/containerd /run/containerd
225+
function cleanup()
226+
{
227+
journalctl -u containerd > /tmp/containerd.log
228+
systemctl stop containerd
229+
}
230+
selinux=$(getenforce)
231+
if [[ $selinux == Enforcing ]]; then
232+
setenforce 0
233+
fi
234+
systemctl enable --now ${GOPATH}/src/github.com/containerd/containerd/containerd.service
235+
if [[ $selinux == Enforcing ]]; then
236+
setenforce 1
237+
fi
238+
trap cleanup EXIT
239+
ctr version
240+
critest --parallel=$(nproc) ${CRITEST_ARGS}
241+
SHELL
242+
end
79243

80-
# create /integration.sh
81-
cat > /integration.sh <<EOF
82-
#!/bin/bash
83-
set -eux -o pipefail
84-
cd /root/go/src/github.com/containerd/containerd
85-
make integration EXTRA_TESTFLAGS=-no-criu TEST_RUNTIME=io.containerd.runc.v2 RUNC_FLAVOR=$RUNC_FLAVOR
86-
EOF
87-
chmod +x /integration.sh
88-
89-
# create /critest.sh
90-
cat > /critest.sh <<EOF
91-
#!/bin/bash
92-
set -eux -o pipefail
93-
containerd -log-level debug &> /tmp/containerd-cri.log &
94-
critest --runtime-endpoint=unix:///var/run/containerd/containerd.sock --parallel=2
95-
TEST_RC=\\$?
96-
test \\$TEST_RC -ne 0 && cat /tmp/containerd-cri.log
97-
pkill containerd
98-
rm -rf /etc/containerd
99-
exit \\$TEST_RC
100-
EOF
101-
chmod +x /critest.sh
102-
SHELL
103244
end

‎script/setup/config-containerd‎

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,36 @@
1+
#!/usr/bin/env bash
2+
3+
# Copyright The containerd Authors.
4+
5+
# Licensed under the Apache License, Version 2.0 (the "License");
6+
# you may not use this file except in compliance with the License.
7+
# You may obtain a copy of the License at
8+
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
11+
# Unless required by applicable law or agreed to in writing, software
12+
# distributed under the License is distributed on an "AS IS" BASIS,
13+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
# See the License for the specific language governing permissions and
15+
# limitations under the License.
16+
17+
#
18+
# establishes /etc/containerd/config.toml
19+
# parameterized by the current SELinux mode
20+
#
21+
set -eux -o pipefail
22+
23+
enable_selinux=false
24+
25+
if type -p getenforce &>/dev/null && [[ $(getenforce) != Disabled ]]; then
26+
enable_selinux=true
27+
fi
28+
29+
mkdir -p /etc/containerd
30+
31+
cat << EOF | sudo tee /etc/containerd/config.toml
32+
version = 2
33+
[plugins]
34+
[plugins."io.containerd.grpc.v1.cri"]
35+
enable_selinux = ${enable_selinux}
36+
EOF

0 commit comments

Comments
 (0)
X Tutup