K8S/OPENSHIFT: POD CRASHING,SECURITY CONTEXT CONTRAINTS PROBLEM?

 

Your new daily given task is creating a new app in a new namespace. You investigated and found your image. Then you started to create your app. After app created you checked and realized that the your new apps pod is not starting!

Here look at this example. I Used gitlab here as an instance.

[ozgurk@myworkstation ~]$ oc new-app --name gitlab \
>    --docker-image quay.io/redhattraining/gitlab-ce:8.4.3-ce.0
--> Creating resources ...
    imagestream.image.openshift.io "gitlab" created
    deployment.apps "gitlab" created
    service "gitlab" created
--> Success

 Until here, everything looking normal. Check your pod status : 

[ozgurk@myworkstation ~]$ oc get pods
NAME                      READY   STATUS   RESTARTS   AGE
gitlab-6d61db3479-yyjl   0/1     Error    1          43s

As seen above, our pod is in a trouble. It's better to start investigation from pod logs.

[ozgurk@myworkstation ~]$ oc logs pod/gitlab-6d61db3479-yyjl 

================================================================================
Recipe Compile Error in /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/default.rb
================================================================================

Chef::Exceptions::InsufficientPermissions
-----------------------------------------
directory[/etc/gitlab] (gitlab::default line 26) had an error: Chef::Exceptions::InsufficientPermissions: Cannot create directory[/etc/gitlab] at /etc/gitlab due to insufficient permissions

As you see above, there is a permission issue occurred. So, next questions should be why and how to correct. 

In normal circumstances,  containerized apps doesn't need any special permission. But some apps need more permission than had their own to live and work. On a baremetal or vm instance you can correct permission problems by changing group, service account or basically chmoding chowning. But in a container, your actions will not work, because your actions will not persist with container due to normal life-cycle of the pods.

At first, you  must login as an admin, if you're already using an admin account, skip this  step:

[ozgurk@myworkstation ~]$ oc login -u myadmin -p myadminpassword
Login successful.

Then run the command below. your pod name will be different  than mine. Use your very own :)

[ozgurk@myworkstation ~]$ oc get pod/gitlab-6d61db3479-yyjl  -o yaml \
| oc adm policy scc-subject-review -f -
RESOURCE                      ALLOWED BY
Pod/gitlab-6d61db3479-yyjl   anyuid

By running the command we learned the required permission to run our pod.

In real, anyuid is a Security Context Constraints. For anyuid,  which means that the pod can run as any user ID available in the container. This  allows containers that require a specific user to run the commands using a specific user ID.

This is a very common scenario and problem. We will not create a spesific user on the container, already we cannot! For tihs reason we  must create a service account bound to a pod.

 To create a new service account and assign the anyuid SCC to it, run ;

[ozgurk@myworkstation ~]$ oc create sa gitlab-sa
serviceaccount/gitlab-sa created

After your service account created, bind it to anyuid SCC by a cluster policy

[ozgurk@myworkstation ~]$ oc adm policy add-scc-to-user anyuid -z gitlab-sa
clusterrole.rbac.authorization.k8s.io/system:openshift:scc:anyuid added: "gitlab-sa"

Assign your service account to your deployment :

[ozgurk@myworkstation ~]$ oc set serviceaccount deployment/gitlab gitlab-sa
deployment.apps/gitlab serviceaccount updated

 The command above will update your deployment,  then your pod will be recreated with new settings.

[ozgurk@myworkstation ~]$ oc get pods
NAME                   READY   STATUS    RESTARTS   AGE
gitlab-43e3e31-yn3ge   1/1     Running   0          15s

Voila! Congratulations, your problem is solved now. Now , you're free to go what you want with your new app.