バッチ処理を cron から k8s cronjob に移行する

Shamshir のスタンドアローン版を k8s(minikube) の cronjob に移植した記録です。

cron でバッチを動かす時の問題点

node, ruby, python 他インタプリンタ系の言語でプログラムを作ると nodenv + npm/yarn, rbenv + bundler, pyenv + pip などのエコシステムを使って環境構築することになります。

これを cron で動かそうとすると ~/.nodenv/shims/node などエコシステム上の実行ファイルを crond に教えてあげる必要が出てきます。このため起動方法を bash -l cron.sh など(-l: Make bash act as if it had been invoked as a login shell (see INVOCATION below) 個人の動作環境をシミュレートするような特別な「忖度」が必要になってくる場合も。

このため 自分の環境では動くのに crontab に設定すると動かない! どうして!? となってしまうことが稀によくあります。(^_^; エコシステムを切り出したコンテナを作ることでこういった問題は解決でき、デプロイ環境が安定します。

もともとコンテナの定義として Content-agnostic, Infrastructure-agnostic という agnostic(= someone who believes that people cannot know whether God exists or not 神様がいるかどうかなんかわかりっこない = 神がいようといまいと関係ない: X に依存しない?) という概念があります。これが、コンテナが Content-agnostic (中身がスクリプトであろうがバイナリであろうが何のプログラミング言語で書かれていようが適切なイメージであれば動く)、Infrastructure-agnostic(物理でも仮想でもLinuxでもMacでも runc や gVisor といった OCI ランタイムがあればそこで差分を吸収して関係なく動く) と呼ばれる所以だと思います。

k8s cronjob はシンプルなバッチ実行環境ですが、プロダクション環境では依存関係などを設定可能な argo などのワークフローに載せることで処理の分割、依存関係の設定、部分実行/再実行がしやすくなるはずです。

k8s cronjob に移植する時の注意点

  • minikube でレジストリからイメージを DL せずローカルのイメージを使う場合( imagePullPolicy: Never )、eval $(minikube docker-env) で minikube の仮想 worker の docker context に切り替えて docker build する必要がある(下記 docker context の切り替えを参照)

docker context の切り替え

1. もともとの docker context は DOCKER ENDPOINT に unix socket 通信を使う

$ docker context ls
NAME        DESCRIPTION                               DOCKER ENDPOINT               KUBERNETES ENDPOINT                   ORCHESTRATOR
default *   Current DOCKER_HOST based configuration   unix:///var/run/docker.sock   https://192.168.49.2:8443 (default)   swarm

$ docker images
REPOSITORY                    TAG         IMAGE ID       CREATED          SIZE
thetitle                      latest      203fc3c16979   25 minutes ago   125MB
thetitle                      v1.0        203fc3c16979   25 minutes ago   125MB
kyagi/thetitle                1.0         22d1a9e10b4a   42 minutes ago   125MB
node                          16-alpine   0e1547c0f4a4   3 weeks ago      110MB
gcr.io/k8s-minikube/kicbase   v0.0.29     64d09634c60d   2 months ago     1.14GB

2. eval $(minikube docker-env) で切り替えると

$ eval $(minikube docker-env)

3. minikube の仮想 worker の docker context になり DOCKER ENDPOINT が tcp://192.168.x.x になる。minikube にイメージを渡すためにはこちらのコンテキストで docker build する必要がある

$ docker context ls
NAME        DESCRIPTION                               DOCKER ENDPOINT           KUBERNETES ENDPOINT                   ORCHESTRATOR
default *   Current DOCKER_HOST based configuration   tcp://192.168.49.2:2376   https://192.168.49.2:8443 (default)   swarm
Warning: DOCKER_HOST environment variable overrides the active context. To use a context, either set the global --context flag, or unset DOCKER_HOST environment variable.

$ docker images
REPOSITORY                                TAG         IMAGE ID       CREATED             SIZE
shamshir                                  v1.0        145688e57875   About an hour ago   165MB
thetitle                                  v1.0        39a4338f4a4f   About an hour ago   125MB
<none>                                    <none>      da981f50f1eb   2 hours ago         125MB
<none>                                    <none>      f062c5aecb69   2 hours ago         125MB
kyagi/kubia                               latest      d5e0f5c0c6f0   13 days ago         906MB
node                                      16-alpine   0e1547c0f4a4   3 weeks ago         110MB
busybox                                   latest      ec3f0931a6e6   3 weeks ago         1.24MB
node                                      14-alpine   755b96824e40   4 weeks ago         119MB
k8s.gcr.io/kube-apiserver                 v1.23.1     b6d7abedde39   2 months ago        135MB
k8s.gcr.io/kube-proxy                     v1.23.1     b46c42588d51   2 months ago        112MB
k8s.gcr.io/kube-scheduler                 v1.23.1     71d575efe628   2 months ago        53.5MB
k8s.gcr.io/kube-controller-manager        v1.23.1     f51846a4fd28   2 months ago        125MB
k8s.gcr.io/etcd                           3.5.1-0     25f8c7f3da61   4 months ago        293MB
k8s.gcr.io/coredns/coredns                v1.8.6      a4ca41631cc7   4 months ago        46.8MB
k8s.gcr.io/pause                          3.6         6270bb605e12   6 months ago        683kB
kubernetesui/dashboard                    v2.3.1      e1482a24335a   8 months ago        220MB
kubernetesui/metrics-scraper              v1.0.7      7801cfc6d5c0   8 months ago        34.4MB
gcr.io/k8s-minikube/storage-provisioner   v5          6e38f40d628d   11 months ago       31.5MB

4. ログアウトするともとの context にもどる

いままでとこれから

いままで)

$ cat cron.sh
#!/usr/bin/bash

set -m
shamshir_pat=<HERE_IS_TOKEN> node shamshir-stand-alone.js --owner kyagi --repo awesome-project --label "releasable" --quorum 2

$ crontab -l
*/15 10-19 * * 1-5 cd /home/ec2-user/git/shamshir/src;  bash -l -c /home/ec2-user/git/shamshir/src/cron.sh

これから)

$ kubectl config current-context
minikube

$ kubectl get cronjobs
NAME       SCHEDULE           SUSPEND   ACTIVE   LAST SCHEDULE   AGE
shamshir   */15 10-19 * * *   False     0        16s             9m49s

$ cat cronjob5.yaml
apiVersion: batch/v1
kind: CronJob
metadata:
  name: shamshir
spec:
  schedule: "*/15 10-19 * * *"
  jobTemplate:
    spec:
      backoffLimit: 5
      ttlSecondsAfterFinished: 100
      template:
        spec:
          containers:
          - name: shamshir
            image: shamshir:v1.0
            env:
              - name: TZ
                value: Asia/Tokyo
              - name: shamshir_pat
                valueFrom:
                  secretKeyRef:
                    name: shamshir
                    key: pat
            imagePullPolicy: Never
          restartPolicy: OnFailure
      parallelism: 1
      completions: 1
  concurrencyPolicy: "Forbid"
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 5

$ cat secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: shamshir
type: Opaque
stringData:
    pat: HERE_IS_TOKEN

$ cat Dockerfile
FROM node:16-alpine

# Create app directory
WORKDIR /app

COPY package*.json ./

RUN npm install

# Bundle app source
COPY . .

WORKDIR /app/src

CMD [ "node", "shamshir-stand-alone.js", "--owner", "kyagi", "--repo", "awesome-project", "--label", "releasable", "--quorum", "2" ]

$ kubectl logs shamshir-27439718-hl7jf
{"level":"info","message":"Shamshir started.","mode":"live","owner":"kyagi","repo":"awesome-project","service":"shamshir","timestamp":"2022-01-31 00:54:22"}
{"level":"info","message":"Shamshir got pulls: 2602,2598,2596,2575,2573,2557,2553,2551,2540,2539,2481,2478,2295,2281,1981,1951,1685","mode":"live","owner":"kyagi","repo":"awesome-project","service":"shamshir","timestamp":"2022-01-31 00:54:22"}
{"level":"info","message":"Shamshir added releasable label to pull/2598.","mode":"live","owner":"kyagi","repo":"awesome-project","service":"shamshir","timestamp":"2022-01-31 00:54:24"}
{"level":"info","message":"Shamshir added releasable label to pull/2596.","mode":"live","owner":"kyagi","repo":"awesome-project","service":"shamshir","timestamp":"2022-01-31 00:54:24"}
{"level":"info","message":"Shamshir added releasable label to pull/2575.","mode":"live","owner":"kyagi","repo":"awesome-project","service":"shamshir","timestamp":"2022-01-31 00:54:25"}
{"level":"info","message":"Shamshir removed releasable label from pull/2573.","mode":"live","owner":"kyagi","repo":"awesome-project","service":"shamshir","timestamp":"2022-01-31 00:54:26"}
{"level":"info","message":"Shamshir added releasable label to pull/2551.","mode":"live","owner":"kyagi","repo":"awesome-project","service":"shamshir","timestamp":"2022-01-31 00:54:28"}
{"level":"info","message":"Shamshir added releasable label to pull/2540.","mode":"live","owner":"kyagi","repo":"awesome-project","service":"shamshir","timestamp":"2022-01-31 00:54:29"}
{"level":"info","message":"Shamshir added releasable label to pull/2539.","mode":"live","owner":"kyagi","repo":"awesome-project","service":"shamshir","timestamp":"2022-01-31 00:54:30"}
{"level":"info","message":"Shamshir added releasable label to pull/2478.","mode":"live","owner":"kyagi","repo":"awesome-project","service":"shamshir","timestamp":"2022-01-31 00:54:31"}
{"level":"info","message":"Shamshir added releasable label to pull/2295.","mode":"live","owner":"kyagi","repo":"awesome-project","service":"shamshir","timestamp":"2022-01-31 00:54:32"}
{"level":"info","message":"Shamshir added releasable label to pull/1951.","mode":"live","owner":"kyagi","repo":"awesome-project","service":"shamshir","timestamp":"2022-01-31 00:54:34"}
{"level":"info","message":"Shamshir finished.","mode":"live","owner":"kyagi","repo":"awesome-project","service":"shamshir","timestamp":"2022-01-31 00:54:34"}

参考情報

minikube.sigs.k8s.io