任务管理
基于K8s Job的流水线设计
- 镜像构建:
- 镜像部署: kubectl 工具镜像
- 虚拟机部署: 在 Docker 容器中配置 Ansible
镜像构建
- registry.k8s.io -> registry.lank8s.cn
- gcr.io -> gcr.lank8s.cn
原生地址: Google Kaniko 仓库地址
转化地址: 如何拉取gcr.io的镜像
提前同步好的镜像:
拉取最新版本的镜像:
docker pull anjia0532/kaniko-project.executor:v1.9.2
docker pull anjia0532/kaniko-project.executor:v1.9.2-debug
手动操作
启动一个deubg环境, 可以看看里面的工具(二进制可执行文件,工具的用法)
docker run -it --rm --entrypoint=/busybox/sh anjia0532/kaniko-project.executor:v1.23.2-debug
/ # ls -l /kaniko/
total 75448
-rwxr-xr-x 1 0 0 10900549 Sep 8 18:23 docker-credential-acr-env
-rwxr-xr-x 1 0 0 8981984 Sep 8 18:22 docker-credential-ecr-login
-rwxr-xr-x 1 0 0 7814415 Sep 8 18:21 docker-credential-gcr
-rwxr-xr-x 1 0 0 35250176 Sep 26 19:27 executor
drwxr-xr-x 3 0 0 4096 Sep 26 19:27 ssl
-rwxr-xr-x 1 0 0 14303232 Sep 26 19:27 warmer
/ # /kaniko/executor -h
Usage:
executor [flags]
executor [command]
Available Commands:
completion Generate the autocompletion script for the specified shell
help Help about any command
version Print the version number of kaniko
Flags:
--build-arg multi-arg type This flag allows you to pass in ARG values at build time. Set it repeatedly for multiple values.
--cache Use cache when building image
--cache-copy-layers Caches copy layers
--cache-dir string Specify a local directory to use as a cache. (default "/cache")
--cache-repo string Specify a repository to use as a cache, otherwise one will be inferred from the destination provided
--cache-run-layers Caches run layers (default true)
--cache-ttl duration Cache timeout in hours. Defaults to two weeks. (default 336h0m0s)
--cleanup Clean the filesystem at the end
--compressed-caching Compress the cached layers. Decreases build time, but increases memory usage. (default true)
-c, --context string Path to the dockerfile build context. (default "/workspace/")
--context-sub-path string Sub path within the given context.
--custom-platform string Specify the build platform if different from the current host
--customPlatform string This flag is deprecated. Please use '--custom-platform'.
-d, --destination multi-arg type Registry the final image should be pushed to. Set it repeatedly for multiple destinations.
--digest-file string Specify a file to save the digest of the built image to.
-f, --dockerfile string Path to the dockerfile to be built. (default "Dockerfile")
--force Force building outside of a container
--force-build-metadata Force add metadata layers to build image
--git gitoptions Branch to clone if build context is a git repository (default branch=,single-branch=false,recurse-submodules=false)
-h, --help help for executor
--ignore-path multi-arg type Ignore these paths when taking a snapshot. Set it repeatedly for multiple paths.
--ignore-var-run Ignore /var/run directory when taking image snapshot. Set it to false to preserve /var/run/ in destination image. (default true)
--image-fs-extract-retry int Number of retries for image FS extraction
--image-name-tag-with-digest-file string Specify a file to save the image name w/ image tag w/ digest of the built image to.
--image-name-with-digest-file string Specify a file to save the image name w/ digest of the built image to.
--insecure Push to insecure registry using plain HTTP
--insecure-pull Pull from insecure registry using plain HTTP
--insecure-registry multi-arg type Insecure registry using plain HTTP to push and pull. Set it repeatedly for multiple registries.
--kaniko-dir string Path to the kaniko directory, this takes precedence over the KANIKO_DIR environment variable. (default "/kaniko")
--label multi-arg type Set metadata for an image. Set it repeatedly for multiple labels.
--log-format string Log format (text, color, json) (default "color")
--log-timestamp Timestamp in log output
--no-push Do not push the image to the registry
--no-push-cache Do not push the cache layers to the registry
--oci-layout-path string Path to save the OCI image layout of the built image.
--push-retry int Number of retries for the push operation
--registry-certificate key-value-arg type Use the provided certificate for TLS communication with the given registry. Expected format is 'my.registry.url=/path/to/the/server/certificate'.
--registry-mirror multi-arg type Registry mirror to use as pull-through cache instead of docker.io. Set it repeatedly for multiple mirrors.
--reproducible Strip timestamps out of the image to make it reproducible
--single-snapshot Take a single snapshot at the end of the build.
--skip-tls-verify Push to insecure registry ignoring TLS verify
--skip-tls-verify-pull Pull from insecure registry ignoring TLS verify
--skip-tls-verify-registry multi-arg type Insecure registry ignoring TLS verify to push and pull. Set it repeatedly for multiple registries.
--skip-unused-stages Build only used stages if defined to true. Otherwise it builds by default all stages, even the unnecessaries ones until it reaches the target stage / end of Dockerfile
--snapshot-mode string Change the file attributes inspected during snapshotting (default "full")
--snapshotMode string This flag is deprecated. Please use '--snapshot-mode'.
--tar-path string Path to save the image in as a tarball instead of pushing
--tarPath string This flag is deprecated. Please use '--tar-path'.
--target string Set the target build stage to build
--use-new-run Use the experimental run implementation for detecting changes without requiring file system snapshots.
-v, --verbosity string Log level (trace, debug, info, warn, error, fatal, panic) (default "info")
Use "executor [command] --help" for more information about a command.
手动挂载并执行构建:
# 挂在项目到workspace目录下, 注意指定工作目录:/workspace
docker run -it -v ${PWD}/mflow:/workspace -w /workspace --entrypoint=/busybox/sh docker.io/anjia0532/kaniko-project.executor:v1.9.2-debug
# 执行构建
/kaniko/executor --no-push
基于k8s操作
docker pull bitnami/git
测试下能否正常使用
# 挂载secret
# docker run -it -v ${HOME}/.ssh/:/root/.ssh -w /workspace bitnami/git
docker run --rm -it -v ${HOME}/.ssh/:/root/.ssh -w /workspace registry.cn-hangzhou.aliyuncs.com/godev/git:2.39.2
# 测试下载, 关于更多git参数说明请参考看: https://git-scm.com/docs/git-config
GIT_SSH_COMMAND='ssh -i ssh -i ./id_rsa.pub -o StrictHostKeyChecking=no' git clone git@github.com:infraboard/mpaas.git src --single-branch --branch=master
创建代码拉取的secret, 可以参考: use-case-pod-with-ssh-keys
kubectl create secret generic git-ssh-key --from-file=id_rsa=${HOME}/.ssh/id_rsa
创建镜像推送的secret, Pushing to Docker Hub 推送至指定远端镜像仓库须要credential的支持，因此须要将credential以secret的方式挂载到/kaniko/.docker/这个目录下，文件名称为config.json，内容以下:
YWRtaW46SGFyYm9yMTIzNDUK是通过registry用户名与密码以下命令获取
$ echo -n admin:Harbor12345 | base64
YWRtaW46SGFyYm9yMTIzNDUK
手动挂载并测试能否推送:
# 挂在项目到workspace目录下, 注意指定工作目录:/workspace
docker run --rm -it -v ${HOME}/Projects/inforboard/mflow:/workspace -v ${HOME}/Projects/inforboard/mflow/apps/job/impl/test/config.json:/kaniko/.docker/config.json -w /workspace --entrypoint=/busybox/sh registry.cn-hangzhou.aliyuncs.com/godev/kaniko-project.executor:v1.9.2-debug
# 执行构建
/kaniko/executor --cache=true --cache-repo=registry.cn-hangzhou.aliyuncs.com/build_cache/mpaas --compressed-caching=false --destination=registry.cn-hangzhou.aliyuncs.com/infraboard/mpaas:v0.0.1
最后创建secret
$ kubectl create secret generic kaniko-secret --from-file=apps/job/impl/test/config.json
secret/kaniko-secret created
$ kubectl get secret kaniko-secret
NAME TYPE DATA AGE
kaniko-secret Opaque 1 23s
镜像部署
拉取工具镜像
docker pull bitnami/kubectl
本地测试
docker run -it -v ~/.kube/config:/.kube/config bitnami/kubectl get ns
k8s支持远程访问部署配置, 比如:
kubectl apply -f https://k8s.io/examples/controllers/nginx-deployment.yaml
kubectl apply -f http://localhost:8080/api/mflow/v1/export/deploys/cfrcv8ea0brnte3v3jc0
只更新版本
# 更新镜像版本
kubectl set image deployment/nginx busybox=busybox nginx=nginx:1.9.1
# 补充任务标签
kubectl annotate deployments cmdb-deployment task.mflow.inforboar.io/id="test" --overwrite
如果执行失败，也可以收到试试命令
kubectl set image deployment/cmdb-deployment cmdb-deployment=busybox:1.30
kubectl annotate deployments cmdb task.mflow.inforboar.io/id=cfthf85s99bgu9olqr50