207 lines
12 KiB
Markdown
207 lines
12 KiB
Markdown
# 任务管理
|
||
|
||
基于K8s Job的流水线设计
|
||
|
||
+ 镜像构建:
|
||
+ [如何在Docker 中使用 Docker](https://www.6hu.cc/archives/78414.html)
|
||
+ [Build Images In Kubernetes](https://github.com/GoogleContainerTools/kaniko)
|
||
+ [kubernetes【工具】kaniko【1】【介绍】-无特权构建镜像](https://blog.csdn.net/xixihahalelehehe/article/details/121659254)
|
||
+ 镜像部署: [kubectl 工具镜像](https://hub.docker.com/r/bitnami/kubectl)
|
||
+ 虚拟机部署: [在 Docker 容器中配置 Ansible](https://learn.microsoft.com/zh-cn/azure/developer/ansible/configure-in-docker-container?tabs=azure-cli)
|
||
|
||
## 镜像构建
|
||
|
||
[最新国内镜像](https://liangyuanpeng.com/post/service-lank8s.cn/)
|
||
+ registry.k8s.io -> registry.lank8s.cn
|
||
+ gcr.io -> gcr.lank8s.cn
|
||
|
||
原生地址:
|
||
[Google Kaniko 仓库地址](https://console.cloud.google.com/gcr/images/kaniko-project/GLOBAL/executor)
|
||
|
||
转化地址:
|
||
[如何拉取gcr.io的镜像](https://github.com/anjia0532/gcr.io_mirror/search?p=1&q=kaniko&type=issues)
|
||
|
||
提前同步好的镜像:
|
||
+ [gcr.io/kaniko-project/executor:v1.23.2](https://github.com/anjia0532/gcr.io_mirror/issues/4321)
|
||
+ [gcr.io/kaniko-project/executor:v1.23.2-debug](https://github.com/anjia0532/gcr.io_mirror/issues/4068)
|
||
|
||
拉取最新版本的镜像:
|
||
```
|
||
docker pull anjia0532/kaniko-project.executor:v1.9.2
|
||
docker pull anjia0532/kaniko-project.executor:v1.9.2-debug
|
||
```
|
||
|
||
### 手动操作
|
||
|
||
启动一个deubg环境, 可以看看里面的工具(二进制可执行文件,工具的用法)
|
||
```sh
|
||
docker run -it --rm --entrypoint=/busybox/sh anjia0532/kaniko-project.executor:v1.23.2-debug
|
||
|
||
/ # ls -l /kaniko/
|
||
total 75448
|
||
-rwxr-xr-x 1 0 0 10900549 Sep 8 18:23 docker-credential-acr-env
|
||
-rwxr-xr-x 1 0 0 8981984 Sep 8 18:22 docker-credential-ecr-login
|
||
-rwxr-xr-x 1 0 0 7814415 Sep 8 18:21 docker-credential-gcr
|
||
-rwxr-xr-x 1 0 0 35250176 Sep 26 19:27 executor
|
||
drwxr-xr-x 3 0 0 4096 Sep 26 19:27 ssl
|
||
-rwxr-xr-x 1 0 0 14303232 Sep 26 19:27 warmer
|
||
/ # /kaniko/executor -h
|
||
Usage:
|
||
executor [flags]
|
||
executor [command]
|
||
|
||
Available Commands:
|
||
completion Generate the autocompletion script for the specified shell
|
||
help Help about any command
|
||
version Print the version number of kaniko
|
||
|
||
Flags:
|
||
--build-arg multi-arg type This flag allows you to pass in ARG values at build time. Set it repeatedly for multiple values.
|
||
--cache Use cache when building image
|
||
--cache-copy-layers Caches copy layers
|
||
--cache-dir string Specify a local directory to use as a cache. (default "/cache")
|
||
--cache-repo string Specify a repository to use as a cache, otherwise one will be inferred from the destination provided
|
||
--cache-run-layers Caches run layers (default true)
|
||
--cache-ttl duration Cache timeout in hours. Defaults to two weeks. (default 336h0m0s)
|
||
--cleanup Clean the filesystem at the end
|
||
--compressed-caching Compress the cached layers. Decreases build time, but increases memory usage. (default true)
|
||
-c, --context string Path to the dockerfile build context. (default "/workspace/")
|
||
--context-sub-path string Sub path within the given context.
|
||
--custom-platform string Specify the build platform if different from the current host
|
||
--customPlatform string This flag is deprecated. Please use '--custom-platform'.
|
||
-d, --destination multi-arg type Registry the final image should be pushed to. Set it repeatedly for multiple destinations.
|
||
--digest-file string Specify a file to save the digest of the built image to.
|
||
-f, --dockerfile string Path to the dockerfile to be built. (default "Dockerfile")
|
||
--force Force building outside of a container
|
||
--force-build-metadata Force add metadata layers to build image
|
||
--git gitoptions Branch to clone if build context is a git repository (default branch=,single-branch=false,recurse-submodules=false)
|
||
-h, --help help for executor
|
||
--ignore-path multi-arg type Ignore these paths when taking a snapshot. Set it repeatedly for multiple paths.
|
||
--ignore-var-run Ignore /var/run directory when taking image snapshot. Set it to false to preserve /var/run/ in destination image. (default true)
|
||
--image-fs-extract-retry int Number of retries for image FS extraction
|
||
--image-name-tag-with-digest-file string Specify a file to save the image name w/ image tag w/ digest of the built image to.
|
||
--image-name-with-digest-file string Specify a file to save the image name w/ digest of the built image to.
|
||
--insecure Push to insecure registry using plain HTTP
|
||
--insecure-pull Pull from insecure registry using plain HTTP
|
||
--insecure-registry multi-arg type Insecure registry using plain HTTP to push and pull. Set it repeatedly for multiple registries.
|
||
--kaniko-dir string Path to the kaniko directory, this takes precedence over the KANIKO_DIR environment variable. (default "/kaniko")
|
||
--label multi-arg type Set metadata for an image. Set it repeatedly for multiple labels.
|
||
--log-format string Log format (text, color, json) (default "color")
|
||
--log-timestamp Timestamp in log output
|
||
--no-push Do not push the image to the registry
|
||
--no-push-cache Do not push the cache layers to the registry
|
||
--oci-layout-path string Path to save the OCI image layout of the built image.
|
||
--push-retry int Number of retries for the push operation
|
||
--registry-certificate key-value-arg type Use the provided certificate for TLS communication with the given registry. Expected format is 'my.registry.url=/path/to/the/server/certificate'.
|
||
--registry-mirror multi-arg type Registry mirror to use as pull-through cache instead of docker.io. Set it repeatedly for multiple mirrors.
|
||
--reproducible Strip timestamps out of the image to make it reproducible
|
||
--single-snapshot Take a single snapshot at the end of the build.
|
||
--skip-tls-verify Push to insecure registry ignoring TLS verify
|
||
--skip-tls-verify-pull Pull from insecure registry ignoring TLS verify
|
||
--skip-tls-verify-registry multi-arg type Insecure registry ignoring TLS verify to push and pull. Set it repeatedly for multiple registries.
|
||
--skip-unused-stages Build only used stages if defined to true. Otherwise it builds by default all stages, even the unnecessaries ones until it reaches the target stage / end of Dockerfile
|
||
--snapshot-mode string Change the file attributes inspected during snapshotting (default "full")
|
||
--snapshotMode string This flag is deprecated. Please use '--snapshot-mode'.
|
||
--tar-path string Path to save the image in as a tarball instead of pushing
|
||
--tarPath string This flag is deprecated. Please use '--tar-path'.
|
||
--target string Set the target build stage to build
|
||
--use-new-run Use the experimental run implementation for detecting changes without requiring file system snapshots.
|
||
-v, --verbosity string Log level (trace, debug, info, warn, error, fatal, panic) (default "info")
|
||
|
||
Use "executor [command] --help" for more information about a command.
|
||
```
|
||
|
||
手动挂载并执行构建:
|
||
```sh
|
||
# 挂在项目到workspace目录下, 注意指定工作目录:/workspace
|
||
docker run -it -v ${PWD}/mflow:/workspace -w /workspace --entrypoint=/busybox/sh docker.io/anjia0532/kaniko-project.executor:v1.9.2-debug
|
||
# 执行构建
|
||
/kaniko/executor --no-push
|
||
```
|
||
|
||
### 基于k8s操作
|
||
|
||
[使用git工具镜像下载依赖](https://hub.docker.com/r/bitnami/git)
|
||
```sh
|
||
docker pull bitnami/git
|
||
```
|
||
|
||
测试下能否正常使用
|
||
```sh
|
||
# 挂载secret
|
||
# docker run -it -v ${HOME}/.ssh/:/root/.ssh -w /workspace bitnami/git
|
||
docker run --rm -it -v ${HOME}/.ssh/:/root/.ssh -w /workspace registry.cn-hangzhou.aliyuncs.com/godev/git:2.39.2
|
||
# 测试下载, 关于更多git参数说明请参考看: https://git-scm.com/docs/git-config
|
||
GIT_SSH_COMMAND='ssh -i ssh -i ./id_rsa.pub -o StrictHostKeyChecking=no' git clone git@github.com:infraboard/mpaas.git src --single-branch --branch=master
|
||
```
|
||
|
||
创建代码拉取的secret, 可以参考: [use-case-pod-with-ssh-keys](https://kubernetes.io/zh-cn/docs/concepts/configuration/secret/#use-case-pod-with-ssh-keys)
|
||
```
|
||
kubectl create secret generic git-ssh-key --from-file=id_rsa=${HOME}/.ssh/id_rsa
|
||
```
|
||
|
||
创建镜像推送的secret, [Pushing to Docker Hub](https://github.com/GoogleContainerTools/kaniko#pushing-to-docker-hub) 推送至指定远端镜像仓库须要credential的支持，因此须要将credential以secret的方式挂载到/kaniko/.docker/这个目录下，文件名称为config.json，内容以下:
|
||
[](./impl/test/kaniko_config_example.json)
|
||
|
||
YWRtaW46SGFyYm9yMTIzNDUK是通过registry用户名与密码以下命令获取
|
||
```sh
|
||
$ echo -n admin:Harbor12345 | base64
|
||
YWRtaW46SGFyYm9yMTIzNDUK
|
||
```
|
||
|
||
手动挂载并测试能否推送:
|
||
```sh
|
||
# 挂在项目到workspace目录下, 注意指定工作目录:/workspace
|
||
docker run --rm -it -v ${HOME}/Projects/inforboard/mflow:/workspace -v ${HOME}/Projects/inforboard/mflow/apps/job/impl/test/config.json:/kaniko/.docker/config.json -w /workspace --entrypoint=/busybox/sh registry.cn-hangzhou.aliyuncs.com/godev/kaniko-project.executor:v1.9.2-debug
|
||
# 执行构建
|
||
/kaniko/executor --cache=true --cache-repo=registry.cn-hangzhou.aliyuncs.com/build_cache/mpaas --compressed-caching=false --destination=registry.cn-hangzhou.aliyuncs.com/infraboard/mpaas:v0.0.1
|
||
```
|
||
|
||
最后创建secret
|
||
```sh
|
||
$ kubectl create secret generic kaniko-secret --from-file=apps/job/impl/test/config.json
|
||
secret/kaniko-secret created
|
||
|
||
$ kubectl get secret kaniko-secret
|
||
NAME TYPE DATA AGE
|
||
kaniko-secret Opaque 1 23s
|
||
```
|
||
|
||
共享配置Job共享Workdir:
|
||
[](./impl/test/build.yml)
|
||
|
||
## 镜像部署
|
||
|
||
[k8s 应用部署相关文档](https://kubernetes.io/docs/concepts/workloads/controllers/)
|
||
|
||
拉取工具镜像
|
||
```
|
||
docker pull bitnami/kubectl
|
||
```
|
||
|
||
本地测试
|
||
```sh
|
||
docker run -it -v ~/.kube/config:/.kube/config bitnami/kubectl get ns
|
||
```
|
||
|
||
k8s支持远程访问部署配置, 比如:
|
||
```
|
||
kubectl apply -f https://k8s.io/examples/controllers/nginx-deployment.yaml
|
||
kubectl apply -f http://localhost:8080/api/mflow/v1/export/deploys/cfrcv8ea0brnte3v3jc0
|
||
```
|
||
|
||
只更新版本
|
||
```
|
||
# 更新镜像版本
|
||
kubectl set image deployment/nginx busybox=busybox nginx=nginx:1.9.1
|
||
# 补充任务标签
|
||
kubectl annotate deployments cmdb-deployment task.mflow.inforboar.io/id="test" --overwrite
|
||
```
|
||
|
||
如果执行失败，也可以收到试试命令
|
||
```
|
||
kubectl set image deployment/cmdb-deployment cmdb-deployment=busybox:1.30
|
||
kubectl annotate deployments cmdb task.mflow.inforboar.io/id=cfthf85s99bgu9olqr50
|
||
```
|
||
|