Introduction

One of the most important questions when it comes to dealing with GitOps is knowing where to store your secrets and how to manage them securely. Some of the tools available for this purpose are Secrets Manager and External Secrets. However, I was looking for something simpler for my team. Argo Vault Plugin (AVP) is an easy-to-use tool for managing secrets in GitOps solutions under Argo and Kubernetes. Additionally, you can also use other secrets management tools like AWS Secrets Manager or others instead of Vault.

On the other hand, I must say that the installation of AVP was not as easy compared to other tools such as secrets-manager (https://github.com/tuenti/secrets-manager) or external-secrets (https://github.com/external-secrets/kubernetes-external-secrets).

If you would like to learn more about AVP, including how it works, I recommend following the official documentation at https://argocd-vault-plugin.readthedocs.io/en/stable/. In that tutorial, I will explain and summarize how to install the tool using HELM, covering the two options that we tried. It will be a straightforward, step-by-step process.

Installing AVP in Argo CD

When reading the official website at https://github.com/argoproj-labs/argocd-vault-plugin/blob/main/docs/installation.md, we can observe that there are four different ways to set up the Argo Vault Plugin. This variety of options might pose a challenge in determining the most suitable approach for our specific needs. Therefore, it is essential to evaluate each method carefully to determine which one is the best fit for us.

In order to use the plugin in Argo CD, you have four distinct options available:

Installation via argocd-cm ConfigMap using Kustomization or Helm
Installation via argocd-cm ConfigMap creating a custom argocd-repo-server image
Installation via a sidecar container using Kustomization or Helm
Installation via a sidecar container creating a custom sidecar image with AVP

But two of these solutions can be installed via Kustomization or Helm. Therefore, this tutorial is focused on people who are using Argo installations through Helm, rather than ArgoCD installations with Kustomization.

What we try and choose?

Installation via argocd-cm ConfigMap using Helm and not kustomization
Installation via a sidecar container using Helm and not kustomization

In this tutorial, we used Helm, which is interesting because when we attempted to install AVP using Kustomization, we encountered numerous warnings after applying the kustomize command. These warnings arose because the previous installation of ArgoCD was done with Helm instead of Kustomization or raw Kubernetes and all the argo cluster was in an inconsistence state .

If you are not using Helm, you can attempt to install the Argo Vault Plugin using the provided links and commands. However, it is recommended to follow the official documentation for more detailed instructions. If you are doing that please skip that tutorial.

# argocd vault plugin using kustomization
kubectl apply -k bootstrap/overlays/argocd-vault-plugin/

https://github.com/argoproj-labs/argocd-vault-plugin/blob/main/manifests/cmp-sidecar/kustomization.yaml

https://github.com/argoproj-labs/argocd-vault-plugin/tree/main/manifests/cmp-configmap

Finally, we chose the HELM method and installed AVP using Sidecar and Configmap. Since we made that decision, we focused solely on HELM installation. During the process, we encountered some challenges as there were not many examples available on the internet, and we noticed numerous open issues on the https://github.com/argoproj-labs/argocd-vault-plugin/issues page.

I was almost done with the sidecar installation, but after switching to argocd-cm configmap, I gained a better understanding of the setup, and it finally started working. I believe that if I had continued with the sidecar method, it would have been successfully with next examples of the article related with the sidecar installation with helm.

So we opted for the old method (argocd-cm configmap) because it is easier to debug, has more documentation and more examples available in the moment that I write that article. However, in future Argo installations, we may consider revisiting the sidecar method, as it is newer and expected to be more stable and bug-free.

Also It's worth mentioning that we tested AV version 1.11, 1.12, and 1.14, along with Argocd 2.6.7 and helmchart 5.29.1.

Installation via `argocd-cm` ConfigMap using Helm

summary

I installed the ArgoCD Vault Plugin using the configmap solution, which turned out to be the easiest installation method out of the two I tested. It was a straightforward process that involved just two steps.

Create k8s Secret with authorization configuration that Vault plugin will use.
we have to install ArgoCD from the official Helm Chart but with extra configuration that provides modifications one sidecar container and one configmap update.

And now the argo vault plugin is inside the argocd-repo-server POD.

Step 1

About the first step, you can see a secret example , that secret was simply created using a "kubectl apply -f secret", here the AVP_ROLE_ID, and AVP_SECRET_ID must be done by some method that you will prefer. In my case, I am using a GITLAB pipeline but you can do it manually or with other solutions. Here a chicken and egg situation. How to boostrap that first step with the first secrets that unlock the other secrets.

kind: Secret
apiVersion: v1
metadata:
  name: argocd-vault-plugin-credentials
type: Opaque
stringData:
  AVP_TYPE: "vault"
  VAULT_ADDR: "https://vault.internal.ag"
  AVP_AUTH_TYPE: "approle"
  AVP_ROLE_ID: "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
  AVP_SECRET_ID: "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"

Step 2

That step is the regular helm installation of ARGOCD but in the argocd-stg.yaml we added some specific new lines that will be in the following second box code block example.

helm repo add argo https://argoproj.github.io/argo-helm

# argocd installation using HELM
helm upgrade \
  --wait \
  --install argocd \
  --namespace $NAMESPACE \
  --version 5.29.1 \
  -f bootstrap/scripts/values/argocd-stg.yaml \
  --set "configs.cm.dex.config.connectors.config.caData=$ARGOSECRET" \
  argo/argo-cd

...
...
configs:
  cm:
    create: true
  ....
  ....
  .... 
    # argo vault plugin
    configManagementPlugins: |
      - name: argocd-vault-plugin
        generate:
          command: ["argocd-vault-plugin"]
          args: ["generate", "./"]
      - name: argocd-vault-plugin-helm
        generate:
          command: ["sh", "-c"]
          args: ['helm template "$ARGOCD_APP_NAME" -n "$ARGOCD_APP_NAMESPACE" . | argocd-vault-plugin generate -']
      # This lets you pass args to the Helm invocation as described here: https://argocd-vault-plugin.readthedocs.io/en/stable/usage/#with-helm
      # IMPORTANT: passing $helm_args effectively allows users to run arbitrary code in the Argo CD repo-server.
      # Only use this when the users are completely trusted. If possible, determine which Helm arguments are needed by 
      # your users and explicitly pass only those arguments.
      - name: argocd-vault-plugin-helm-with-args
        generate:
          command: ["sh", "-c"]
          args: ['helm template "$ARGOCD_APP_NAME" -n "$ARGOCD_APP_NAMESPACE" ${helm_args} . | argocd-vault-plugin generate -']
      # This lets you pass a values file as a string as described here:
      # https://argocd-vault-plugin.readthedocs.io/en/stable/usage/#with-helm
      - name: argocd-vault-plugin-helm-with-values
        generate:
          command: ["bash", "-c"]
          args: ['helm template "$ARGOCD_APP_NAME" -n "$ARGOCD_APP_NAMESPACE" -f <(echo "$ARGOCD_ENV_HELM_VALUES") . | argocd-vault-plugin generate -']
      - name: argocd-vault-plugin-kustomize
        generate:
          command: ["sh", "-c"]
          args: ["kustomize build . | argocd-vault-plugin generate -"]
    # end argo vault plugin 



...
...
...

## Repo Server
repoServer:
  name: repo-server
  replicas: 2
  # argo vault plugin CMP
  envFrom: 
   - secretRef:
       name: argocd-vault-plugin-credentials
  rbac:
    - verbs:
        - get
        - list
        - watch
      apiGroups:
        - ''
      resources:
        - secrets
        - configmaps

  initContainers:
  - name: download-tools
    image: alpine:3.8
    command: [sh, -c]

    # Don't forget to update this to whatever the stable release version is
    # Note the lack of the `v` prefix unlike the git tag
    env:
      - name: AVP_VERSION
        value: "1.14.0"
    args:
      - >-
        wget -O argocd-vault-plugin
        https://github.com/argoproj-labs/argocd-vault-plugin/releases/download/v${AVP_VERSION}/argocd-vault-plugin_${AVP_VERSION}_linux_amd64 &&
        chmod +x argocd-vault-plugin &&
        mv argocd-vault-plugin /custom-tools/
    volumeMounts:
      - mountPath: /custom-tools
        name: custom-tools
          
  volumes:
    - name: custom-tools
      emptyDir: {}

  volumeMounts:
    - name: custom-tools
      mountPath: /usr/local/bin/argocd-vault-plugin
      subPath: argocd-vault-plugin
  ## end argo vault plugin CMP
....
....
....

Installation via `sidecar plugin` using Helm

I found some interesting links on the internet, but for me, the best example is the one provided in this GitHub repository: https://github.com/luafanti/arogcd-vault-plugin-with-helm. Additionally, the best documentation to follow for this type of installation of the AVP sidecar plugin using Helm can be found here: https://dev.to/luafanti/injecting-secrets-from-vault-into-helm-charts-with-argocd-49k. I highly recommend you to follow these resources. Essentially, in this article, I will be explaining the same process.

summary

I installed the ArgoCD Vault Plugin using a sidecar container.

Create k8s Secret with authorization configuration that Vault plugin will use.
Create k8s ConfigMap
we have to install ArgoCD from the official Helm Chart but with extra configuration that provides modifications required to install Vault plugin via sidecar container.

Now the argocd-repo-server has sidecar container avp-helm with one container more

so the POD "argocd-repo-server" instead to have 1/1 will have 2/2. The official AVP documentation recomends to follow this web page https://argo-cd.readthedocs.io/en/stable/operator-manual/config-management-plugins/ to understand how argo plugins are working through sidecar container.

Without AVP:

argocd-repo-server-66757788cb-f5nwd                            1/1

With AVP:

argocd-repo-server-66757788cb-f5nwd                            2/2

In the next steps 1, 2, and 3, I will summarize what I have gathered from the other articles.

Step 1

So like we did in the previous installation method. we need the secret

kind: Secret
apiVersion: v1
metadata:
  name: argocd-vault-plugin-credentials
type: Opaque
stringData:
  AVP_TYPE: "vault"
  VAULT_ADDR: "https://vault.internal.io"
  AVP_AUTH_TYPE: "approle"
  AVP_ROLE_ID: "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
  AVP_SECRET_ID: "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"

Step 2

Comparing the previous installation, here we have a new step to do . You can do it using a simple kubectl apply resource or foor example if you love to use everything with HELM you can use the raw incubator helm chart to create raw k8s resources. More info https://github.com/helm/charts/tree/master/incubator/raw

kubectl apply -f bootstrap/overlays/argocd-vault-plugin/argocd-vault-plugin-cmp.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: cmp-plugin
data:
  avp-kustomize.yaml: |
    ---
    apiVersion: argoproj.io/v1alpha1
    kind: ConfigManagementPlugin
    metadata:
      name: argocd-vault-plugin-kustomize
    spec:
      allowConcurrency: true

      # Note: this command is run _before_ anything is done, therefore the logic is to check
      # if this looks like a Kustomize bundle
      discover:
        find:
          command:
            - find
            - "."
            - -name
            - kustomization.yaml
      generate:
        command:
          - sh
          - "-c"
          - "kustomize build . | argocd-vault-plugin generate -"
      lockRepo: false
  avp-helm.yaml: |
    ---
    apiVersion: argoproj.io/v1alpha1
    kind: ConfigManagementPlugin
    metadata:
      name: argocd-vault-plugin-helm
    spec:
      allowConcurrency: true

      # Note: this command is run _before_ any Helm templating is done, therefore the logic is to check
      # if this looks like a Helm chart
      discover:
        find:
          command:
            - sh
            - "-c"
            - "find . -name 'Chart.yaml' && find . -name 'values.yaml'"
      generate:
        # **IMPORTANT**: passing `${ARGOCD_ENV_helm_args}` effectively allows users to run arbitrary code in the Argo CD 
        # repo-server (or, if using a sidecar, in the plugin sidecar). Only use this when the users are completely trusted. If
        # possible, determine which Helm arguments are needed by your users and explicitly pass only those arguments.
        command:
          - sh
          - "-c"
          - |
            helm template $ARGOCD_APP_NAME -n $ARGOCD_APP_NAMESPACE ${ARGOCD_ENV_HELM_ARGS} . |
            argocd-vault-plugin generate --verbose-sensitive-output -
      lockRepo: false
  avp.yaml: |
    apiVersion: argoproj.io/v1alpha1
    kind: ConfigManagementPlugin
    metadata:
      name: argocd-vault-plugin
    spec:
      allowConcurrency: true
      discover:
        find:
          command:
            - sh
            - "-c"
            - "find . -name '*.yaml' | xargs -I {} grep \"<path\\|avp\\.kubernetes\\.io\" {} | grep ."
      generate:
        command:
          - argocd-vault-plugin
          - generate
          - "."
      lockRepo: false
--

Important note: maybe here we need to use the latest examples of AVP or maybe you will need to check if there is a new change or updates.

Step 3

now the last step is a regular helm installation but with some values update like we did in the previous installation in the step 2.

# argocd installation using HELM
helm upgrade \
  --wait \
  --install argocd \
  --namespace $NAMESPACE \
  --version 5.29.1 \
  -f bootstrap/scripts/values/argocd-stg.yaml \
  --set "configs.cm.dex.config.connectors.config.caData=$ARGOSECRET" \
  argo/argo-cd

...
...
...
## Repo Server
repoServer:
  name: repo-server
  replicas: 2
  # argo vault plugin
  envFrom: 
   - secretRef:
       name: argocd-vault-plugin-credentials
  rbac:
    - verbs:
        - get
        - list
        - watch
      apiGroups:
        - ''
      resources:
        - secrets
        - configmaps
  initContainers:
    - name: download-tools
      image: registry.access.redhat.com/ubi8
      env:
        - name: AVP_VERSION
          value: 1.14.0
      command: [sh, -c]
      args:
        - >-
          curl -L https://github.com/argoproj-labs/argocd-vault-plugin/releases/download/v$(AVP_VERSION)/argocd-vault-plugin_$(AVP_VERSION)_linux_amd64 -o argocd-vault-plugin &&
          chmod +x argocd-vault-plugin &&
          mv argocd-vault-plugin /custom-tools/
      volumeMounts:
        - mountPath: /custom-tools
          name: custom-tools
  extraContainers:
    # argocd-vault-plugin with Helm
    - name: avp-helm
      command: [/var/run/argocd/argocd-cmp-server]
      image: quay.io/argoproj/argocd:v2.6.7
      securityContext:
        runAsNonRoot: true
        runAsUser: 999
      volumeMounts:
        - mountPath: /var/run/argocd
          name: var-files
        - mountPath: /home/argocd/cmp-server/plugins
          name: plugins
        - mountPath: /tmp
          name: cmp-tmp
        # Register plugins into sidecar
        - mountPath: /home/argocd/cmp-server/config/plugin.yaml
          subPath: avp-helm.yaml
          name: cmp-plugin
        # Important: Mount tools into $PATH
        - name: custom-tools
          subPath: argocd-vault-plugin
          mountPath: /usr/local/bin/argocd-vault-plugin
  volumes:
    - configMap:
        name: cmp-plugin      
      name: cmp-plugin
    - name: cmp-tmp
      emptyDir: {}
    - name: custom-tools
      emptyDir: {}
  ## end argo vault plugin
...
...
...

After finish the installation, remember that we need to choose between these 5 plugins that we have now available to render the new vault path secrets.

argocd-vault-plugin
argocd-vault-plugin-helm
argocd-vault-plugin-helm-with-args
argocd-vault-plugin-helm-with-values
argocd-vault-plugin-kustomize

Conclusions

Why AVP instead secrets-manager or external-secrets:

it is not necessary any CRD, any k8s secret resource deployed, any special k8s resource to install.
Argocd is rendering all the secrets before to deploy the HELM. It is a previous step before.
One of the most important reasons is because now is easy more human readable the helm and how to extract the secrets from vault
easier than secrets-manager and external-secrets
handicap: the installation was complicated because the documentation was not well updated with the last versions and was mainly focus in kustomize not helm. Now having this documentation ready could be easier. Hope that could help.
another handicap. sometimes important secrets must be in a k8s resource secret instead having these environment variables like is doing this tool. So here could be interesting to use both solutions.

Also it is important to mention that using “argocd-cm” could be easier method out of the two but it is important to mention that this option is planned to be removed. “drop support for argocd-cm Config Management Plugins in favor of sidecars” argoproj/argo-cd#8117"

Helm examples with AVP

argocd-vault-plugin-helm
argocd-vault-plugin-helm-with-args
argocd-vault-plugin-helm-with-values

---
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: secrets-manager
spec:
  destination:
    namespace: yournamespace
    server: https://kubernetes.default.svc
  project: yourproject
  source:
    repoURL: https://chartmuseum.internal.io
    chart: secrets-manager
    targetRevision: 1.0.2
    plugin:
      name: argocd-vault-plugin-helm-with-values
      env:
        - name: HELM_VALUES
          value: |
            createCustomResource: false
            resources:
              limits:
                cpu: 100m
                memory: 128Mi
              requests:
                cpu: 100m
                memory: 128Mi
            secretsManager:
              watchNamespaces: "yournamespace"
              vault:
                url: https://vault.internal.io
                roleId: <path:kv-v2/data/yoursecretpath/gitlab#VAULT_ROLE_ID>
                secretId: <path:kv-v2/data/yoursecretpath/gitlab#VAULT_SECRET_ID>

how to debug Argo Vault Plugin

Inside the POD or Sidecar of argocd-repo-server, you can run the following

kubectl exec argocd-repo-server-74846c5b99-hl678 -c repo-server -it /bin/sh
$ env| grep AVP |wc -l
4

to check if your VAULT variables are well spread it.

and also you can run the AVP binary inside the argocd-repo-server

kubectl exec argocd-repo-server-74846c5b99-hl678 -c avp-helm -it /bin/sh 
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
$ argocd-vault-plugin generate .

Interesting Documentation

https://argocd-vault-plugin.readthedocs.io/

https://piotrminkowski.com/2022/08/08/manage-secrets-on-kubernetes-with-argocd-and-vault/

https://luafanti.medium.com/injecting-secrets-from-vault-into-helm-charts-with-argocd-43fc1df57e74

https://itnext.io/argocd-secret-management-with-argocd-vault-plugin-539f104aff05

https://github.com/jkayani/avp-demo-kubecon-2021

https://www.opsmx.com/blog/how-to-integrate-argo-cd-and-vault-for-managing-secrets-in-gitops/

Introduction

Installing AVP in Argo CD

What we try and choose?

Installation via argocd-cm ConfigMap using Helm

summary

Step 1

Step 2

Installation via sidecar plugin using Helm

summary

Step 1

Step 2

Step 3

Conclusions

Helm examples with AVP

how to debug Argo Vault Plugin

Interesting Documentation

Installation via `argocd-cm` ConfigMap using Helm

Installation via `sidecar plugin` using Helm