Repository architecture and tenant management

The GitOps repository uses self-contained tenant folders, allowing each tenant to have its own chart, values, and overlays.

Directory structure

charts/: Contains Helm charts organized by tenant.
- konnector-bundle/: The Helm chart pulled from the Cortex tenant OCI registry.
- tenant-values.yaml: Your specific tenant value overrides downloaded from Cortex.
- overlays/: Contains the Kustomize config and specific application patches for flux/ and argocd/.

Prepare the deployment files in the repo

Regardless of your chosen controller, you must first prepare the tenant directory:

Add a new Kubernetes integration in Cortex, from Settings → Data Sources & Integrations → Add New → Kubernetes.
Configure the settings to meet your requirements and then click Generate. Note: Make sure to disable Auto upgrade, so it does not conflict with the Flux or ArgoCD state management.
Download the values and auth files from step 1.
Authenticate the Helm client to the tenant's artifact registry in step 2.
In step 3, instead of directly deploying, pull the charts to add them to your GitOps repository.
Copy the Helm release repo path and the version number.
```
helm pull <repo-path> --version <version> --untar
```
Copy the connector bundle directory directly to the charts directory.
Copy the values file directly to the charts directory.

Create in the charts directory a new directory called overlays. Flux

Create a helm-release.yaml file that contains a Flux HelmRelease custom resource definition (CRD) object. This object specifies the base chart location in the repository and a ConfigMap generated from Kustomization instructions. These instructions point to tenant-specific values that patch the default values. Include the base Flux instructions for maintaining this release in the cluster.

# Flux HelmRelease for KSPM Agent
#
# This is a self-contained HelmRelease that deploys the konnector-bundle chart
# from the Git repository. No separate base or patch files are needed.
#
# Prerequisites:
#   - Flux is installed on the cluster
#   - A GitRepository source named "kspm-gitops" exists in flux-system namespace
#   - The tenant's konnector-bundle chart is at the path specified below
#
# Value merge order (last wins):
#   1. Chart's built-in values.yaml (always loaded first by Helm)
#   2. valuesFrom entries (ConfigMaps/Secrets, processed in order)
#   3. spec.values (inline values, highest priority)
#
# NOTE: If you change the namespace from "panw" to something else, you must
# update it in ALL of these places:
#   1. metadata.namespace below
#   2. tenant-values.yaml (global.namespace)
#   3. configMapGenerator namespace in kustomization.yaml
# The namespace is automatically created by Helm via install.createNamespace.
---
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: kspm-agent
  # Target namespace — change here if using a custom namespace.
  # Helm will create this namespace automatically (install.createNamespace: true).
  namespace: panw
  labels:
    app.kubernetes.io/part-of: kspm-agent
    app.kubernetes.io/managed-by: flux
spec:
  interval: 5m
  timeout: 5m
  chart:
    spec:
      # Path to the tenant's chart (relative to Git repo root)
      # Replace <TENANT_ID> with your actual tenant ID
      chart: ./charts/konnector-bundle
      sourceRef:
        kind: GitRepository
        name: kspm-gitops
        namespace: flux-system
      reconcileStrategy: Revision
  install:
    createNamespace: true
    remediation:
      retries: 3
  upgrade:
    remediation:
      retries: 3
    cleanupOnFail: true
  uninstall:
    keepHistory: false
  # Values sources - processed in order, later entries override earlier ones
  valuesFrom:
    # Tenant-specific overrides loaded from a ConfigMap
    # (equivalent to: helm install -f tenant-values.yaml)
    - kind: ConfigMap
      name: tenant-values
      valuesKey: values.yaml
  values:
    # Inline value overrides (highest priority)
    # Uncomment and set values here to override everything above
    # global:
    #   optionalValues:
    #     CONSOLE_LOG_LEVEL: "DEBUG"

b. Create kustomization.yaml in the same overlays directory where the helmrelease.yaml .

# Flux Kustomization for KSPM Agent
#
# This kustomization deploys the HelmRelease along with a ConfigMap
# containing the tenant-specific values.
#
# To deploy with Flux:
#   flux create kustomization kspm-agent \
#     --source=GitRepository/kspm-gitops \
#     --path="./charts/overlays" \
#     --prune=true \
#     --interval=5m
#
# Directory structure expected:
#   charts/
#   ├── tenant-values.yaml          # Your tenant-specific Helm values
#   ├── konnector-bundle/           # The Helm chart
#   └── overlays/
#       ├── kustomization.yaml      # This file
#       └── helmrelease.yaml        # The HelmRelease + Namespace
---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - helmrelease.yaml

configMapGenerator:
  # Creates a ConfigMap from your tenant-values.yaml file.
  # The HelmRelease references this ConfigMap via valuesFrom to inject
  # tenant-specific overrides into the Helm chart.
  - name: tenant-values
    # Must match the namespace in helmrelease.yaml and tenant-values.yaml
    namespace: panw
    files:
      - values.yaml=../../tenant-values.yaml
    options:
      disableNameSuffixHash: true

Note: If the namespace provided in the tenant is different than panw, change the namespace in both the kustomization and helmrelease files.

ArgoCD a. Create an application.yaml file in the overlays directory.

# ArgoCD Application for KSPM Agent
#
# This is a self-contained ArgoCD Application that deploys the konnector-bundle
# chart from the Git repository. No separate base or patch files are needed.
#
# Prerequisites:
#   - ArgoCD is installed on the cluster
#   - The Git repository is accessible from ArgoCD
#   - The tenant's konnector-bundle chart is at the path specified below
#
# Value merge order (last wins):
#   1. Chart's built-in values.yaml (always loaded first by Helm)
#   2. valueFiles entries (processed in order)
#   3. Inline values (highest priority)
#
# NOTE: If you change the namespace from "panw" to something else, you must
# update it in ALL of these places:
#   1. spec.destination.namespace below
#   2. AppProject destinations (if using a custom project)
#   3. tenant-values.yaml (global.namespace)
# The namespace is automatically created via syncOptions CreateNamespace=true.
#
# Replace the following placeholders before deploying:
#   <REPO_URL>   - Your Git repository URL (e.g., https://gitlab.example.com/kspm/kspm-gitops.git)
#   <TENANT_ID>  - Your tenant ID
---
apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: kspm
  namespace: argocd
  labels:
    app.kubernetes.io/part-of: kspm-agent
    app.kubernetes.io/managed-by: argocd
spec:
  description: "KSPM Agent GitOps project"
  destinations:
    # Target namespace — change here if using a custom namespace
    - namespace: panw
      server: https://kubernetes.default.svc
  sourceRepos:
    - "*"
  # Allow all cluster-scoped resources that the agent might need
  clusterResourceWhitelist:
    - group: ""
      kind: Namespace
    - group: rbac.authorization.k8s.io
      kind: ClusterRole
    - group: rbac.authorization.k8s.io
      kind: ClusterRoleBinding
    - group: policy
      kind: PodSecurityPolicy
  # Allow all namespaced resources
  namespaceResourceWhitelist:
    - group: "*"
      kind: "*"
---
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: kspm-agent
  namespace: argocd
  labels:
    app.kubernetes.io/part-of: kspm-agent
    app.kubernetes.io/managed-by: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  project: kspm
  # Use 'sources' (plural) to enable $values references from the same repo
  sources:
    # Source 1: The Helm chart
    - repoURL: <REPO_URL>
      path: charts/konnector-bundle
      targetRevision: main
      helm:
        releaseName: kspm-agent
        valueFiles:
          # Chart's own values.yaml
          - values.yaml
          # Tenant-specific overrides — path relative to repo root via $values
          - $values/charts/tenant-values.yaml
        # Inline values (highest priority)
        # values: |
        #   global:
        #     optionalValues:
        #       CONSOLE_LOG_LEVEL: "DEBUG"
    # Source 2: Values reference (same repo, enables $values/ prefix in valueFiles)
    - repoURL: <REPO_URL>
      targetRevision: main
      ref: values
  destination:
    server: https://kubernetes.default.svc
    # Target namespace — change here if using a custom namespace
    namespace: panw
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      # Automatically creates the namespace if it doesn't exist
      - CreateNamespace=true
      - ServerSideApply=true
    retry:
      limit: 3
      backoff:
        duration: 5s
        factor: 2
        maxDuration: 3m
  # Ignore differences in resources that are modified at runtime by the konnector
  # manager. Without this, ArgoCD's selfHeal would revert runtime changes.
  ignoreDifferences:
    # The backend-auth-secret tokens are placeholder values in the chart template.
    # The cortex-manager replaces them with real tokens at runtime.
    - group: ""
      kind: Secret
      name: backend-auth-secret
      jsonPointers:
        - /data
    # The admission-control-tls secret is created by the admission controller at runtime
    - group: ""
      kind: Secret
      name: admission-control-tls
      jsonPointers:
        - /data
    # The webhook CA bundle is injected by the admission controller
    - group: admissionregistration.k8s.io
      kind: ValidatingWebhookConfiguration
      jsonPointers:
        - /webhooks/0/clientConfig/caBundle

b. Create a kustomization.yaml file in the overlays directory.

# ArgoCD Kustomization for KSPM Agent
#
# This kustomization deploys the ArgoCD AppProject and Application.
#
# To deploy:
#   kubectl apply -k charts/tenants/<TENANT_ID>/overlays/argocd/
#
# Directory structure expected:
#   charts/tenants/<TENANT_ID>/
#   ├── tenant-values.yaml          # Your tenant-specific Helm values
#   ├── konnector-bundle/           # The Helm chart
#   └── overlays/argocd/
#       ├── kustomization.yaml      # This file
#       └── application.yaml        # The AppProject + Application
---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - application.yaml

Add the new tenant directory to Git, commit the changes, and then push them to your main branch.

PreviousDeploy the Kubernetes Connector via GitOps

NextDeploy with Flux CD

Last updated 1 month ago

Was this helpful?