This is the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

Resource Management

Manage resources across tenants and namespaces, including replication, quotas, and limits with custom quota mechanisms.

1 - Resource Pools

Resource Pools are our answer to manage resources in a multi-tenant Kubernetes cluster. Strategies on granting quotas on tenant-basis

Concept

resourcepools

Benefits

  • Shifting left now comes to Resource-Management. From the perspective of Cluster-Administrators you just define the Quantity and the Audience for Resources. The rest is up to users managing these namespaces (audience).
  • Better automation options and integrations. One important aspect for us is, how we still can be beneficial with concepts like VClusters (VCluster/K3K) or CPs as pods (Kamaji). We think with this solution we have found a way to make capsule still beneficial and even open new use-cases for larger Kubernetes platforms.
  • Enables more use-cases and provides more flexibility than standard ResourceQuotas or our previous ResourceQuota-Implementation. Autobalancing is no longer given by default, however can be implemented according to your platform’s needs see future ideas.

ResourcePool

ResourcePools allow you to define a set of resources, similar to how ResourceQuotas work. ResourcePools are defined at the cluster scope and should be managed by cluster administrators. However, they provide an interface where cluster administrators can specify from which namespaces resources in a ResourcePool can be claimed. Claiming is done via a namespaced CRD called ResourcePoolClaim.

It is then up to the group of users within those namespaces to manage the resources they consume per namespace. Each ResourcePool provisions a ResourceQuota into all the selected namespaces. Essentially, when ResourcePoolClaims are assigned to a ResourcePool, they stack additional resources on top of that ResourceQuota, based on the namespace from which the ResourcePoolClaim was created.

You can create any number of ResourcePools for any kind of namespace — they do not need to be part of a Tenant. Note that the usual ResourceQuota mechanisms apply when, for example, the same resources are defined in multiple ResourcePools for the same namespaces (e.g., the lowest defined quota for a resource is always considered).

apiVersion: capsule.clastix.io/v1beta2
kind: ResourcePool
metadata:
  name: example
spec:
  quota:
    hard:
      limits.cpu: "2"
      limits.memory: 2Gi
      requests.cpu: "2"
      requests.memory: 2Gi
      requests.storage: "5Gi"
  selectors:
  - matchLabels:
      capsule.clastix.io/tenant: example

Selection

The selection of namespaces is done via labels, you can define multiple independent LabelSelectors for a ResourcePool. This gives you a lot of flexibility if you want to span over different kind of namespaces (eg. all namespaces of multiple Tenants, System Namespaces, stages of Tenants etc.)

Here’s an example of a simple Selector:

---
apiVersion: capsule.clastix.io/v1beta2
kind: ResourcePool
metadata:
  name: solar
spec:
  quota:
    hard:
      limits.cpu: "2"
      limits.memory: 2Gi
  selectors:
  - matchLabels:
      capsule.clastix.io/tenant: solar

This will select all the namespaces, which are part of the Tenant solar. Each statement under selectors is treated independent, so for example this is how you can select multiple Tenant’s namespaces:

apiVersion: capsule.clastix.io/v1beta2
kind: ResourcePool
metadata:
  name: green
spec:
  quota:
    hard:
      limits.cpu: "2"
      limits.memory: 2Gi
  selectors:
  - matchLabels:
      capsule.clastix.io/tenant: solar
  - matchLabels:
      capsule.clastix.io/tenant: wind

Quota

Nothing special here, just all the fields you know from ResourceQuotas. The amount defined in quota.hard represents the total resources which can be claimed from the selected namespaces. Through claims the ResourceQuota is then increased or decreased. Note the following:

  • You can’t decrease the .spec.quota.hard if the current allocation from claims is greater than the new decreased number. You must first release claims, to free up that space.
  • You can decrease or remove resources, if they are unused (0)

Other than that, you can use all the fields from ResourceQuotas

---
apiVersion: capsule.clastix.io/v1beta2
kind: ResourcePool
metadata:
  name: best-effort-pool
spec:
  selectors:
  - matchExpressions:
    - { key: capsule.clastix.io/tenant, operator: Exists }
  quota:
    hard:
      cpu: "1000"
      memory: "200Gi"
      pods: "10"
    scopeSelector:
      matchExpressions:
      - operator: In
        scopeName: PriorityClass
        values:
        - "best-effort"

Each ResourcePool is representative for one ResourceQuota. In contrast to the old implementation, where multiple ResourceQuotas could have been defined in a slice. So if you eg. want to use different scopeSelectors or similar, you should create a new ResourcePool for each.

---
apiVersion: capsule.clastix.io/v1beta2
kind: ResourcePool
metadata:
  name: gold-storage
spec:
  selectors:
  - matchExpressions:
    - { key: company.com/env, operator: In, values: [prod, pre-prod] }
  quota:
    hard:
      requests.storage: "10Gi"
      persistentvolumeclaims: "10"
    scopeSelector:
      matchExpressions:
      - operator: In
        scopeName: VolumeAttributesClass
        values: ["gold"]

Defaults

Defaults can contain resources, which are not mentioned in the Quota of a ResourcePool. This is mainly to allow you, to block resources for example:

---
apiVersion: capsule.clastix.io/v1beta2
kind: ResourcePool
metadata:
  name: example
spec:
  defaults:
    requests.storage: "0Gi"
  quota:
    hard:
      limits.cpu: "2"
      limits.memory: 2Gi
      requests.cpu: "2"
      requests.memory: 2Gi
      requests.storage: "5Gi"
  selectors:
  - matchLabels:
      capsule.clastix.io/tenant: example

This results in a ResourceQuota from this pool in all selected, which blocks the allocation of requests.storage:

NAME                     AGE   REQUEST                 LIMIT
capsule-pool-example     3s    requests.storage: 0/0

If no Defaults are defined, the ResourceQuota for the ResourcePool is still provisioned but it’s .spec.hard is empty.

---
apiVersion: capsule.clastix.io/v1beta2
kind: ResourcePool
metadata:
  name: example
spec:
  quota:
    hard:
      limits.cpu: "2"
      limits.memory: 2Gi
      requests.cpu: "2"
      requests.memory: 2Gi
      requests.storage: "5Gi"
  selectors:
  - matchLabels:
      capsule.clastix.io/tenant: example

This allows users to essentially schedule anything in the namespace:

NAME                     AGE     REQUEST           LIMIT
capsule-pool-exmaple     2m47s

To prevent this, you might consider using the DefaultsZero option. This option can also be combined with setting other defaults, not part of the .spec.quota.hard. Here we are additionally restricting the creation of persistentvolumeclaims:

apiVersion: capsule.clastix.io/v1beta2
kind: ResourcePool
metadata:
  name: example
spec:
  defaults:
    "count/persistentvolumeclaims": 3
  config:
    defaultsZero: true
  quota:
    hard:
      limits.cpu: "2"
      limits.memory: 2Gi
      requests.cpu: "2"
      requests.memory: 2Gi
      requests.storage: "5Gi"
  selectors:
  - matchLabels:
      capsule.clastix.io/tenant: example

Results in:

NAME                     AGE   REQUEST                                                                                             LIMIT
capsule-pool-example     10h   count/persistentvolumeclaims: 0/3, requests.cpu: 0/0, requests.memory: 0/0, requests.storage: 0/0   limits.cpu: 0/0, limits.memory: 0/0

Options

Options that can be defined on a per-ResourcePool basis and influence the general behavior of the ResourcePool.

OrderedQueue

When ResourecePoolClaims are allocated to a pool, they are placed in a queue. The pool attempts to allocate claims in the order of their creation timestamps. However, even if a claim was created earlier, if it requests more resources than are currently available, it will remain in the queue. Meanwhile, a lower-priority claim that fits within the available resources may still be allocated—despite its lower priority.

Enabling this option enforces strict ordering: claims cannot be skipped, even if they block other claims from being fulfilled due to resource exhaustion. The CreationTimestamp is strictly respected, meaning that once a claim is queued, no subsequent claim can bypass it—even if it requires fewer resources.

Default: false

DefaultsZero

Sets the default values for the ResourceQuota created for the ResourcePool. When enabled, all resources in the quota are initialized to zero. This is useful in scenarios where users should not be able to consume any resources without explicitly creating claims. In such cases, it makes sense to initialize all available resources in the ResourcePool to 0.

Default: false

DeleteBoundResources

By default, when a ResourcePool is deleted, any ResourcePoolClaims bound to it are only disassociated—not deleted. Enabling this option ensures that all ResourcePoolClaims in a bound state are deleted when the corresponding ResourcePool is deleted.

Default: false

LimitRanges

When defining ResourcePools you might want to consider distributing LimitRanges via Tenant Replications:

apiVersion: capsule.clastix.io/v1beta2
kind: TenantResource
metadata:
  name: example
  namespace: solar-system
spec:
  resyncPeriod: 60s
  resources:
    - namespaceSelector:
        matchLabels:
          capsule.clastix.io/tenant: example
      rawItems:
        - apiVersion: v1
          kind: LimitRange
          metadata:
            name: cpu-resource-constraint
          spec:
            limits:
            - default: # this section defines default limits
                cpu: 500m
              defaultRequest: # this section defines default requests
                cpu: 500m
              max: # max and min define the limit range
                cpu: "1"
              min:
                cpu: 100m
              type: Container

ResourcePoolClaims

ResourcePoolClaims declared claims of resources from a single ResourcePool. When a ResourcePoolClaim is successfully bound to a ResourcePool, it’s requested resources are stacked to the ResourceQuota from the ResourcePool in the corresponding namespaces, where the ResourcePoolClaim was declared. So the declaration of a ResourcePoolClaim is very simple:

apiVersion: capsule.clastix.io/v1beta2
kind: ResourcePoolClaim
metadata:
  name: get-me-cpu
  namespace: solar-test
spec:
  pool: "sample"
  claim:
    requests.cpu: "2"
    requests.memory: 2Gi

ResourcePoolClaims are decoupled from the lifecycle of ResourcePools. If a ResourcePool is deleted where a ResourcePoolClaim was bound to, the ResourcePoolClaim becomes unassigned, but is not deleted.

Allocation

The Connection between ResourcePools and ResourcePoolClaims is done via the .spec.pool field. With that field you must be very specific, from which ResourcePool a ResourcePoolClaim claims resources. On the counter-part, the ResourcePool, the namespace from the ResourcePoolClaim must be allowed to claim resources from the ResourcePool.

If you are trying to allocate a Pool which does not exist or is not allowed to be claimed from, from the namespace the ResourcePoolClaim was made, you will get a failed Assigned status:

solar-test   get-me-cpu          Assigned   Failed   ResourcePool.capsule.clastix.io "sample" not found   12s

Similar errors may occur if you are trying to claim resources from a pool, where the given resources are not claimable.

Auto-Assignment

If no .spec.pool was delivered a Webhook will try to evaluate a matching ResourcePool for the ResourcePoolClaim. In that process of evaluation the following criteria are considered:

  • A ResourcePool has all the resources in their definition available the ResourcePoolClaim is trying to claim.

If no Pool can be auto-assigned, the ResourcePoolClaim will enter an Unassigned state. Where it remains until ResourcePools considering the namespaces the ResourcePoolClaim is deployed in have more resources or a new ResourcePool is defined manually.

The Auto-Assignment Process is only executed, when .spec.pool is unset on Create or Update operations.

Bound

A ResourcePoolClaim is considered Bound, when the requested resources from the claim were successfully allocated from the ResourcePool. And the resources are actually used by any ResourceQuota in the namespace the claim was created in. If the resources are not used yet, the ResourcePoolClaims is considered Unused and can be deleted, change to a different ResourcePool or released without any further actions. However when it’s resources are used, the claim is Bound and can not be modified or deleted until the resources are released (not longer in use).

The selection of which ResourcePoolClaim is Bound is based on a greedy pattern. Meaning we sort the ResourcePoolClaims by their CreationTimestamp and try to allocate them one by one until no more resources are available from the ResourcePool.

Let’s see this in action. We can see that both claims are unused and can be released.

kubectl get resourcepoolclaim -n solar-test

NAME             POOL         READY   MESSAGE      BOUND   REASON            AGE
get-me-solar     solar-pool   True    reconciled   False   claim is unused   9h
get-me-solar-2   solar-pool   True    reconciled   False   claim is unused   9h


kubectl get resourcequota  -n solar-test

NAME                         REQUEST                                       LIMIT   AGE
capsule-pool-solar-pool      requests.cpu: 4/4, requests.memory: 4Gi/4Gi           7m53s

We now create a pod to consume the amount of resources provided by the claim get-me-solar (cpu: 2 and memory: 2Gi). We can see that half of the claim is now used:

kubectl get resourcepoolclaim -n solar-test

NAME             POOL         READY   MESSAGE      BOUND   REASON            AGE
get-me-solar     solar-pool   True    reconciled   True    claim is used     12m
get-me-solar-2   solar-pool   True    reconciled   False   claim is unused   12m

kubectl get resourcequota  -n solar-test

NAME                         REQUEST                                       LIMIT   AGE
capsule-pool-solar-pool      requests.cpu: 2/4, requests.memory: 2Gi/4Gi           11m

We can remove get-me-solar-2, as it’s still unused:

kubectl delete resourcepoolclaim -n solar-test get-me-solar-2

resourcepoolclaim.capsule.clastix.io "get-me-solar-2" deleted

However interactions with get-me-solar are now limited, as it’s Bound:

kubectl delete resourcepoolclaim -n solar-test get-me-solar

Error from server (Forbidden): admission webhook "resourcepoolclaims.projectcapsule.dev" denied the request: cannot delete the pool while claim is used in resourcepool solar-pool

If we remove the pod again, the ResourcePoolClaim becomes unused again and can be deleted or modified.

kubectl get resourcepoolclaim -n solar-test

NAME           POOL         READY   MESSAGE      BOUND   REASON            AGE
get-me-solar   solar-pool   True    reconciled   False   claim is unused   16m


kubectl get resourcequota  -n solar-test

NAME                         REQUEST                                     LIMIT   AGE
capsule-pool-solar-pool      requests.cpu: 0/2, requests.memory: 0/2Gi           17m

Release

If a ResourcePoolClaim is deleted, the resources are released back to the ResourcePool. This means that the resources are no longer reserved for the claim and can be used by other claims.

  • By deleting the ResourcePoolClaim object (Recommended).
  • By annotating the ResourcePoolClaim with projectcapsule.dev/release: "true". This will release the ResourcePoolClaim from the ResourcePool without deleting the object itself and instantly requeue.

Both these actions can only be performed if the ResourcePoolClaim is in a Bound state False (not used currently). Otherwise your first have to free the resources used by the claim in order to release it. You can verify the Bound state for all ResourcePoolClaims in a namespace with.

Immutable

Once a ResourcePoolClaim has successfully claimed resources from a ResourcePool, the claim is immutable. This means that the claim cannot be modified or deleted until the resources have been released back to the ResourcePool. This means ResourcePoolClaim can not be expanded or shrunk, without releasing.

Queue

ResourcePoolClaims can always be created, even if the targeted ResourcePool does not have enough resources available at the time. In that case ResourcePoolClaims are put into a Queue-State, where they wait until they can claim the resources they are after. They following describes the different exhaustion indicators and what they mean, in case a ResourcePoolClaim gets scheduled.

When a ResourcePoolClaims is in Queued-State it is still mutable. So Resources and Pool-Assignment can still be changed.

Exhaustions

There are different types of exhaustions which may occur when attempting to allocate a claim. They Status of each claim indicates

PoolExhausted

The requested resources are not available on the ResourcePool. Until other resources release resources or the pool size is increased the ResourcePoolClaim is queued. In this example the ResourcePoolClaim is trying to claim requests.memory=2Gi. However only requests.memory=1Gi are still available to be claimed from the ResourcePool

NAMESPACE    NAME         POOL      STATUS   REASON          MESSAGE                                                          AGE
solar-test   get-mem      sampler   Bound    QueueExhausted   requested: requests.memory=2Gi, queued: requests.memory=1Gi   9m19s

In this case you have the following options:

  1. Request less resources for claiming - requests.memory=1Gi
  2. Wait until resources become from the ResourcePool. When 1Gi of requests.memory gets released, the ResourcePoolClaim will be able to bind requests.memory=2Gi.
  3. Release another ResourcePoolClaim which might free up requests.memory

However, claims which are requesting less than the ResourcePoolClaim solar-test, will be able to allocate their resources. Let’s say we have this second ResourcePoolClaim:

---
apiVersion: capsule.clastix.io/v1beta2
kind: ResourcePoolClaim
metadata:
  name: skip-the-line
  namespace: solar-test
spec:
  pool: "sampler"
  claim:
    requests.memory: 512Mi

Applying this ResourcePoolClaim leads to it being able to bind these resources. This behavior can be controlled with orderedQueue.

NAMESPACE    NAME            POOL      STATUS   REASON          MESSAGE                                                            AGE
solar-test   get-me-cpu      sampler   Bound    PoolExhausted   requested: requests.memory=2Gi, available: requests.memory=512Mi   16m
solar-test   skip-the-line   sampler   Bound    Succeeded       Claimed resources                                                  2s

If orderedQueue is enabled, only the first item that exhausted a resource from the ResourcePool get the PoolExhausted state. Following claims fro the same resources get QueueExhausted.

QueueExhausted

A ResourcePoolClaim with higher priority is trying to allocate these resources, but is exhausting the ResourcePool. The ResourcePool has orderedQueue enabled, meaning that the ResourcePoolClaim with the highest priority must first schedule it’s resources, before any other ResourcePoolClaim can claim further resources. This queue is resource based (eg. requests.memory), ResourcePoolClaim with lower priority may still be Bound, if they are not trying to allocate resources which are being exhausted by another ResourcePoolClaim with highest priority.

NAMESPACE    NAME         POOL      STATUS   REASON          MESSAGE                                                          AGE
solar-test   get-mem      sampler   Bound    QueueExhausted   requested: requests.memory=2Gi, queued: requests.memory=1Gi   9m19s

The above means, that as ResourcePoolClaim with higher priority is trying to allocate requests.memory=1Gi but that already leads to an PoolExhausted for that ResourcePoolClaim.

Priority

The Priority of how the claims are processed, is deterministic defined based on the following order of attributes from each claim:

  • CreationTimestamp - Oldest first
  • Name - Tiebreaker
  • Namespace - Tiebreaker

Tiebreaker: If two claims have the same CreationTimestamp, they are then sorted alphabetically by their Name. If two claims have the same CreationTimestamp and Name, they are then sorted alphabetically by their Namespace. This means that if two claims are created at the same time, and have the same name, the claim with the lexicographically smaller Name will be processed first. If two claims have the same CreationTimestamp, Name, and Namespace, then the namespace is tiebreaking. This may be relevant in GitOps setups.

Operating

Monitoring

Dashboards can be deployed via helm-chart, enable the following values:

monitoring:
  dashboards:
    enabled: true

Dashboard which grants a detailed overview over the ResourcePools

Resourcepool Dashboard

Rules

Example rules to give you some idea, what’s possible.

  1. Alert on ResourcePools usage
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: capsule-resourcepools-alerts
spec:
groups:
  - name: capsule-resourcepools.rules
    rules:
      - alert: CapsuleResourcePoolHighUsageWarning
        expr: |
          capsule_pool_usage_percentage > 90          
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: High resource usage in Resourcepool
          description: |
            Resource {{ $labels.resource }} in pool {{ $labels.pool }} is at {{ $value }}% usage for the last 10 minutes.            

      - alert: CapsuleResourcePoolHighUsageCritical
        expr: |
          capsule_pool_usage_percentage > 95          
        for: 10m
        labels:
          severity: critical
        annotations:
          summary: Critical resource usage in Resourcepool
          description: |
            Resource {{ $labels.resource }} in pool {{ $labels.pool }} has exceeded 95% usage for the last 10 minutes.            

      - alert: CapsuleResourcePoolExhausted
        expr: |
          capsule_pool_condition{condition="Exhausted"} == 1          
        for: 60m
        labels:
          severity: critical
        annotations:
          summary: Resource pool exhausted
          description: |
            Pool {{ $labels.pool }} has been Exhausted for more than 60 minutes.            

      - alert: CapsuleResourcePoolNotReady
        expr: |
          capsule_pool_condition{condition="Ready"} == 0          
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: Resource pool not ready
          description: |
            Pool {{ $labels.pool }} has not been Ready for more than 10 minutes.            


  - name: capsule-resourcepoolclaims.rules
    rules:
      - alert: CapsuleResourcePoolClaimExhausted
        expr: |
          capsule_claim_condition{condition="Exhausted"} == 1          
        for: 24h
        labels:
          severity: critical
        annotations:
          summary: ResourcePoolClaim exhausted
          description: |
            ResourcePoolClaim {{ $labels.name }}/{{ $labels.target_namespace }} has been Exhausted for more than 24 hours.            

      - alert: CapsuleResourcePoolClaimNotReady
        expr: |
          capsule_claim_condition{condition="Ready"} == 0          
        for: 60m
        labels:
          severity: warning
        annotations:
          summary: ResourcePoolClaim not ready
          description: |
            ResourcePoolClaim {{ $labels.name }}/{{ $labels.target_namespace }}  has not been Ready for more than 60 minutes.

Metrics

The following Metrics are exposed and can be used for monitoring:


# HELP capsule_claim_condition The current condition status of a claim.
# TYPE capsule_claim_condition gauge
capsule_claim_condition{condition="Bound",name="get-me-customer",target_namespace="solar-test"} 1
capsule_claim_condition{condition="Bound",name="get-me-solar",target_namespace="solar-test"} 1
capsule_claim_condition{condition="Bound",name="get-me-solar-2",target_namespace="solar-test"} 0
capsule_claim_condition{condition="Exhausted",name="get-me-customer",target_namespace="solar-test"} 0
capsule_claim_condition{condition="Exhausted",name="get-me-solar",target_namespace="solar-test"} 0
capsule_claim_condition{condition="Exhausted",name="get-me-solar-2",target_namespace="solar-test"} 1
capsule_claim_condition{condition="Ready",name="get-me-customer",target_namespace="solar-test"} 1
capsule_claim_condition{condition="Ready",name="get-me-solar",target_namespace="solar-test"} 1
capsule_claim_condition{condition="Ready",name="get-me-solar-2",target_namespace="solar-test"} 1

# HELP capsule_claim_pool The current assigned pool of a claim.
# TYPE capsule_claim_pool gauge
capsule_claim_pool{name="get-me-solar",pool="solar-compute",target_namespace="solar-test"} 1
capsule_claim_pool{name="get-me-solar-2",pool="solar-compute",target_namespace="solar-test"} 1

# HELP capsule_claim_resource The given amount of resources from the claim
# TYPE capsule_claim_resource gauge
capsule_claim_resource{name="compute",resource="limits.cpu",target_namespace="solar-prod"} 0.375
capsule_claim_resource{name="compute",resource="limits.memory",target_namespace="solar-prod"} 4.02653184e+08
capsule_claim_resource{name="compute",resource="requests.cpu",target_namespace="solar-prod"} 0.375
capsule_claim_resource{name="compute",resource="requests.memory",target_namespace="solar-prod"} 4.02653184e+08
capsule_claim_resource{name="compute-10",resource="limits.memory",target_namespace="solar-prod"} 1.073741824e+10
capsule_claim_resource{name="compute-2",resource="limits.cpu",target_namespace="solar-prod"} 0.5
capsule_claim_resource{name="compute-2",resource="limits.memory",target_namespace="solar-prod"} 5.36870912e+08
capsule_claim_resource{name="compute-2",resource="requests.cpu",target_namespace="solar-prod"} 0.5
capsule_claim_resource{name="compute-2",resource="requests.memory",target_namespace="solar-prod"} 5.36870912e+08
capsule_claim_resource{name="compute-3",resource="requests.cpu",target_namespace="solar-prod"} 0.5
capsule_claim_resource{name="compute-4",resource="requests.cpu",target_namespace="solar-test"} 0.5
capsule_claim_resource{name="compute-5",resource="requests.cpu",target_namespace="solar-test"} 0.5
capsule_claim_resource{name="compute-6",resource="requests.cpu",target_namespace="solar-test"} 5
capsule_claim_resource{name="pods",resource="pods",target_namespace="solar-test"} 3

# HELP capsule_pool_available Current resource availability for a given resource in a resource pool
# TYPE capsule_pool_available gauge
capsule_pool_available{pool="solar-compute",resource="limits.cpu"} 1.125
capsule_pool_available{pool="solar-compute",resource="limits.memory"} 1.207959552e+09
capsule_pool_available{pool="solar-compute",resource="requests.cpu"} 0.125
capsule_pool_available{pool="solar-compute",resource="requests.memory"} 1.207959552e+09
capsule_pool_available{pool="solar-size",resource="pods"} 4


# HELP capsule_pool_condition Current conditions for a given resource in a resource pool
# TYPE capsule_pool_condition gauge
capsule_pool_condition{condition="Exhausted",pool="solar-size"} 0
capsule_pool_condition{condition="Exhausted",pool="solar-compute"} 1
capsule_pool_condition{condition="Ready",pool="solar-size"} 1
capsule_pool_condition{condition="Ready",pool="solar-compute"} 1

# HELP capsule_pool_exhaustion Resources become exhausted, when there's not enough available for all claims and the claims get queued
# TYPE capsule_pool_exhaustion gauge
capsule_pool_exhaustion{pool="solar-compute",resource="limits.memory"} 1.073741824e+10
capsule_pool_exhaustion{pool="solar-compute",resource="requests.cpu"} 5.5

# HELP capsule_pool_exhaustion_percentage Resources become exhausted, when there's not enough available for all claims and the claims get queued (Percentage)
# TYPE capsule_pool_exhaustion_percentage gauge
capsule_pool_exhaustion_percentage{pool="solar-compute",resource="limits.memory"} 788.8888888888889
capsule_pool_exhaustion_percentage{pool="solar-compute",resource="requests.cpu"} 4300

# HELP capsule_pool_limit Current resource limit for a given resource in a resource pool
# TYPE capsule_pool_limit gauge
capsule_pool_limit{pool="solar-compute",resource="limits.cpu"} 2
capsule_pool_limit{pool="solar-compute",resource="limits.memory"} 2.147483648e+09
capsule_pool_limit{pool="solar-compute",resource="requests.cpu"} 2
capsule_pool_limit{pool="solar-compute",resource="requests.memory"} 2.147483648e+09
capsule_pool_limit{pool="solar-size",resource="pods"} 7

# HELP capsule_pool_namespace_usage Current resources claimed on namespace basis for a given resource in a resource pool for a specific namespace
# TYPE capsule_pool_namespace_usage gauge
capsule_pool_namespace_usage{pool="solar-compute",resource="limits.cpu",target_namespace="solar-prod"} 0.875
capsule_pool_namespace_usage{pool="solar-compute",resource="limits.memory",target_namespace="solar-prod"} 9.39524096e+08
capsule_pool_namespace_usage{pool="solar-compute",resource="requests.cpu",target_namespace="solar-prod"} 1.375
capsule_pool_namespace_usage{pool="solar-compute",resource="requests.cpu",target_namespace="solar-test"} 0.5
capsule_pool_namespace_usage{pool="solar-compute",resource="requests.memory",target_namespace="solar-prod"} 9.39524096e+08
capsule_pool_namespace_usage{pool="solar-size",resource="pods",target_namespace="solar-test"} 3

# HELP capsule_pool_namespace_usage_percentage Current resources claimed on namespace basis for a given resource in a resource pool for a specific namespace (percentage)
# TYPE capsule_pool_namespace_usage_percentage gauge
capsule_pool_namespace_usage_percentage{pool="solar-compute",resource="limits.cpu",target_namespace="solar-prod"} 43.75
capsule_pool_namespace_usage_percentage{pool="solar-compute",resource="limits.memory",target_namespace="solar-prod"} 43.75
capsule_pool_namespace_usage_percentage{pool="solar-compute",resource="requests.cpu",target_namespace="solar-prod"} 68.75
capsule_pool_namespace_usage_percentage{pool="solar-compute",resource="requests.cpu",target_namespace="solar-test"} 25
capsule_pool_namespace_usage_percentage{pool="solar-compute",resource="requests.memory",target_namespace="solar-prod"} 43.75
capsule_pool_namespace_usage_percentage{pool="solar-size",resource="pods",target_namespace="solar-test"} 42.857142857142854

# HELP capsule_pool_resource Type of resource being used in a resource pool
# TYPE capsule_pool_resource gauge
capsule_pool_resource{pool="solar-compute",resource="limits.cpu"} 1
capsule_pool_resource{pool="solar-compute",resource="limits.memory"} 1
capsule_pool_resource{pool="solar-compute",resource="requests.cpu"} 1
capsule_pool_resource{pool="solar-compute",resource="requests.memory"} 1
capsule_pool_resource{pool="solar-size",resource="pods"} 1

# HELP capsule_pool_usage Current resource usage for a given resource in a resource pool
# TYPE capsule_pool_usage gauge
capsule_pool_usage{pool="solar-compute",resource="limits.cpu"} 0.875
capsule_pool_usage{pool="solar-compute",resource="limits.memory"} 9.39524096e+08
capsule_pool_usage{pool="solar-compute",resource="requests.cpu"} 1.875
capsule_pool_usage{pool="solar-compute",resource="requests.memory"} 9.39524096e+08
capsule_pool_usage{pool="solar-size",resource="pods"} 3

# HELP capsule_pool_usage_percentage Current resource usage for a given resource in a resource pool (percentage)
# TYPE capsule_pool_usage_percentage gauge
capsule_pool_usage_percentage{pool="solar-compute",resource="limits.cpu"} 43.75
capsule_pool_usage_percentage{pool="solar-compute",resource="limits.memory"} 43.75
capsule_pool_usage_percentage{pool="solar-compute",resource="requests.cpu"} 93.75
capsule_pool_usage_percentage{pool="solar-compute",resource="requests.memory"} 43.75
capsule_pool_usage_percentage{pool="solar-size",resource="pods"} 42.857142857142854

Migration

To Migrate from the old ResourceQuota to ResourcePools, you can follow the steps below. This guide assumes you want to port the old ResourceQuota to the new ResourcePools in exactly the same capacity and scope.

The steps shown are an example to migrate a single Tenants ResourceQuota to a ResourcePool.

1. Overview

We are working with the following tenant. Asses the Situation of resourceQuotas. This guide is mainly relevant if the scope is Tenant:

apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
  labels:
    kubernetes.io/metadata.name: migration
  name: migration
spec:
  owners:
  - clusterRoles:
    - admin
    - capsule-namespace-deleter
    kind: User
    name: bob
  preventDeletion: false
  resourceQuotas:
    items:
    - hard:
        limits.cpu: "2"
        limits.memory: 2Gi
        requests.cpu: "2"
        requests.memory: 2Gi
    - hard:
        pods: "7"
    scope: Tenant
status:
  namespaces:
  - migration-dev
  - migration-prod
  - migration-test
  size: 3
  state: Active

2. Abstracting to ResourcePools

We are now abstracting . For each item, we are creating a ResourcePool with the same values. The ResourcePool will be scoped to the Tenant and will be used for all namespaces in the tenant. Let’s first migrate the first item:

---
apiVersion: capsule.clastix.io/v1beta2
kind: ResourcePool
metadata:
  name: migration-compute
spec:
  config:
    defaultsZero: true
  selectors:
  - matchLabels:
      capsule.clastix.io/tenant: migration
  quota:
    hard:
      limits.cpu: "2"
      limits.memory: 2Gi
      requests.cpu: "2"
      requests.memory: 2Gi

The naming etc. is up to you. Important, we again select all namespaces from the migration tenant with the selector capsule.clastix.io/tenant: migration. The defined config is what we deem to be most compatible with the old ResourceQuota behavior. You may change these according to your requirements.

The same process can be repeated for the second item (or each of your items). The ResourcePool will be scoped to the Tenant and will be used for all namespaces in the tenant. Let’s migrate the second item:

---
apiVersion: capsule.clastix.io/v1beta2
kind: ResourcePool
metadata:
  name: migration-size
spec:
  config:
    defaultsZero: true
  selectors:
  - matchLabels:
      capsule.clastix.io/tenant: migration
  quota:
    hard:
      pods: "7"

3. Create ResourcePoolClaims

Now we need to create the ResourcePoolClaims for the ResourcePools. The ResourcePoolClaims are used to claim resources from the ResourcePools to the respective namespaces. Let’s start with the namespace migration-dev:

kubectl get resourcequota -n migration-dev

NAME                  AGE     REQUEST                                                   LIMIT
capsule-migration-0   5m21s   requests.cpu: 375m/1500m, requests.memory: 384Mi/1536Mi   limits.cpu: 375m/1500m, limits.memory: 384Mi/1536Mi
capsule-migration-1   5m21s   pods: 3/3

Our goal is now to port the current usage into ResourcePoolClaims. Here you must make sure, that you might need to allocate more resources to your claims, than currently is needed (eg. to allow RollingUpdates etc.).. For the example we are porting the current usage over 1:1 to ResourcePoolClaims

We created the ResourcePool named migration-compute, where we are going to claim the resources from (for capsule-migration-0). This results in the following ResourcePoolClaim:

---
apiVersion: capsule.clastix.io/v1beta2
kind: ResourcePoolClaim
metadata:
  name: compute
  namespace: migration-dev
spec:
  pool: "migration-compute"
  claim:
    requests.cpu: 375m
    requests.memory: 384Mi
    limits.cpu: 375m
    limits.memory: 384Mi

The same can be done for the capsule-migration-1 ResourceQuota.

---
apiVersion: capsule.clastix.io/v1beta2
kind: ResourcePoolClaim
metadata:
  name: pods
  namespace: migration-dev
spec:
  pool: "migration-size"
  claim:
    pods: "3"

You can create the claims, they will remain in failed state until we apply the ResourcePools:

kubectl get resourcepoolclaims -n migration-dev

NAME      POOL   STATUS     REASON   MESSAGE                                                         AGE
compute          Assigned   Failed   ResourcePool.capsule.clastix.io "migration-compute" not found   2s
pods             Assigned   Failed   ResourcePool.capsule.clastix.io "migration-size" not found      2s

4. Applying and Verifying the ResourcePools

You may now apply the ResourcePools prepared in step 2:

kubectl apply -f pools.yaml
resourcepool.capsule.clastix.io/migration-compute created
resourcepool.capsule.clastix.io/migration-size created

After applying, you should instantly see, that the ResourcePoolClaims in the migration-dev namespace could be Bound to the corresponding ResourcePools:

kubectl get resourcepoolclaims -n migration-dev
NAME      POOL                STATUS   REASON      MESSAGE             AGE
compute   migration-compute   Bound    Succeeded   Claimed resources   4m9s
pods      migration-size      Bound    Succeeded   Claimed resources   4m9s

Now you can verify the new ResourceQuotas in the migration-dev namespace:

kubectl get resourcequota -n migration-dev
NAME                             AGE    REQUEST                                                   LIMIT
capsule-migration-0              23m    requests.cpu: 375m/1500m, requests.memory: 384Mi/1536Mi   limits.cpu: 375m/1500m, limits.memory: 384Mi/1536Mi
capsule-migration-1              23m    pods: 3/3

capsule-pool-migration-compute   110s   requests.cpu: 375m/375m, requests.memory: 384Mi/384Mi     limits.cpu: 375m/375m, limits.memory: 384Mi/384Mi
capsule-pool-migration-size      110s   pods: 3/3

That looks already super promising. Now You need to repeat these steps for migration-prod and migration-test. (Script Contributions are welcome).

5. Removing old ResourceQuotas

Once we have migrated all resources over the ResourcePoolClaims, we can remove the ResourceQuota system. First of all, we are removing the .spec.resourceQuotas entirely. Currently it will again add the .spec.resourceQuotas.scope field, important is, that no more .spec.resourceQuotas.items exist:

apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
  labels:
    kubernetes.io/metadata.name: migration
  name: migration
spec:
  owners:
  - clusterRoles:
    - admin
    - capsule-namespace-deleter
    kind: User
    name: bob
  resourceQuotas: {}

This will remove all ResourceQuotas from namespace, verify like:

kubectl get resourcepoolclaims -n migration-dev

capsule-pool-migration-compute   130m   requests.cpu: 375m/375m, requests.memory: 384Mi/384Mi   limits.cpu: 375m/375m, limits.memory: 384Mi/384Mi
capsule-pool-migration-size      130m   pods: 3/3

Success 🍀

Why this is our answer

This part should provide you with a little bit of back story, as to why this implementation was done the way it currently is. Let’s start.

Since the begining of capsule we are struggeling with a concurrency probelm regarding ResourcesQuotas, this was already early detected in Issue 49. Let’s quickly recap what really the problem is with the current ResourceQuota centric approach.

With the current ResourceQuota with Scope: Tenant we encounter the problem, that resourcequotas spread across multiple namespaces refering to one tenant quota can be overprovisioned, if an operation is executed in parallel (eg. total is services/count: 3, in each namespace you could then create 3 services, leading to a possible overprovision of hard * amount-namespaces). The Problem in this approach is, that we are not doing anything with Webhooks, therefor we rely on the speed of the controller, where this entire construct becomes a matter of luck and racing conditions.

So, there needs to be change. But times have also changed and we have listened to our users, so the new approach to ResourceQuotas should:

  • Not exclusively be scoped to one Tenant. Often scenarios include granting resources to multiple Tenants eg.
    • When a application has multiple stages split into multiple stages
    • An Application-Team owns multiple Tenants
    • You want to share resources amongst applications of the same stage.
  • Select based on namespaces, even if they are not part of the Tenant ecosystem. Often the requirement to control resources for operators, which make up your distribution, must also be guardlined across n-namespaces.
  • Supplement new generation technology like Kamaji, vCluster or K3K. All these tool abstract Kubernetes into Pods. We also want to provide a solution which still proves capsule relevant in combination with such modern tools.
  • Shifting Resource-Management to Tenant-Owners while Cluster-Administrators orchestrate a greater Pool of resources.
  • Consistency!!

Our initial Idea for a redesign was simple: What if we just intercepted operations on the resourcequota/status subresource and calculate the offsets (or essentially what still can fit) on a Admission-Webhook. If another operation would have taken place the client operation would have thrown a conflict and rejected the admission, until it retries. Makes sense, right?

Here we have the problem, that even if we would block resourcequota status updates and wait until the actual quantity was added to the total, the resources have already been scheduled. The reason for that, is that the status for resourcequotas is eventually consistent, but what really matters at that moment is the hard spec (see this response from a maintainer kubernetes/kubernetes#123434 (comment)). So essentially no matter the status, you can always provision as much resources, as the .spec.hard of a ResourceQuota indicates. This makes perfect sense, if your ResourceQuota is acting in a single namespace. However in our scenario, we have the same ResourceQuota in n-namespaces. So the overprovisioning problem still persists.

Thinking of other ways: So the next idea was essentially increasing the ResourceQuota.spec.hard based on the workloads which are added to a namespaces (essentially a reversed approach). The workflow for this would look like something like this:

  • All resourcequotas get for their hard spec 0

  • New resource is requested (Evaluation what’s needed at Admission)

  • Controller gives the requested resources to the quota (by adding it to the total and updating the hard)

This way it’s only possible to scheduled “ordered”. In conclusion this would also downscale the resourcequota when the resources are no longer needed. This is how ResourceQuotas from the Kubernetes Core-API reject workload, when you try to allocate a Quantity in a namespaces, but the ResourceQuota does not have enough space.

But there’s some problems with this approach as well:

  • if you eg. schedule a pod and the quota is count/0 there’s no admission call on the resourcequota, which would be the easiest. So we would need to find a way to know, there’s something new requesting resources. For example Rancher works around this problem with namespaced DefaultLimits. But this is not the agile approach we would like to offer.
  • The only indication that I know of is that we get an Event, which we can intercept with admission (ResourceQuota Denied), regarding quotaoverprovision.

If you eg update the resource quota that a pod now has space, it takes some time until that’s registered and actually scheduled (just tested it for pods). I guess the timing depends on the kube-controller-manager flag --concurrent-resource-quota-syncs and/or `–resource-quota-sync-period

So it’s really really difficult to increase quotas by the resources which are actually requested, especially the adding new resources process is where the performance would take a heavy hit.

Still thinking on this idea, the optimal solution would have been to calculate everything at admission and keep the usage vs available state on a global resources but not provisioning namespaced ResourceQuotas. This would have taken a bit pressure from the entire operation, as the resources would not have to be calculated twice (For our GlobalResourceQuota and the core ResourceQuota). In addition we should have added

So that’s when we discarded everything and came up with the concept of ResourcePools and ResourcePoolClaims.

Future Ideas

We want to keep this API as lightweight as possible. But we have already identified use-cases with customers, which make heavy use of ResourcePools:

  • JIT-Claiming: Every Workload queues it’s own claim when being submitted to admission. The respective claims are bound to the lifecycle of the provisioned resource.

  • Node-Population: Populate the Quantity of a ResourcePool based on selected nodes.

2 - Custom Quotas

CustomQuotas let you define and enforce arbitrary, label-scoped limits for any Kubernetes resource kind or CRD, at namespace or cluster scope.

Concept

CustomQuotas complement Kubernetes ResourceQuota by enforcing limits on custom usage metrics extracted from objects themselves.

Capsule provides two quota resources:

  • CustomQuota: namespaced CRD that limits usage inside one namespace. It can only target namespaced resources.
  • GlobalCustomQuota: cluster-scoped CRD that aggregates usage across a set of namespaces selected by label selectors.

A quota is defined by:

  • a limit
  • one or more sources
  • optional selectors to restrict which objects are counted

Each matching object contributes a quantity to the quota. Capsule persists the current aggregate in status.usage.used and keeps the list of counted objects in status.claims.

Calculation

CustomQuotas are calculated in two cooperating parts:

  • Admission webhook: performs enforcement during CREATE, UPDATE, and DELETE
  • Controller reconcile loop: rebuilds the quota status from the actual cluster state and keeps it authoritative

Admission

The admission webhook intercepts operations for the configured resource kinds and evaluates whether the change would violate any matching quota.

For each matching quota it:

  1. matches the object against the quota source GVK
  2. evaluates source selectors
  3. computes the requested usage for the operation
  4. creates or updates a short-lived reservation in the corresponding QuantityLedger
  5. denies the request if persistedUsed + inflightReserved + requested > limit

This makes quota enforcement safe even during bursts of concurrent requests.

Without the Admission Webhook enabled, CustomQuotas are observational only.
The controllers still rebuild and report usage, but requests are not denied.

By default, no objects are sent to this webhook. You must explicitly enable it and configure matching rules.

Example: enable calculations for all Pod create/update/delete operations in tenant namespaces:

webhooks:
  hooks:
    calculations:
      enabled: true
      namespaceSelector:
        matchExpressions:
          - key: capsule.clastix.io/tenant
            operator: Exists
      rules:
          - apiGroups:
              - ""
            apiVersions:
              - ""
            operations:
              - CREATE
              - UPDATE
              - DELETE
            resources:
              - "pods"
            scope: Namespaced

Make sure to configure this webhook carefully, as it can impact cluster performance and availability if it matches a large number of operations. Start with a narrow scope (e.g., specific GVKs and namespace labels) and monitor the impact before expanding it. Also make sure to exclude system critical components or namespaces to avoid accidental disruptions.

webhooks:
  hooks:
    calculations:
      enabled: true
      namespaceSelector:
        matchExpressions:
          - key: name
            operator: NotIn
            values: ["kube-system", "kube-public", "kube-node-lease"]
      rules:
        - apiGroups:
            - ""
          apiVersions:
            - ""
          operations:
            - CREATE
            - UPDATE
            - DELETE
          resources:
            - "pods"
          scope: Namespaced
      matchConditions:
        # Execlude Event and Subresource requests to avoid performance issues and disruptions in case of issues with the webhook (Example).
        - name: ignore-subresources
          expression: '!has(request.subResource) || request.subResource == ""'
        - name: ignore-events
          expression: 'request.resource.resource != "events"'
  
        # Execlude Entities which never count towards quotas to avoid performance issues and disruptions in case of issues with the webhook (Example).
        - name: 'exclude-kubelet-requests'
          expression: '!("system:nodes" in request.userInfo.groups)'
        - name: 'exclude-kube-system'
          expression: '!("system:serviceaccounts:kube-system" in request.userInfo.groups)'

Without the Admission Webhook enabled, CustomQuotas are purely observational and do not enforce limits. You can use this mode to monitor usage and understand the impact before enabling enforcement.

JSONPath

The Custom Quota system relies on JSONPath expressions to extract numeric values from objects. The spec.sources[*].path field defines the JSONPath to the value that should be counted towards the quota. This allows you to define quotas based on any numeric field in any Kubernetes resource, including custom resources.

The following constraints apply to the JSONPath:

  • Expressions must start with a dot (.) and use standard JSONPath syntax. (valid .spec.storage.usage).
  • Paths can not be empty.
  • The maximum length of the path is 1024 characters.
  • Expressions can not contain any of the following characters:
    • \n (newline)
    • \r (carriage return)
    • \t (tab)
  • Values can resolve to array results, which are then summed up. (For example, .spec.containers[*].resources.limits.cpu would sum the CPU limits of all containers in a Pod.)
  • Missing fields are treated as zero (0). We allow Keys to be missing be default. Meaning if you eg define this JP .spec.initContainers[*].resources.limits.cpu on a Pod that has no initContainers, it will simply contribute 0 to the usage instead of causing an error. This is useful for flexibility and to avoid unintended disruptions, but it also means that you need to be careful when defining your JSONPaths to ensure they accurately capture the intended usage.

Quota Matches

As it’s the case with native ResourceQuotas, when a request is made, Capsule evaluates all existing CustomQuotas and GlobalCustomQuotas to determine which ones match the request. Always the smallest quantity of quotas is enforced, meaning that if multiple quotas match a request, the one with the least available capacity will be the one that determines whether the request is allowed or denied.

Let’s look at this example. We have a GlobalCustomQuota targeting all namespaces of the tenant solar with a limit of 6 Pods, and a CustomQuota in the namespace solar-test (part of tenant solar) with a limit of 3 Pods:

---
apiVersion: capsule.clastix.io/v1beta2
kind: GlobalCustomQuota
metadata:
  name: pod-count-limit
spec:
  limit: 6
  namespaceSelectors:
  - matchLabels:
      capsule.clastix.io/tenant: solar
  sources:
  - group: ""
    kind: Pod
    op: count
    version: v1
---
apiVersion: capsule.clastix.io/v1beta2
kind: CustomQuota
metadata:
  name: pod-count-limit
  namespace: solar-test
spec:
  limit: 3
  sources:
  - group: ""
    kind: Pod
    op: count
    version: v1

When we now try to create 6 Pods in the namespace solar-test, we can observe that the GlobalCustomQuota allows only 6 Pods in total across all namespaces of the tenant, while the CustomQuota allows only 3 Pods in the solar-test namespace:

kubectl get pod -n solar-test

NAME                                READY   STATUS    RESTARTS   AGE
nginx-deployment-6ff89574f8-2jbvp   1/1     Running   0          4m20s
nginx-deployment-6ff89574f8-sdzvr   1/1     Running   0          4m20s
nginx-deployment-6ff89574f8-tvk74   1/1     Running   0          4m20s

We can see that requests are blocked because of the limits by the CustomQuota first, as it has the least available capacity (3 available vs 6 available in the GlobalCustomQuota):

115s        Warning   FailedCreate        replicaset/nginx-deployment-6ff89574f8   Error creating: admission webhook "calculation.custom-quotas.projectcapsule.dev" denied the request: creating resource exceeds limit for CustomQuota "pod-count-limit" (requested=1, currentUsed=4, available=0, limit=3, inflightReserved=1)

Namespace Scope

GlobalCustomQuota and CustomQuota can operate in any namespace, they don’t have to be part of a capsule tenant. This means that you can define a CustomQuota in any namespace, even if it’s not part of a tenant, and it will still be enforced for objects in that namespace. Similarly, you can define a GlobalCustomQuota that selects namespaces based on labels, regardless of whether those namespaces are part of a tenant or not.

Race Conditions

GlobalCustomQuotas and CustomQuotas are designed are considered when the target GVK has been posted to their status. If you quickly create workloads that match the GVK of a quota before the quota has been fully reconciled and posted to status, there is a possibility that those workloads are not counted towards the quota usage until the next reconciliation loop. This is because the admission webhook relies on the quota status to determine which quotas to enforce, and if the quota has not yet been reconciled and posted to status, it may not be considered during admission.

Sources

A quota may define one or many sources. Each source describes:

  • which objects are candidates (group, version, kind)
  • what value is extracted from them (path, if applicable)
  • how that value contributes to usage (op)
  • optional additional source-level selectors

In practice, a source answers:

“For objects of this kind, what should count toward the quota?”

Sources are evaluated independently and then aggregated into one total.

GVK

Each source must identify a Kubernetes resource type by Group / Version / Kind. Example for a core Kubernetes Pod:

group: ""
version: v1
kind: Pod

Example for CRDs:

apiVersion: objectbucket.io/v1alpha1
kind: ObjectBucketClaim

apiVersion: s3.aws.upbound.io/v1beta1
kind: Bucket

How matching works:

  • only objects whose GVK exactly matches the source are considered
  • for CustomQuota, the target resource must be namespaced
  • for GlobalCustomQuota, both namespaced Kubernetes resources and namespaced CRDs are supported across all selected namespaces if a source refers to a GVK that is not installed or not discoverable, the controller reports a reconcile failure in the quota condition

A source does not automatically follow subresources, versions, or related objects. If you want to count two kinds, define two sources.

For example, to count both Pods and PVCs, use two sources:

spec:
  sources:
    - apiVersion: v1
      kind: Pod
      op: count
    - apiVersion: v1
      kind: PersistentVolumeClaim
      op: count

Path

path defines which value is extracted from a matching object.

Use path when the operation is add or sub and you want to sum up numeric fields from the objects, such as CPU requests or storage sizes. You can not use path with count.

The path expression leverages JSONPath syntax and must resolve to a numeric value or an array of numeric values. The resulting number is added to the quota usage according to the defined operation. Here some examples of paths:

Count requested PVC storage:

path: .spec.resources.requests.storage

Sum all container CPU requests in a Pod:

path: .spec.containers[*].resources.requests.cpu

Sum ephemeral volume claim sizes declared inside a Pod:

path: .spec.volumes[*].ephemeral.volumeClaimTemplate.spec.resources.requests.storage

Important notes:

  • the extracted value must be parseable as a Kubernetes Quantity
  • if the expression resolves to multiple values, Capsule sums them
  • missing fields contribute 0

Operations

Each source has an op (operation) field. For every matching object, the controller rebuild determines the effective usage contribution per source:

  • add: used += value
  • sub: used -= value
  • count: used += 1

On updates, usage is recalculated from the current object state and the authoritative quota status is rebuilt from scratch from all matching objects.

add

Adds the extracted quantity to the quota usage. Typical use cases:

  • CPU requests
  • memory limits
  • PVC storage
  • emptyDir or ephemeral storage sizes

This is the default behavior.

Example:

spec:
  sources:
    - apiVersion: v1
      kind: PersistentVolumeClaim
      op: add
      path: .spec.resources.requests.storage

sub

Subtracts the extracted quantity from the quota usage. This is useful when you want a source to offset or discount usage from another source.

count

Counts matching objects as 1 each. Rules for count:

  • path must not be set
  • each matching object contributes exactly 1

Example:

spec:
  sources:
    - apiVersion: v1
      kind: Pod
      op: count

Selectors

Each source can optionally include extra selectors to further restrict which objects contribute to usage. Capsule evaluates selectors after the object already matched the source GVK. Each entry will be aggregated with OR semantics, meaning that if an object matches any of the selector entries, it is counted. LabelSelectors and FieldSelectors can be combined within the same selector entry with AND semantics, meaning that an object must match both to be counted.

LabelSelectors

A source selector may contain Kubernetes-style matchLabels / matchExpressions against the object labels.

spec:
  sources:
    - apiVersion: v1
      kind: PersistentVolumeClaim
      op: add
      path: .spec.resources.requests.storage
      selectors:
        - matchExpressions:
            - key: "team"
              operator: In
              values: ["platform", "dev"]
        - matchLabels:
            - key: "team"
              operator: In
              values: ["platform", "dev"]

FieldSelectors

fieldSelectors are additional per-source filters. Each entry is a JSONPath expression evaluated against the candidate object.

A selector entry matches when its JSONPath result is truthy:

  • empty result, false or 0: false
  • any other non-empty result: true

Given:

spec:
  sources:
    - apiVersion: v1
      kind: PersistentVolumeClaim
      op: add
      path: .spec.resources.requests.storage
      selectors:
        - fieldSelectors:
          - '.spec.accessModes[?(@=="ReadWriteOnce")]'
          - '.status.phase'

the selector matches only if:

  • the label selector matches, and
  • .spec.accessModes[?(@=="ReadWriteMany")] returns a non-empty result, and
  • .status.phase returns a non-empty result

Within one selectors entry:

  • labelSelector AND all fieldSelectors

Across multiple selectors entries:

  • OR semantics

FieldSelectors are not Kubernetes API field selectors. They are evaluated by Capsule using JSONPath after the object has been listed.

Examples

Match PVCs that contain ReadWriteMany

selectors:
  - fieldSelectors:
      - '.spec.accessModes[?(@=="ReadWriteMany")]'

Match objects where a field exists

selectors:
  - fieldSelectors:
      - '.spec.storageClassName'

Match objects where a boolean field is true

selectors:
  - fieldSelectors:
      - '.spec.suspend'

If .spec.suspend resolves to true, it matches. If it resolves to false or is missing, it does not match.

Match objects with a specific condition present in an array

selectors:
  - fieldSelectors:
      - '.status.conditions[?(@.type=="Ready")]'

This matches if at least one Ready condition exists.

GlobalCustomQuota

GlobalCustomQuota aggregates usage across multiple namespaces.

Sources

Sources can be distributed across many namespaces. Other than that, they follow the same Sources rules.

Selectors

Selectors preevaluated items considered for the quota. Only items matching the selectors are counted towards usage. Selectors from Sources are applied after the source GVK is matched, so they can be used to further filter which objects are counted based on their labels or fields. However they can’t select items which are not selected by the selectors on GlobalCustomQuota level. This means that if you want to select items across multiple namespaces, you need to use namespaceSelectors and not selectors.

NamespaceSelectors

Definition of spec.namespaceSelectors determines which namespaces are in scope. Only objects from matching namespaces are considered This enforces a 500Gi cap on ObjectBucketClaim storage in namespaces labeled with capsule.clastix.io/tenant=solar:

apiVersion: capsule.clastix.io/v1beta2
kind: GlobalCustomQuota
metadata:
  name: object-bucket-claim-storage
spec:
  limit: "500Gi"
  namespaceSelectors:
    - matchLabels:
        capsule.clastix.io/tenant: solar
  sources:
    - apiVersion: objectbucket.io/v1alpha1
      kind: ObjectBucketClaim
      op: add
      path: .spec.additionalConfig.maxSize

The collected namespaces are also reported in status.namespaces and can be used for informational purposes or by external systems to understand which namespaces are contributing to the quota usage.

kubectl get globalcustomquota pod-count-limit -o yaml

apiVersion: capsule.clastix.io/v1beta2
kind: GlobalCustomQuota
metadata:
  name: object-bucket-claim-storage
spec:
  limit: "500Gi"
  namespaceSelectors:
    - matchLabels:
        capsule.clastix.io/tenant: solar
  sources:
    - apiVersion: objectbucket.io/v1alpha1
      kind: ObjectBucketClaim
      op: add
      path: .spec.additionalConfig.maxSize
status:
  conditions:
  - lastTransitionTime: "2026-04-17T08:13:17Z"
    message: reconciled
    reason: Succeeded
    status: "True"
    type: Ready
  namespaces:
  - solar-prod
  targets:
    - version: v1alpha1
      kind: ObjectBucketClaim
      group: objectbucket.io
      op: add
      path: .spec.additionalConfig.maxSize
  usage:
    available: "500Gi"
    used: "0"

ScopeSelectors

Sources can be distributed across multiple namespaces. Other than that follow #sources rules. This enforces a 500Gi cap on ObjectBucketClaim storage in namespaces labeled with capsule.clastix.io/tenant=solar, counting only claims labeled with objectbucket.io/storage-class=gold:

apiVersion: capsule.clastix.io/v1beta2
kind: GlobalCustomQuota
metadata:
  name: object-bucket-claim-storage
spec:
  limit: "500Gi"
  namespaceSelectors:
    - matchLabels:
        capsule.clastix.io/tenant: solar
  scopeSelectors:
    - matchLabels:
        objectbucket.io/storage-class: gold
  sources:
    - apiVersion: objectbucket.io/v1alpha1
      kind: ObjectBucketClaim
      op: add
      path: .spec.additionalConfig.maxSize

Options

Additional options available for GlobalCustomQuota.

emitMetricPerClaimUsage

Additionaly expose usage metrics for each claim contributing to the quota. This is disabled by default to avoid high cardinality in the metrics, but can be enabled for more granular monitoring and alerting. By default this option is disabled.

apiVersion: capsule.clastix.io/v1beta2
kind: GlobalCustomQuota
metadata:
  name: object-bucket-claim-storage
spec:
  options:
    emitMetricPerClaimUsage: true
  ...

Example metrics:

# HELP capsule_global_custom_quota_resource_item_usage Claimed resources from given item
# TYPE capsule_global_custom_quota_resource_item_usage gauge
capsule_global_custom_quota_resource_item_usage{custom_quota="cpu-limits",group="",kind="Pod",name="nginx-deployment-6ff89574f8-299zf",target_namespace="solar-test"} 0.25
capsule_global_custom_quota_resource_item_usage{custom_quota="cpu-limits",group="",kind="Pod",name="nginx-deployment-6ff89574f8-5hzp9",target_namespace="solar-test"} 0.25
capsule_global_custom_quota_resource_item_usage{custom_quota="cpu-limits",group="",kind="Pod",name="nginx-deployment-6ff89574f8-9zzzw",target_namespace="solar-test"} 0.25
capsule_global_custom_quota_resource_item_usage{custom_quota="cpu-limits",group="",kind="Pod",name="nginx-deployment-6ff89574f8-gnf8f",target_namespace="solar-test"} 0.25
capsule_global_custom_quota_resource_item_usage{custom_quota="cpu-limits",group="",kind="Pod",name="nginx-deployment-6ff89574f8-l68c5",target_namespace="solar-test"} 0.25
capsule_global_custom_quota_resource_item_usage{custom_quota="cpu-limits",group="",kind="Pod",name="nginx-deployment-6ff89574f8-lrzvd",target_namespace="solar-test"} 0.25

Examples

Feel free to contribute examples if you have found interesting use cases!

Limit total max storage across bucket claims for selected namespaces

This enforces a 500Gi cap on max storage requested by ObjectBucketClaims in all namespaces labeled with capsule.clastix.io/tenant=solar, but only counting those claims with the storage class label objectbucket.io/storage-class=gold.

apiVersion: capsule.clastix.io/v1beta2
kind: ClusterCustomQuota
metadata:
  name: object-bucket-claim-storage
spec:
  limit: "500Gi"
  sources:
    - apiVersion: objectbucket.io/v1alpha1
      kind: ObjectBucketClaim
      op: add
      path: .spec.additionalConfig.maxSize
  selectors:
    - matchLabels:
        capsule.clastix.io/tenant: solar
  scopeSelectors:
    - matchLabels:
        objectbucket.io/storage-class: gold

Limit the number of LoadBalancer Services across tenant namespaces

This limits the total number of Services of type LoadBalancer across all namespaces of one tenant.

apiVersion: capsule.clastix.io/v1beta2
kind: GlobalCustomQuota
metadata:
  name: customer-a-loadbalancers
spec:
  limit: 3
  namespaceSelectors:
    - matchLabels:
        customer: a
  sources:
    - apiVersion: v1
      kind: Service
      op: count
      selectors:
        - fieldSelectors:
            - '.spec.type[?(@=="LoadBalancer")]'

Aggregate ephemeral and persistent storage

This policy combines:

  • storage requested by Pod ephemeral volume claim templates
  • storage requested by PVCs
---
apiVersion: capsule.clastix.io/v1beta2
kind: GlobalCustomQuota
metadata:
  name: solar-storage-aggregate
spec:
  limit: 5Gi
  namespaceSelectors:
  - matchLabels:
      capsule.clastix.io/tenant: solar
  sources:
    - apiVersion: v1
      kind: Pod
      op: add
      path: ".spec.volumes[*].ephemeral.volumeClaimTemplate.spec.resources.requests.storage"
    - apiVersion: v1
      kind: PersistentVolumeClaim
      op: add
      path: ".spec.resources.requests.storage"
      selectors:
        - fieldSelectors:
            - '.spec.accessModes[?(@=="ReadWriteOnce")]'

Count Crossplane Buckets across tenant namespaces

This limits how many Crossplane S3 buckets may exist across a tenant.

apiVersion: capsule.clastix.io/v1beta2
kind: GlobalCustomQuota
metadata:
  name: tenant-crossplane-buckets
spec:
  limit: 5
  namespaceSelectors:
    - matchLabels:
        capsule.clastix.io/tenant: solar
  sources:
    - apiVersion: s3.aws.upbound.io/v1beta1
      kind: Bucket
      op: count

Monitoring

See how you can monitor GlobalCustomQuota usage via Prometheus metrics. The example metrics are based on this GlobalCustomQuota definition:

apiVersion: capsule.clastix.io/v1beta2
kind: GlobalCustomQuota
metadata:
  name: cpu-limits
spec:
  limit: 5
  namespaceSelectors:
  - matchLabels:
      capsule.clastix.io/tenant: solar
  sources:
  - apiVersion: v1
    kind: Pod
    op: add
    path: .spec.containers[*].resources.limits.cpu
  - apiVersion: v1
    kind: Pod
    op: add
    path: .spec.initContainers[*].resources.limits.cpu
status:
  claims:
  - group: ""
    kind: Pod
    name: nginx-deployment-6ff89574f8-299zf
    namespace: solar-test
    uid: f7ff7d7c-7128-4f44-ad13-3c44882420f8
    usage: 250m
    version: v1
  - group: ""
    kind: Pod
    name: nginx-deployment-6ff89574f8-9zzzw
    namespace: solar-test
    uid: 24c1bdea-000d-4e10-8af6-eb23c44ceaa3
    usage: 250m
    version: v1
  - group: ""
    kind: Pod
    name: nginx-deployment-6ff89574f8-gnf8f
    namespace: solar-test
    uid: 25368dd6-b3e7-4cbd-9fc8-9082db50372e
    usage: 250m
    version: v1
  - group: ""
    kind: Pod
    name: nginx-deployment-6ff89574f8-l68c5
    namespace: solar-test
    uid: bb697ba6-6512-4d63-acf8-6d058364c9d4
    usage: 250m
    version: v1
  - group: ""
    kind: Pod
    name: nginx-deployment-6ff89574f8-lrzvd
    namespace: solar-test
    uid: 50556db5-0134-4f0a-a0b8-56235f2bdc59
    usage: 250m
    version: v1
  - group: ""
    kind: Pod
    name: nginx-deployment-6ff89574f8-5hzp9
    namespace: solar-test
    uid: 7c6d1252-f649-4106-bfae-22c558c798df
    usage: 250m
    version: v1
  conditions:
  - lastTransitionTime: "2026-04-17T08:29:15Z"
    message: reconciled
    reason: Succeeded
    status: "True"
    type: Ready
  namespaces:
  - solar-prod
  - solar-test
  targets:
  - group: ""
    kind: Pod
    op: add
    path: .spec.containers[*].resources.limits.cpu
    scope: namespace
    version: v1
  - group: ""
    kind: Pod
    op: add
    path: .spec.initContainers[*].resources.limits.cpu
    scope: namespace
    version: v1
  usage:
    available: 3500m
    used: 1500m

Metrics

The following metrics are exposed for each GlobalCustomQuota:

# HELP capsule_global_custom_quota_condition Provides per global custom quota condition status
# TYPE capsule_global_custom_quota_condition gauge
capsule_global_custom_quota_condition{condition="Ready",custom_quota="cpu-limits"} 1

# TYPE capsule_global_custom_quota_resource_limit gauge
capsule_global_custom_quota_resource_limit{custom_quota="cpu-limits"} 5

# TYPE capsule_global_custom_quota_resource_available gauge
capsule_global_custom_quota_resource_available{custom_quota="cpu-limits"} 3.5

# TYPE capsule_global_custom_quota_resource_usage gauge
capsule_global_custom_quota_resource_usage{custom_quota="cpu-limits"} 1.5

## -- Requires .spec.options.emitMetricPerClaimUsage to be enabled
## May cause high cardinality if many claims are present, use with caution.

# HELP capsule_global_custom_quota_resource_item_usage Claimed resources from given item
# TYPE capsule_global_custom_quota_resource_item_usage gauge
capsule_global_custom_quota_resource_item_usage{custom_quota="cpu-limits",group="",kind="Pod",name="nginx-deployment-6ff89574f8-299zf",target_namespace="solar-test"} 0.25
capsule_global_custom_quota_resource_item_usage{custom_quota="cpu-limits",group="",kind="Pod",name="nginx-deployment-6ff89574f8-5hzp9",target_namespace="solar-test"} 0.25
capsule_global_custom_quota_resource_item_usage{custom_quota="cpu-limits",group="",kind="Pod",name="nginx-deployment-6ff89574f8-9zzzw",target_namespace="solar-test"} 0.25
capsule_global_custom_quota_resource_item_usage{custom_quota="cpu-limits",group="",kind="Pod",name="nginx-deployment-6ff89574f8-gnf8f",target_namespace="solar-test"} 0.25
capsule_global_custom_quota_resource_item_usage{custom_quota="cpu-limits",group="",kind="Pod",name="nginx-deployment-6ff89574f8-l68c5",target_namespace="solar-test"} 0.25
capsule_global_custom_quota_resource_item_usage{custom_quota="cpu-limits",group="",kind="Pod",name="nginx-deployment-6ff89574f8-lrzvd",target_namespace="solar-test"} 0.25

CustomQuota

CustomQuota is namespaced and only counts resources in the same namespace as the quota.

Sources

Sources can originate in the same Namespace as the CustomQuota is deployed in. Other than that, they follow the same Sources rules.

Selectors

Selectors preevaluated items considered for the quota. Only items matching the selectors are counted towards usage. Selectors from Sources are applied after the source GVK is matched, so they can be used to further filter which objects are counted based on their labels or fields. However they can’t select items which are not selected by the selectors on CustomQuota level.

ScopeSelectors

Sources can be distributed across multiple namespaces. Other than that follow #sources rules. This enforces a 500Gi cap on ObjectBucketClaim storage counting only claims labeled with objectbucket.io/storage-class=gold:

apiVersion: capsule.clastix.io/v1beta2
kind: CustomQuota
metadata:
  name: object-bucket-claim-storage
  namespace: solar-test
spec:
  limit: "500Gi"
  scopeSelectors:
    - matchLabels:
        objectbucket.io/storage-class: gold
  sources:
    - apiVersion: objectbucket.io/v1alpha1
      kind: ObjectBucketClaim
      op: add
      path: .spec.additionalConfig.maxSize

Options

Additional options available for CustomQuota.

emitMetricPerClaimUsage

Additionaly expose usage metrics for each claim contributing to the quota. This is disabled by default to avoid high cardinality in the metrics, but can be enabled for more granular monitoring and alerting. By default this option is disabled.

apiVersion: capsule.clastix.io/v1beta2
kind: CustomQuota
metadata:
  name: pod-count-limit
  namespace: wind-test
spec:
  options:
    emitMetricPerClaimUsage: true
  ...

Example metrics:

# HELP capsule_custom_quota_resource_item_usage Claimed resources from given item
# TYPE capsule_custom_quota_resource_item_usage gauge
capsule_custom_quota_resource_item_usage{custom_quota="pod-count-limit",group="",kind="Pod",name="nginx-deployment-77bc6bd484-4qm4h",target_namespace="wind-test"} 1
capsule_custom_quota_resource_item_usage{custom_quota="pod-count-limit",group="",kind="Pod",name="nginx-deployment-77bc6bd484-bsnfz",target_namespace="wind-test"} 1
capsule_custom_quota_resource_item_usage{custom_quota="pod-count-limit",group="",kind="Pod",name="nginx-deployment-77bc6bd484-f8qcv",target_namespace="wind-test"} 1

Examples

Feel free to contribute examples if you have found interesting use cases!

Limit total PVC storage in one namespace

apiVersion: capsule.clastix.io/v1beta2
kind: CustomQuota
metadata:
  name: pvc-storage-limit
  namespace: team-a
spec:
  limit: "200Gi"
  scopeSelectors:
    - matchLabels:
        team: platform
  sources:
    - apiVersion: v1
      kind: PersistentVolumeClaim
      op: add
      path: .spec.resources.requests.storage

Limit the number of LoadBalancer Services in one namespace

apiVersion: capsule.clastix.io/v1beta2
kind: CustomQuota
metadata:
  name: namespace-loadbalancers
  namespace: team-a
spec:
  limit: 2
  sources:
    - apiVersion: v1
      kind: Service
      op: count
      selectors:
        - fieldSelectors:
            - '.spec.type[?(@=="LoadBalancer")]'

Limit total memory requests of Pods in one namespace

apiVersion: capsule.clastix.io/v1beta2
kind: CustomQuota
metadata:
  name: pod-memory-requests
  namespace: team-a
spec:
  limit: 16Gi
  sources:
    - apiVersion: v1
      kind: Pod
      op: add
      path: .spec.containers[*].resources.requests.memory
    - apiVersion: v1
      kind: Pod
      op: add
      path: .spec.initContainers[*].resources.requests.memory

Count Crossplane SQL instances in one namespace

apiVersion: capsule.clastix.io/v1beta2
kind: CustomQuota
metadata:
  name: sql-instances
  namespace: team-a
spec:
  limit: 3
  sources:
    - apiVersion: database.gcp.upbound.io/v1beta1
      kind: SQLDatabaseInstance
      op: count

Count only suspended CronJobs

apiVersion: capsule.clastix.io/v1beta2
kind: CustomQuota
metadata:
  name: suspended-cronjobs
  namespace: team-a
spec:
  limit: 5
  sources:
    - apiVersion: batch/v1
      kind: CronJob
      op: count
      selectors:
        - fieldSelectors:
            - '.spec.suspend'

Monitoring

See how you can monitor CustomQuota usage via Prometheus metrics. The example metrics are based on this CustomQuota definition:

apiVersion: capsule.clastix.io/v1beta2
kind: CustomQuota
metadata:
  name: pod-count-limit
  namespace: wind-test
spec:
  limit: 3
  options:
    emitMetricPerClaimUsage: false
  sources:
  - apiVersion: "v1"
    kind: Pod
    op: count
status:
  claims:
  - group: ""
    kind: Pod
    name: nginx-deployment-77bc6bd484-4qm4h
    namespace: wind-test
    uid: c6df70ce-f483-4b02-af65-c8c150d22ed2
    usage: "1"
    version: v1
  - group: ""
    kind: Pod
    name: nginx-deployment-77bc6bd484-f8qcv
    namespace: wind-test
    uid: eeae006b-5ce8-442b-b6c3-f208387545a7
    usage: "1"
    version: v1
  - group: ""
    kind: Pod
    name: nginx-deployment-77bc6bd484-bsnfz
    namespace: wind-test
    uid: 9e1135a7-b286-4768-becd-147b37c999f8
    usage: "1"
    version: v1
  conditions:
  - lastTransitionTime: "2026-04-23T09:21:29Z"
    message: reconciled
    reason: Succeeded
    status: "True"
    type: Ready
  targets:
  - group: ""
    kind: Pod
    op: count
    scope: namespace
    version: v1
  usage:
    available: "0"
    used: "3"

Metrics

The following metrics are exposed for each CustomQuota:

# HELP capsule_custom_quota_condition Provides per custom quota condition status
# TYPE capsule_custom_quota_condition gauge
capsule_custom_quota_condition{condition="Ready",custom_quota="pod-count-limit",target_namespace="wind-test"} 1

# HELP capsule_custom_quota_resource_available Available resources for given custom quota
# TYPE capsule_custom_quota_resource_available gauge
capsule_custom_quota_resource_available{custom_quota="pod-count-limit",target_namespace="wind-test"} 0

# HELP capsule_custom_quota_resource_limit Current resource limit for given custom quota
# TYPE capsule_custom_quota_resource_limit gauge
capsule_custom_quota_resource_limit{custom_quota="pod-count-limit",target_namespace="wind-test"} 3

# HELP capsule_custom_quota_resource_usage Current resource usage for given custom quota
# TYPE capsule_custom_quota_resource_usage gauge
capsule_custom_quota_resource_usage{custom_quota="pod-count-limit",target_namespace="wind-test"} 3


## -- Requires .spec.options.emitMetricPerClaimUsage to be enabled
## May cause high cardinality if many claims are present, use with caution.

# HELP capsule_custom_quota_resource_item_usage Claimed resources from given item
# TYPE capsule_custom_quota_resource_item_usage gauge
capsule_custom_quota_resource_item_usage{custom_quota="pod-count-limit",group="",kind="Pod",name="nginx-deployment-77bc6bd484-4qm4h",target_namespace="wind-test"} 1
capsule_custom_quota_resource_item_usage{custom_quota="pod-count-limit",group="",kind="Pod",name="nginx-deployment-77bc6bd484-bsnfz",target_namespace="wind-test"} 1
capsule_custom_quota_resource_item_usage{custom_quota="pod-count-limit",group="",kind="Pod",name="nginx-deployment-77bc6bd484-f8qcv",target_namespace="wind-test"} 1