OSC Ceph S3 Extension
Introduction
The OSC Ceph S3 Extension provides S3-compatible object storage for Gardener Shoot clusters backed by Ceph RadosGW. It enables users to create S3 users and buckets directly from their Shoot cluster using Kubernetes Custom Resources.
The extension manages the full lifecycle of storage resources: provisioning, credential management, quota enforcement, and deletion with configurable retention policies.
Enabling the Extension
To enable the Ceph S3 extension for a Shoot cluster, add the extension service ceph-s3 to the Shoot manifest:
Check if the extension is active:
Disabling the Extension
To disable a globally enabled extension, set disabled: true:
Disabling the extension does not delete existing S3 resources. Manage resource lifecycle explicitly before disabling.
Before you start
If you also run the legacy osc-s3-bucket-service MinIO extension, its S3User and S3Bucket Kinds share the s3user / s3bucket names with this extension (under the different API group storage.osc.extensions.gardener.cloud instead of ceph-s3's s3.osc.t-systems.com). A bare kubectl get s3user / kubectl get s3bucket is therefore ambiguous and may return the wrong extension's resources.
To avoid the collision, either use the fully-qualified name, e.g. kubectl get s3buckets.s3.osc.t-systems.com, or use the ceph-s3 short names s3u (for S3User) and s3b (for S3Bucket), which are unique to this extension and do not collide with the MinIO extension:
The s3 and oscs3 short names, by contrast, belong to the legacy MinIO extension's S3Bucket — use those only when you specifically want that extension's resources.
Quick Start
Unlike the MinIO S3 extension where a bucket carries its own credentials, this extension separates identity from storage. You must first create an S3User (which holds the credentials) and then create an S3Bucket bound to that user. The user's credentials grant access to all buckets it owns. This separation allows defining additional users with different access levels to one S3 Bucket. These users are listed in .spec.additionalUsers.
Step 1: Create an S3 User
Create a minimal S3User resource:
Apply the manifest:
Step 2: Verify the User is Ready
Step 3: Create an S3 Bucket
Create a bucket owned by the user:
Apply the manifest:
Step 4: Verify the Bucket is Ready
Step 5: Access Credentials
The S3 credentials are stored in a Kubernetes Secret referenced by the S3User status:
Retrieve the credentials:
The secret contains the following fields:
accessKeyID-
The S3 access key identifier for authentication
secretAccessKey-
The S3 secret access key for authentication
endpoint-
The S3-compatible endpoint URL for API requests
ca.crt-
The CA certificate used to sign the S3 endpoint's TLS certificate. Required when connecting with TLS verification enabled. Present only when the storage backend uses a non-public CA.
Credentials are automatically provisioned when the S3User reaches the Ready phase. The secret is created in the same namespace as the S3User resource. You can verify the namespace in the .status.secretNamespace property.
Every S3User and S3Bucket is granted a quota that caps how much it can consume, drawn from your tenant's overall quota. See the Tenant Quotas section for how quotas are assigned, inherited, and enforced.
Accessing S3 Storage
Use the credentials from the secret with any S3-compatible client. The endpoint uses S3v4 signature and path-style addressing.
Due to the zero-trust security model, listing buckets (aws s3 ls, mc ls s3/) returns no results even when buckets exist. Access is scoped per bucket — you must address the buckets by their exact provisioned name.
Retrieve the actual bucket name from the S3Bucket status:
Use that value as the bucket name in all S3 client operations.
The Ceph S3 endpoint is only reachable from inside the Shoot cluster. Run any S3 client from a pod or Job on the Shoot (for example a temporary kubectl run container), not from your local machine, unless you have arranged external network access to the endpoint.
Using the AWS CLI
Using the MinIO CLI (mc)
Deleting Resources
To delete a bucket:
To delete a user:
The default deletion policy is Retain. Resources are not immediately removed after kubectl delete — they enter a retention phase. See the Deletion and Retention section for details on how to control this behavior.
S3 Users
The S3User resource represents an S3 storage identity in your Shoot cluster. Each S3User maps to a dedicated user in the Ceph RadosGW storage backend with its own access credentials, quotas, and lifecycle management.
Creating an S3 User
A minimal S3User requires only a display name. The storage tier is optional — if omitted, the controller applies an environment-derived default (see Storage Tiers):
Full example with all available fields:
Verify the user is ready:
Storage Tiers
The storage tier determines the underlying hardware. The tier is immutable after creation.
tier is optional. If omitted, the controller applies a default: if the region allows exactly one tier, that tier is
used; otherwise the operator-configured default tier is used. Check status or spec.tier
after creation to see which tier was applied.
Currently supported tiers:
The tier field cannot be changed after the S3User is created. To switch tiers, create a new S3User with the desired tier and migrate your data.
All S3Buckets owned by or shared with this user must use the same tier. Cross-tier access is not supported because different tiers may use different physical storage clusters.
Bucket Quotas
You can limit how many buckets a user is allowed to own using the quotas.buckets configuration:
- hard: Maximum bucket count. Enforced by the storage backend — creating a bucket beyond this limit fails.
- soft: Optional warning threshold. When set, a Kubernetes warning event is emitted, and a status condition is set once usage reaches it (see Quota Warning Events). It does not block operations.
When set, soft must be strictly less than hard; otherwise the resource is rejected on apply.
If quotas is omitted, the user inherits the full tenant quota.
Checking the Granted Quota
When a bucket quota is set, the value actually approved by the storage backend appears in
status.grantedQuota once the reservation succeeds:
The granted value can differ from the requested spec.quotas — for example when you increase
the limit but the additional buckets no longer fit into the remaining tenant quota. In that case
spec.quotas shows the requested value while status.grantedQuota keeps the previously approved
one, so comparing the two tells you whether an increase was actually granted. When no quota is
set on the user, status.grantedQuota is absent.
Suspending a User
Setting suspended: true immediately blocks all S3 API requests for this user at the storage backend level. Credentials
remain valid and are not changed — access is simply denied.
To restore access:
Use cases for suspension:
- Incident response (immediately revoke access without deleting data)
- Maintenance windows
- Billing holds
- Compliance freezes
The user phase changes to Suspended while suspended and returns to Ready once unsuspended.
Credentials
When an S3User reaches the Ready phase, credentials are automatically delivered to a Kubernetes Secret in the same namespace.
Retrieve the secret name and namespace from the status:
Both status.secretName and status.secretNamespace are set together when credentials are available. Use them to locate the secret precisely, especially when the S3User is in a different namespace than expected.
The secret contains the following keys:
Retrieve credentials:
The credential secret is protected by a finalizer (s3.osc.t-systems.com/credential-secret), which prevents accidental deletion while the S3User is active. The finalizer is removed automatically when the S3User itself is deleted, after which the secret is cleaned up.
If the secret is forcefully deleted (e.g. by bypassing the finalizer with kubectl delete secret <name> --grace-period=0), the extension detects the missing secret and automatically re-triggers credential issuance. No manual action is required — the secret is recreated within the next reconcile cycle.
Endpoint and Certificate Updates
The endpoint and ca.crt values are not fixed for the lifetime of the S3User. When the platform operator changes
the storage backend's endpoint or renews its TLS certificate, the updated values are written into your existing
credential secret automatically. The secret keeps the same name and the keypair is untouched — this is not a rotation,
and no action is required from you.
Read the credential secret at use time, not only at pod start. A workload that copies ca.crt or endpoint
into memory once, or bakes it into an image, keeps using the old value and will fail TLS verification after a
certificate renewal. Mount the secret as a volume (kubelet refreshes mounted secrets in place) rather than
injecting it through env/envFrom, which is only evaluated when the container starts.
The configHash key changes whenever endpoint or ca.crt changes, so a workload that needs to react to an update
can watch that single value instead of comparing certificates:
If the storage backend stops using TLS, or its CA is removed from the configuration, the ca.crt key is deleted
from the secret, so a certificate that is no longer in use cannot be trusted by mistake.
You can observe an endpoint or certificate update on the CredentialsConfigChanged condition, which reports
ConfigChangeInProgress while the new configuration is being delivered to your secret and ConfigChangeCompleted once
it has landed:
Rotating Credentials
You can rotate an S3User's credentials in place by annotating it. The credential secret keeps the same name; only the
accessKeyID and secretAccessKey values change. The endpoint and CA certificate are unaffected.
Request a rotation:
Rotation is zero-downtime: the old key stays valid until the new credentials are confirmed delivered to the credential secret (or a configured grace period elapses), so a running workload never loses access during the rotation. Once the new credentials are in the secret, restart or reload any workload that cached the old values (if you use secretKeyRef directly):
Track progress with the CredentialsRotation condition and the status.lastRotationTime field:
During RotationSettling the secret already holds the new credentials; the condition message names what the
rotation is still waiting for before revoking the previous key (the settle window, or your confirmation under
the manual policy).
When the rotation completes, the s3.osc.t-systems.com/rotation-requested annotation is removed automatically,
s3.osc.t-systems.com/last-rotation-at records the completion time, and status.lastRotationTime is updated. To rotate
again, re-add the annotation.
A rotation request is processed once. If you re-add the annotation while a rotation is still in progress, it is ignored
until the current rotation finishes — the CredentialsRotation condition does not report RotationCompleted until
then. Only one rotation runs at a time per user.
Rotation policy
The s3.osc.t-systems.com/rotation-policy annotation selects how the previous key is revoked once the new credentials are delivered.
Set it alongside the rotation request:
Always make sure the rotation-policy annotation is set before requesting a rotation. If not set, it defaults to auto.
auto(default)-
The previous key is revoked after a settle window elapses following delivery of the new credentials. The
CredentialsRotationcondition message names the settle period. manual-
The previous key stays valid until you confirm that your workloads have adopted the new key. Confirm by passing the new access key ID:
The confirmation is scoped to the access key ID, so a confirmation recorded for an earlier rotation can never silently confirm a later one.
If the confirmation names a key that is not the newly published one, the rotation does not complete: the
CredentialsRotation condition reports RotationConfirmMismatch and its message names the value you applied. The
expected key is deliberately not included — the confirmation proves the rotated credentials actually arrived in your
secret. Read the accessKeyID from your credential secret, verify your workloads use it, and re-apply the annotation
with that value.
forced-
The previous key is revoked as soon as delivery of the new credentials is confirmed.
The rotation-policy and rotation-confirm annotations only take effect together with a rotation request: set alone,
they stay on the S3User and are not acted on until you also set s3.osc.t-systems.com/rotation-requested=true. When
the rotation completes, both annotations are removed automatically along with the request — each rotation chooses its
policy explicitly, so a manual or forced choice from an earlier rotation is never silently reused by the next one.
If a rotation makes no visible progress, check the CredentialsRotation condition: it reports RotationStalled when
new credentials are waiting on a delivery or adoption confirmation, and RotationRejected when the request named an
unknown rotation policy.
Credential Lifecycle Events
In addition to the conditions above, the S3User records a Kubernetes Event the moment its
CredentialsRotation/CredentialsConfigChanged condition transitions, giving you a timestamped history in
kubectl describe. Each event fires exactly when the corresponding condition reaches that state:
Under the auto and manual policies the two rotation events are separated by the settle window (the default is
15 minutes for auto): CredentialsRotated fires when the new key lands in the secret, and CredentialsRevoked fires
later when the old key is revoked. The forced policy revokes immediately without a settle window, so there is no
RotationSettling state — only CredentialsRevoked is recorded.
Deleting a User
With the default Retain deletion policy, the user enters the Retention phase instead of being immediately removed. See the Deletion and Retention section.
To configure immediate deletion without retention:
To schedule automatic purge after a date:
When set, the value is mirrored to status.retentionExpiresAt so you can always read the active expiry from the resource status:
If retentionExpiresAt is omitted with Retain policy, the resource enters PendingPurge state with a default 7-day
grace period from the deletion date. The computed expiry date is shown in status.message. You can also apply
the force-delete annotation to purge immediately without waiting.
S3 Buckets
The S3Bucket resource represents an S3 storage bucket in your Shoot cluster. Each bucket is owned by an S3User and can optionally grant access to additional users with configurable permissions.
Creating a Bucket
A minimal bucket requires only a reference to its owner:
Full example with all available fields:
Requirements:
- The referenced owner (
userRef) must exist and be inReadyphase - If you use
additionalUsers, all users (owner and additional) must share the same storage tier
Verify the bucket is ready:
Sharing Access with Additional Users
You can grant other S3Users access to a bucket with configurable permission levels:
Permission Levels
Example: Shared Bucket
Validation Rules
- The bucket owner cannot be listed as an additional user (owner already has full access)
- All additional users must be in the same tier as the bucket owner
- All referenced users must exist
- User names cannot be empty
Adding and Removing Users
Add an additional user:
Remove an additional user (by index):
Cross-Namespace Ownership
Buckets can reference an S3User in a different namespace. This enables multi-team setups where a central user owns storage used across namespaces:
If namespace is omitted in userRef or additionalUsers[].userRef, it defaults to the bucket's own namespace.
Storage and Object Quotas
You can limit how much storage a bucket consumes and how many objects it can hold:
- hard: Maximum limit. Enforced by the storage backend — operations fail when the limit is reached.
- soft: Optional warning threshold. When set, a Kubernetes warning event is emitted, and a status condition is set once usage reaches it (see Quota Warning Events). It does not block operations.
When set, soft must be strictly less than hard; otherwise the resource is rejected on apply.
Storage Quota
Uses Kubernetes resource quantity format (e.g., "10Gi", "500Mi", "1Ti"):
Object Count Quota
Limits the total number of objects in the bucket:
Full Quota Example
If quotas is omitted, the bucket inherits the full tenant quota.
Checking the Granted Quota
When storage or object quotas are set, the values actually approved by the storage backend appear
in status.grantedQuota once the reservation succeeds:
The granted values can differ from the requested spec.quotas — for example when you increase a
limit but the additional capacity no longer fits into the remaining tenant quota. In that case
spec.quotas shows the requested values while status.grantedQuota keeps the previously approved
ones, so comparing the two tells you whether an increase was actually granted. When no quota is
set on the bucket, status.grantedQuota is absent.
Bucket Name
The actual bucket name in the storage backend differs from the Kubernetes resource name. This is by design for global uniqueness and the zero-trust security model.
Retrieve the provisioned bucket name:
Due to the zero-trust security model, listing buckets (aws s3 ls, mc ls s3/) returns no results. You must address buckets by their exact provisioned name from status.bucketName in all S3 client operations.
Deleting a Bucket
With the default Retain deletion policy, the bucket enters the Retention phase instead of being immediately removed. See the Deletion and Retention section.
To configure immediate deletion (bucket and all its data are permanently removed):
To schedule automatic purge after a date:
When set, the value is mirrored to status.retentionExpiresAt so you can always read the active expiry from the resource status:
If retentionExpiresAt is omitted with Retain policy, the resource enters PendingPurge state with a default 30-days
grace period from the deletion date. The computed expiry date is shown in status.message. You can also apply the
force-delete annotation to purge immediately without waiting.
Bucket Versioning
You can manage versioning for your S3 bucket using MinIO Client (mc) via a temporary pod.
Checking Versioning Status via Kubernetes
The current versioning state of your bucket is automatically reflected in the S3Bucket resource status. You can check it without connecting directly to the S3 endpoint.
The output looks like this:
Managing Versioning
Enable versioning:
Once the controller reconciles the S3Bucket, the status.versioningEnabled field reflects the actual versioning state:
The status.versioningEnabled field is propagated through the controllers' sync chain, so there may be a delay before it reflects changes applied via the S3 API.
Suspend versioning:
Check the current versioning status from API:
Bucket Encryption
Currently, S3Bucket supports only SSE-C (Server-Side Encryption with Customer-Provided Keys), which enables object encryption using keys managed by the user.
SSE-C Customer-Provided Keys
This mode is implemented according to the Amazon SSE-C specification.
Encryption key management is your responsibility. Losing a key means losing access to the encrypted data.
Considerations Before Using SSE-C
- Downloading SSE-C encrypted data requires the same encryption key that was used during upload.
- A reliable mapping between objects and encryption keys is required. If a key is lost, access to the corresponding object is lost.
- Because key management is client-side, safeguards such as key rotation are also client-side responsibilities.
- HTTPS is required for all requests.
- In versioning-enabled buckets, each object version can use a different encryption key, so key-to-object-version mapping must be tracked.
Prerequisites
mc(MinIO Client) configured with your bucket alias or theawsCLIopensslavailable on your system
Usage
-
Generate an encryption key:
-
Upload a file using a key:
-
Download a file:
You must supply the same key used during upload:
Attempting to download without the key returns a
400 Bad Requesterror: -
Verify object metadata:
The output confirms the encryption mode:
Store keys in a secrets manager (e.g., Vault, AWS Secrets Manager) or an encrypted keystore. Never commit them to version control.
Per-Object Keys
SSE-C supports a different key per object (and per version):
Key-to-object mapping is your responsibility. In SSE-C mode, there is no server-side record of which key was used for which object — if the mapping is lost, the encrypted data cannot be recovered.
Cross-Versioning Example
When bucket versioning is enabled, each new upload to the same object path creates a new object version. With SSE-C, each version can be encrypted with a different key.
-
Ensure bucket versioning is enabled:
-
Create version-specific keys:
-
Upload version 1 of the same object path:
-
Upload version 2 of the same object path using a different key:
-
List object versions and collect the version IDs:
Download each version with its matching key:
Verify:
If an incorrect key is used, the request returns HTTP 400 Bad Request:
Expected behavior:
- A version can only be decrypted with the key used for that version.
- Using the wrong key (or no key) for a specific version fails with a request error.
- Key-to-version mapping must be preserved, not only key-to-object mapping.
Key Rotation for SSE-C
SSE-C does not support in-place key rotation. To change the encryption key for an object, download it with the current key and re-upload it with the new key. When versioning is enabled, each version is independently encrypted — old versions remain accessible only with their original key, so either retain the old keys or re-upload the versions you need.
Key rotation using the mc CLI:
-
Set variables
-
Generate keys
-
Create and upload test file with OLD_KEY
-
Verify OLD_KEY works
-
Download to a local temp file using OLD_KEY
Ceph RGW does not support SSE-C on
CopyObject(501 NotImplemented), so key rotation requires a local download first. -
Re-upload from local temp file with NEW_KEY
-
Remove local temp file
-
Verify NEW_KEY works
-
Verify OLD_KEY is rejected
Using the REST API and AWS SDKs
The S3Bucket endpoint is compatible with the AWS SSE-C request model, so SSE-C can be implemented through direct REST API calls or AWS SDKs, with key management handled directly in your application.
Required SSE-C Request Headers
For SSE-C encryption/decryption requests, include the following headers:
x-amz-server-side-encryption-customer-algorithm(must beAES256)x-amz-server-side-encryption-customer-key(base64-encoded 256-bit key)x-amz-server-side-encryption-customer-key-MD5(base64-encoded MD5 of the key)
For copy operations where the source object is SSE-C encrypted, include source decryption headers as well:
x-amz-copy-source-server-side-encryption-customer-algorithmx-amz-copy-source-server-side-encryption-customer-keyx-amz-copy-source-server-side-encryption-customer-key-MD5
SDK and Multipart Notes
- SDKs can attach SSE-C information to PUT, GET, HEAD, and COPY requests.
- For multipart uploads, encryption settings must be provided consistently across multipart requests.
- The same key used for upload must be provided for subsequent download/metadata operations.
- If the key is incorrect or missing, the request fails (
400 Bad Request). - TLS required: non-HTTPS requests will be rejected with an
InvalidRequesterror.
Go SDK Example
The following snippets demonstrate SSE-C key management using the AWS SDK for Go v2. These are usage examples only — production implementations should add proper error handling, key persistence, and secrets management.
Key generation and encoding
Every SSE-C request requires three values derived from the raw key: the base64-encoded key, its base64-encoded MD5 checksum, and the algorithm name AES256.
Upload (PutObject)
Download (GetObject)
The same three SSE-C fields are required on every read request:
Head (metadata only)
Key rotation for SSE-C
The workaround is to download the object with the current key and re-upload it with the new key:
-
Download with the source key:
-
Re-upload with a new key:
S3 client setup (Ceph-specific options)
Other SDK References
For Java, .NET, and other SDKs, see the official AWS documentation:
- Specifying SSE-C (AWS User Guide)
- Using AWS SDKs for SSE-C operations
- Using AWS SDKs for SSE-C multipart uploads
Deletion and Retention
The Ceph S3 extension provides configurable deletion policies and retention mechanisms to protect against accidental data loss. By default, all resources use the Retain policy, which blocks deletion until a retention expiry date is reached (either user-specified or grace period default). You can customize retention by setting spec.retentionExpiresAt, cancel a pending deletion with the s3.osc.t-systems.com/recovery-requested=true annotation, or bypass retention entirely with the s3.osc.t-systems.com/force-delete=true annotation.
Deletion Policies
Retain (Default)
The Retain policy blocks deletion until a retention expiry date is reached. The retention period ends when one of the following conditions is met:
- The
retentionExpiresAttimestamp is reached (user-specified or default period) - The
s3.osc.t-systems.com/force-delete=trueannotation is applied
When a Retain-policy resource is deleted, it enters the Retention phase. During retention:
-
The resource remains visible (
kubectl get s3users/kubectl get s3buckets) -
The resource has a
DeletionTimestampbut is not yet removed -
Data remains accessible until the retention period expires
-
The resource cannot be recreated with the same name until purged
-
Retention always has an expiry date (either user-specified or backend default)
-
The deletion can be canceled with the
recovery-requestedannotation (see Recovery) -
S3User: The user is suspended — S3 access is blocked, credentials stop working
-
S3Bucket: Data remains accessible via other valid credentials until the resource is purged
Delete
The Delete policy removes the resource immediately with no grace period.
- S3User: User identity and credentials are permanently removed
- S3Bucket: Bucket and all its data are permanently removed, regardless of content
Resources with Delete policy cannot be recovered after deletion. Use with caution in production environments.
Retention State Machine
When a resource with Retain policy is deleted, it transitions through the following states:
Retention States
None-
Normal operation. Resource is not in any retention lifecycle.
PendingPurge-
Deletion was requested. The resource is blocked from permanent removal until the retention expiry date is reached (either user-specified via
spec.retentionExpiresAtor storage backend default synced tostatus.retentionExpiresAt). Therecovery-requestedannotation can cancel the deletion, and theforce-deleteannotation can bypass retention for immediate purge. Purging-
The retention expiry date has been reached or the
force-deleteannotation was applied. The resource data is being removed from storage backend and the Kubernetes object will be deleted shortly.
Recovery: Canceling a Pending Deletion
While a resource is still inside its retention window (status.retentionState: PendingPurge), cancel the deletion by applying the recovery-requested annotation:
The resource converges back to the Ready phase within a few seconds, with the same name, spec, and underlying storage data:
- S3User: The suspension is lifted — S3 access works again with the same credentials; the credential secret is re-published
- S3Bucket: The bucket and all its contents remain exactly as they were before deletion
Only the Kubernetes UID of the resource changes, because a deletion in progress cannot be undone in place — the resource is recreated with an identical spec.
Requirements:
- The resource must be in
status.retentionState: PendingPurge(inside the retention window) - The annotation value must be exactly
"true"
Removing metadata.deletionTimestamp directly (for example via kubectl patch) does not work and is not a supported recovery method — the Kubernetes API server does not permit unsetting a deletion timestamp once recorded. The recovery-requested annotation is the only supported mechanism.
Recovery only works while the resource is still inside its retention window. Once retention expires or force-delete triggers the purge (status.retentionState: Purging and beyond), the underlying storage data is gone and recovery is no longer possible. Recreate the resource and restore data from backups instead.
Force Delete
The force-delete annotation bypasses all retention policies and triggers immediate purge.
Force-delete is irreversible. The resource and all associated data are permanently removed, including non-empty buckets. Use only when you are certain the data is no longer needed.
Requirements:
- The resource must already have a
DeletionTimestampset (i.e.,kubectl deletewas already issued) - The annotation value must be exactly
"true"
Workflow:
- Delete the resource (enters retention):
kubectl delete s3user my-user - Verify the resource is in retention:
kubectl get s3user my-user -o jsonpath='{.status.retentionState}' - Apply force-delete to bypass retention:
kubectl annotate s3user my-user s3.osc.t-systems.com/force-delete=true
Events
The system emits Kubernetes events during deletion lifecycle transitions:
Each event is emitted only once, on the transition into that state — not on every reconcile loop.
View events:
Practical Examples
Standard Deletion with Retain Policy
Recovering an Accidentally Deleted User
Immediate Deletion with Delete Policy
Timed Retention with User-Specified Expiry
Bucket Deletion
S3User Deletion with Dependent Buckets
Before deleting an S3User, ensure no S3Buckets reference it as owner (.spec.userRef). If buckets still reference the user:
- The deletion proceeds but dependent buckets will enter an
Errorphase because the referenced user no longer exists - The bucket itself remains functional (data accessible via other valid credentials)
- Remove or reassign buckets before deleting their owner to avoid the error state
Additional user references (.spec.additionalUsers) do not block user deletion. If a referenced additional user is deleted, the bucket moves to Error phase. Remove the non-existent user from .spec.additionalUsers to clear the error. The bucket remains functional throughout.
Shoot Cluster Deletion
Deleting the Shoot cluster (the Shoot CR) triggers deletion of all S3 resources defined in it. This respects the configured deletionPolicy of each resource, but if the Shoot is forcefully removed, data may be lost.
To preserve data before Shoot deletion:
- Migrate data to another Shoot's S3 bucket using standard S3 tools (
aws s3 sync,mc mirror) - Ensure credentials for both source and destination buckets are available
- Delete the Shoot only after confirming data migration is complete
Migrate Data from MinIO to Ceph S3
If your application is still using the legacy MinIO-backed S3 extension, you can migrate its object data to a new Ceph S3 bucket using standard S3 tooling such as the MinIO Client (mc) or the AWS CLI.
Prerequisites
- The
ceph-s3extension enabled on the Shoot cluster — see Enabling the Extension - A new S3Bucket created and in
Readyphase
Migrate with the MinIO Client (mc)
Configure aliases for both endpoints and mirror the bucket. mc mirror copies only objects that are missing or changed in the destination, so it is safe to re-run.
Migrate with the AWS CLI
aws s3 sync performs the same incremental copy between two endpoints.
If both endpoints are reachable from the same location, you can also sync directly without the intermediate directory.
Update Your Application
Once the data is copied, point your workload at the new S3Bucket.
The old MinIO bucket is not deleted automatically — retain it as a fallback until your application is confirmed healthy on the Ceph endpoint. The old MinIO bucket and its S3User/S3Bucket resources belong to the legacy osc-s3-bucket-service extension, not to ceph-s3 — retire them following that extension's own deletion procedure.
Tenant Quotas
Tenant quota is the global capacity cap for a tenant. It defines the maximum resources your tenant can consume and cannot exceed. When you create your first resource, your tenant is granted a default quota.
The tenant quota is a hard cap that you cannot bypass — if you need more capacity, you must request a quota increase from support. The exact values granted to your tenant are not fixed here — read them from status.quotaGranted on your S3TenantQuota resource (see Viewing Tenant-Wide Quota).
What Tenant Quota Applies To
Tenant quota is applied to:
- Storage (always enforced)
- Buckets (optional, depending on configured limits)
- Objects (optional, depending on configured limits)
- Users (optional, depending on configured limits)
For details on which dimensions are actively enforced for your tenant, see Quota Enforcement.
Shared Quota Across Shoots
All Shoot clusters of a tenant on the same Seed consume from the same quota pool. Usage from any Shoot reduces what remains available for all others.
Quota Inheritance
If spec.quotas is omitted, the resource inherits the full currently available tenant quota. If spec.quotas is set, the resource is constrained to those limits, which should be lower than or equal to the available tenant quota.
Hard Limit Enforcement
Tenant quota is a hard limit. When usage reaches the limit, write operations are blocked.
Typical effects when the limit is reached:
- Uploads fail
- Creating new S3Users or S3Buckets fails
- Object creation fails
- Increasing per-resource quotas beyond the tenant cap fails
This applies to the tenant cap and to any stricter per-resource hard quota.
If you reach your tenant quota:
- Delete unnecessary objects, buckets, or users to free capacity
- If more capacity is required, contact support to request a tenant quota increase
Viewing Usage Per Resource
Users can see usage per resource in status.usage.
Check S3User usage:
Check S3Bucket usage:
Example output:
Viewing Tenant-Wide Quota
The S3TenantQuota resource provides a read-only, tenant-visible snapshot of your overall quota state. It has no spec — all information is in .status.
There is one S3TenantQuota per tier. If your tenant uses multiple tiers, you will see resource per each tier: tier3-quota, tierXY-quota, ...
The resource is deployed into kube-system by default. Depending on your cluster operator's configuration, it may also appear in the default namespace. Use -A if you are unsure which namespace is used:
You can also use the short names s3tq or s3tqs:
Example output:
To inspect the full status for a specific tier:
Or as YAML:
Example status block:
Status Fields
The status reflects the state at the last sync. Updates are propagated approximately every 5 minutes.
Quota Enforcement
The quotasEnforced field shows which quota dimensions are actively enforced for your tenant. An enforced dimension
blocks operations when its limit is reached — for example, uploads fail when the storage limit is hit. A dimension that
is not enforced is tracked in the status but does not block operations when exceeded.
Enforcement is configured by the cluster operator and may differ between tenants. To check which dimensions apply to your tenant:
Example output:
In this example, storage, users, and buckets are enforced. The objects dimension is tracked but not enforced — exceeding an object count limit does not block operations.
When some of the quota dimensions are not enforced (status.quotasEnforced[X]: false), in status.grantedQuota you will see the 0 value for the
dimensions that are not enforced, meaning unlimited value for this dimension.
Quota Relationship Diagram
The diagram below shows a practical example with a 1Ti tenant cap split across two Shoots. One bucket has an explicit storage quota; the other inherits the full remaining tenant capacity.
- Blue border: explicit quota set — resource is constrained to its hard limit
- Red dashed border: no quota set — resource may consume all remaining tenant capacity
In this example:
- Tenant Quota Setting:
1Tistorage,10users,20buckets - Current Usage:
720Gistorage (120Gifromapp-bucket+600Gifromarchive-bucket),2users, and2buckets - Remaining Available: Approximately
304Gistorage,8users, and18buckets
Because app-bucket has an explicit quota set, it is safely bounded at 200Gi. On the other hand, archive-bucket has no explicit limit and could consume the entire remaining ~304Gi of storage capacity, leaving nothing for other apps or users in either Shoot.
An explicitly set quota guarantees an upper limit, but does not guarantee reserved capacity.
In the example above, app-bucket is limited to 200Gi and is currently using 120Gi. If the unconstrained archive-bucket consumes the entire ~304Gi of remaining tenant capacity, uploads to app-bucket will fail, even though it has not exhausted its own 200Gi quota.
The tenant-wide quota is an absolute ceiling. Once it is full, all users and buckets in the tenant share the outage, regardless of their individual limits. This is why you should always set explicit quotas on all resources to prevent a single unchecked workload from starving the rest of the tenant.
Practical Examples
1. Explicit Per-Resource Quotas (Recommended)
2. Inherited Quota (No Explicit Limits)
3. Limiting Bucket Count Per User
app-user can own at most 5 buckets. Attempting to create a sixth bucket fails at the storage backend.
Quota Warning Events
The extension warns you as usage approaches a quota limit, so you can react before write operations start failing. Two mechanisms work together:
- Kubernetes warning events fire exactly once each time a threshold is crossed. Events are informational only — they never block operations.
- Status conditions persist the current over-threshold state on the resource. Events expire, conditions do not — to check whether a threshold is currently crossed, read the conditions, not the event log.
View both with kubectl describe (or kubectl get events) on the resource:
Per-Resource Events (S3User / S3Bucket)
For every quota dimension, up to two warnings can fire, each with its own event reason and status condition:
Every reason is dimension-specific, so a dimension crossing never overwrites or hides another:
The event reason is always identical to the status condition it accompanies.
- The percentage threshold is evaluated against
status.grantedQuota— the cap actually approved by the storage backend — not against the requestedspec.quotas.[dimension].hard. A dimension whose quota has been requested but not yet granted produces no threshold warning. - If you set only
hard, you get that dimension's*QuotaThresholdExceededwarning at the application threshold percentage. - If you also set
soft, you additionally get its*QuotaSoftLimitExceededwarning when usage reaches yoursoftvalue. Becausesoftmust be less thanhard, the two warnings mark two distinct points on the way to the hard limit. - If the storage backend grants less than you requested, the granted value may end up below your
softsetting; in that case the threshold warning fires first and the soft warning may never fire before the hard cap.
Dimensions covered:
- bucket count (
S3Userquotas.buckets), - storage and object count (
S3Bucketquotas.storage/quotas.objects).
Example events on an S3Bucket whose storage usage has passed both marks:
When several dimensions cross at the same time, each one gets its own entry:
The message never includes the live usage value — read that from .status.usage.
Status Conditions
Each warning has a matching condition on the resource, using the same name as the event reason
listed in the table above. The condition is True while usage is at or above that limit.
The event fires exactly when the condition transitions from False to True. When usage drops
back below the limit, the condition returns to False silently — no recovery event is emitted.
A later re-crossing fires the same event again; because the message is identical, Kubernetes
aggregates it into the existing event entry and increments its count instead of creating a new
one.
Tenant Events (S3TenantQuota)
A warning event is emitted on the S3TenantQuota for each enforced dimension whose
quotaConsumed reaches the application threshold percentage of quotaGranted (storage is always
evaluated; other dimensions only when enforced). Tenant reasons carry a
Tenant prefix so they read distinctly from the per-resource ones:
TenantStorageQuotaThresholdExceeded, TenantUsersQuotaThresholdExceeded,
TenantBucketsQuotaThresholdExceeded and TenantObjectsQuotaThresholdExceeded.
Each dimension gets its own event and its own *QuotaThresholdExceeded condition (including
UsersQuotaThresholdExceeded), so a dimension crossing later adds its own warning without
disturbing the others:
Application Threshold Percentage
The threshold percentage (default 80%) is configured by your cluster operator. It applies to
every *QuotaThresholdExceeded and Tenant*QuotaThresholdExceeded warning. Soft limits are
unaffected — those fire at the exact value you set.
Validation
When soft is set, it must be strictly less than hard for the same dimension — otherwise both
warnings would fire at the same point. Invalid combinations are rejected when you apply the
resource.
Troubleshooting
Diagnostic Commands
Quick Status Check
Common Issues
Problem: Error Phase with a "contact support" Message
Symptoms: S3User or S3Bucket shows phase Error with one of these messages:
user API access is not configured for this environment - contact supporttenant-id is not defined for this environment - contact supportuser API connection is not configured correctly for this environment - contact support
Causes:
- The storage platform for this environment is not fully provisioned or misconfigured
Solution:
-
No user action is possible — these are platform-side configuration errors. Contact support and include the resource name and the exact message.
-
Once the platform is fixed, the resource recovers automatically; no recreation is needed.
Problem: Credentials Not Appearing
Symptoms: S3User is in Ready phase but secretName is empty or the secret does not exist.
Causes:
- Credentials still being transferred (check
credentialsPhase)
Solution:
-
Check the credentials phase:
-
Verify the secret exists:
Problem: Credential Secret Was Deleted
Symptoms: The credential secret referenced in status.secretName no longer exists, but the S3User is still in Ready phase.
Causes:
- The secret was forcefully deleted (e.g.,
kubectl delete secret <name>) while the S3User was still active.
Solution:
No manual action is required. The extension automatically detects the missing secret and re-triggers credential issuance. The status.credentialsPhase returns to Issued during re-issuance and transitions back to Transferred once the new secret is available. Wait for the reconcile cycle to complete (typically within seconds).
To monitor recovery:
To prevent accidental deletion, the credential secret carries a finalizer (s3.osc.t-systems.com/credential-secret). The finalizer is removed automatically when the S3User itself is deleted. Forcefully bypassing the finalizer (kubectl delete secret <name> --grace-period=0) will trigger automatic re-issuance as described above.
Problem: S3Bucket Stuck in Pending with User Not Found
Symptoms: S3Bucket shows Pending phase with message referencing user not found.
Causes:
- Referenced S3User does not exist
- S3User is in a different namespace than expected
- Typo in
.spec.userRef.name
Solution:
-
Check the status message:
-
Verify the referenced user exists:
-
If the user is in a different namespace, set
.spec.userRef.namespaceexplicitly.
Problem: S3Bucket Error with Tier Mismatch
Symptoms: S3Bucket shows Error phase with tier mismatch message for additional users.
Causes:
- An additional user has a different tier than the bucket owner
- Different tiers use different physical storage clusters, making cross-tier access impossible
Solution:
-
Check the error message:
-
Verify the owner's tier:
-
Verify additional users' tiers:
-
Ensure all users (owner and additional) share the same tier value.
Problem: Deletion Blocked (Retention)
Symptoms: kubectl delete was issued but the resource persists, showing Retention phase.
Causes:
- Default
Retaindeletion policy is active retentionExpiresAthas not been reached
Solution:
-
Check the retention state:
-
Check retention expiry:
-
To force immediate deletion (irreversible):
See the Deletion and Retention section for full details.
Problem: Resource Deleted by Accident
Symptoms: kubectl delete was issued on an S3User or S3Bucket that is still needed. The resource shows Retention phase with status.retentionState: PendingPurge; an S3User is suspended and its credentials stop working.
Causes:
- The deletion was unintentional and the resource uses the
Retaindeletion policy, so it is still inside its retention window
Solution:
-
Verify the resource is still recoverable:
Recovery is only possible while the state is
PendingPurge. If the state isPurgingor the resource is gone, the storage data has been removed — recreate the resource and restore from backups. -
Cancel the deletion:
(Use
s3bucketfor buckets.) -
Wait for the phase to return to
Ready(typically a few seconds). The resource keeps its name, spec, credentials, and data; only its Kubernetes UID changes.
See Recovery: Canceling a Pending Deletion for full details.
Problem: S3User Shows Suspended Phase
Symptoms: S3User phase is Suspended, S3 API requests fail with access denied.
Causes:
.spec.suspendedis set totrue
Solution::
-
Verify the suspension setting:
-
To restore access:
-
Wait for the phase to return to
Ready.
Problem: S3Bucket Owner Cannot Be Additional User
Symptoms: Applying an S3Bucket manifest fails with validation error about owner being listed as additional user.
Causes:
- The same user referenced in
.spec.userRefalso appears in.spec.additionalUsers
Solution:
Remove the owner from the additionalUsers list. The owner already has full access to the bucket.
Problem: Resource Creation Fails — Quota Assignment Limit Reached
Symptoms: Creating a new S3User or S3Bucket with explicit spec.quotas fails. The error references quota exceeded or insufficient remaining quota.
Causes:
- The requested quota assignment would push
quotaAssignedoverquotaGrantedfor one or more dimensions. - Example:
quotaGranted.bucketsis20,quotaAssigned.bucketsis already18, and the new S3User requestsquotas.buckets.hard: 5. The resulting assignment would be23, which exceeds the granted limit of20— the request is rejected.
Solution:
-
Check the current tenant quota state:
-
Identify which dimension is exhausted by comparing
quotaAssignedagainstquotaGranted: -
List existing S3Users and their assigned quotas to find candidates for reduction:
-
Choose one of the following:
- Reduce the quota on an existing user or bucket that has more assigned than it consumes
- Set a lower limit on the new resource so the total assignment stays within the granted cap
- Contact support to request a tenant quota increase if neither adjustment is feasible
Problem: Upload Fails — Storage Quota Consumed
Symptoms: S3 API rejects an upload with an error such as QuotaExceeded or EntityTooLarge. The S3Bucket and S3User are both in Ready phase.
Causes:
quotaConsumed.storageis close toquotaGranted.storageand the upload would push actual consumption over the granted limit.- Example:
quotaGranted.storageis1Gi,quotaConsumed.storageis900Mi, and the file being uploaded is200Mi. The resulting consumption would be1100Mi, exceeding the1Gicap — the storage backend rejects the write. - The S3Bucket itself has a
spec.quotas.storage.hardlimit set, and that per-bucket cap is reached regardless of remaining tenant capacity.
Solution:
-
Check how much storage has been consumed versus what is granted at the tenant level:
-
Confirm storage is the constraint at the tenant level:
-
If the tenant quota has remaining capacity, check whether the S3Bucket has a per-bucket storage limit that is being hit:
If the bucket's
hardlimit is too low, increase it: -
If the tenant quota is the constraint, free storage by deleting unused objects or buckets, then retry the upload.
-
If more capacity is required, contact support to request a tenant quota increase.
See the Viewing Tenant-Wide Quota section for details on reading quota status.