> ## Documentation Index > Fetch the complete documentation index at: https://ryvn.ai/docs/llms.txt > Use this file to discover all available pages before exploring further. # Infrastructure auditing > Archive Kubernetes and cloud API audit logs from an AWS, GCP, or Azure environment to storage that the account owner controls. This page explains how to keep an audit trail for a Ryvn environment. It covers two kinds of records: * **Kubernetes API activity.** Who called the cluster's API server, what they did, and whether it succeeded. * **Cloud API activity.** Changes to cloud resources and IAM in the account or project that hosts the environment. The owner of the cloud account sets this up. For a customer cloud environment, that is usually your customer's cloud or security team, and you can send them this page. Every resource on this page lives in their account. Ryvn doesn't create, read, or manage the archive, and the setup doesn't give Ryvn any new permissions. ## What you collect | Provider | Kubernetes API activity | Cloud API activity | Archive destination | | - | - | - | - | | AWS | EKS `audit` and `authenticator` logs | CloudTrail management events (read and write) | S3 bucket | | GCP | GKE audit logs (`k8s.io`) | Cloud Audit Logs for the project | Cloud Storage bucket | | Azure | AKS `kube-audit` and `guard` logs | Subscription Activity Log, optionally Entra ID logs | Storage account | Ryvn already turns on control-plane logging when it provisions a cluster, but those logs stay next to the cluster in the provider's default log store. The steps below copy them, along with the cloud API audit logs, into an archive that you control and that outlives the environment. ## Before you start Each example assumes a first-time setup for one environment, unused export resource names, and credentials with permission to create and configure the resources shown. Replace the example account, project, environment, and storage names with your own. Bucket and storage account names must be globally unique. These points apply to every provider: * **Create the archive outside the environment.** Use a bucket or storage account that Ryvn doesn't manage, ideally in a dedicated security account, project, or resource group. Deleting the Ryvn environment then leaves the archive intact. * **Set retention, encryption, and access on the archive yourself.** Pick the retention period your compliance program requires. Add object lock or immutable storage if you need tamper resistance. * **Set it up before you need the records.** These exports don't automatically backfill historical records. Configure them before the environment handles production traffic. Some providers offer separate exports of retained logs, such as the CloudWatch export described below. * **Expect some volume.** Kubernetes audit logs include reads subject to the provider's audit policy and configured exemptions. Read requests can generate substantial volume. *** ## AWS On AWS you set up two pipelines that both write to the same S3 bucket: 1. A CloudTrail trail for AWS management events. 2. A CloudWatch Logs subscription that streams the EKS control-plane log group through Amazon Data Firehose to S3. This example uses an EKS cluster and a new archive bucket in `us-east-1`, in the same AWS account, and creates a trail named `ryvn-audit`. Run the commands with credentials for that account. ```bash theme={null} export ACCOUNT_ID=123456789012 export REGION=us-east-1 # Region of the EKS cluster export CLUSTER_NAME=ryvn-eks-production # See step 1 export BUCKET=acme-audit-archive # New bucket, see step 2 export LOG_GROUP=/aws/eks/$CLUSTER_NAME/cluster ``` ### Step 1: Find the cluster and its log group Ryvn names EKS clusters `ryvn-eks-`. Long environment names are shortened, so list the clusters to get the exact name: ```bash theme={null} aws eks list-clusters --region "$REGION" ``` Ryvn enables the `api`, `audit`, `authenticator`, `controllerManager`, and `scheduler` log types. EKS writes them to the CloudWatch log group `/aws/eks//cluster`. Confirm both: ```bash theme={null} aws eks describe-cluster --name "$CLUSTER_NAME" --region "$REGION" --query 'cluster.logging' aws logs describe-log-groups --log-group-name-prefix "$LOG_GROUP" --region "$REGION" ``` ### Step 2: Prepare the S3 bucket Create the archive bucket in `us-east-1`. S3 bucket creation in this region omits `LocationConstraint`: ```bash theme={null} aws s3api create-bucket --bucket "$BUCKET" --region "$REGION" aws s3api put-public-access-block --bucket "$BUCKET" \ --public-access-block-configuration BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true ``` Add a lifecycle rule for your retention period and, if you need it, S3 Object Lock. ### Step 3: Turn on CloudTrail management events If an existing account or organization trail already records read and write management events across all regions, including global service events, you can reuse it instead. Coordinate with its owner and verify its actual trail ARN, home region, and archive destination. The commands below assume a new `ryvn-audit` trail. CloudTrail needs permission to write to the bucket. Add these two statements to the bucket policy. If the bucket already has a policy, merge them into it, because `put-bucket-policy` replaces the whole policy. ```json cloudtrail-bucket-policy.json theme={null} { "Version": "2012-10-17", "Statement": [ { "Sid": "CloudTrailAclCheck", "Effect": "Allow", "Principal": { "Service": "cloudtrail.amazonaws.com" }, "Action": "s3:GetBucketAcl", "Resource": "arn:aws:s3:::BUCKET", "Condition": { "StringEquals": { "aws:SourceArn": "arn:aws:cloudtrail:REGION:ACCOUNT_ID:trail/ryvn-audit" } } }, { "Sid": "CloudTrailWrite", "Effect": "Allow", "Principal": { "Service": "cloudtrail.amazonaws.com" }, "Action": "s3:PutObject", "Resource": "arn:aws:s3:::BUCKET/cloudtrail/AWSLogs/ACCOUNT_ID/*", "Condition": { "StringEquals": { "s3:x-amz-acl": "bucket-owner-full-control", "aws:SourceArn": "arn:aws:cloudtrail:REGION:ACCOUNT_ID:trail/ryvn-audit" } } } ] } ``` Replace `BUCKET`, `REGION`, and `ACCOUNT_ID`, then apply the policy and create the trail: ```bash theme={null} aws s3api put-bucket-policy --bucket "$BUCKET" --policy file://cloudtrail-bucket-policy.json aws cloudtrail create-trail --name ryvn-audit --region "$REGION" \ --s3-bucket-name "$BUCKET" --s3-key-prefix cloudtrail \ --is-multi-region-trail --include-global-service-events --enable-log-file-validation aws cloudtrail put-event-selectors --trail-name ryvn-audit --region "$REGION" \ --advanced-event-selectors '[{"Name":"Management events","FieldSelectors":[{"Field":"eventCategory","Equals":["Management"]}]}]' aws cloudtrail start-logging --name ryvn-audit --region "$REGION" ``` This records read and write management events in every region. It doesn't record data events such as S3 object reads, which can be expensive. Add them separately if you need them. If the bucket uses a customer-managed KMS key, pass it with `--kms-key-id` and grant CloudTrail access in the key policy. See [AWS's KMS key policy for CloudTrail](https://docs.aws.amazon.com/awscloudtrail/latest/userguide/create-kms-key-policy-for-cloudtrail.html). ### Step 4: Stream EKS logs to S3 CloudWatch Logs can't write to S3 directly. A subscription filter sends new log events to a Firehose stream, and Firehose writes them to S3. You need two IAM roles, one for each hop. Check the log group's existing subscriptions first. By default, AWS allows two subscription filters per log group. ```bash theme={null} aws logs describe-subscription-filters --log-group-name "$LOG_GROUP" --region "$REGION" ``` **Create the Firehose role.** Firehose assumes this role to write to the `eks/` prefix of the bucket. ```json firehose-trust.json theme={null} { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Service": "firehose.amazonaws.com" }, "Action": "sts:AssumeRole" } ] } ``` ```json firehose-s3.json theme={null} { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": ["s3:GetBucketLocation", "s3:ListBucket", "s3:ListBucketMultipartUploads"], "Resource": "arn:aws:s3:::BUCKET" }, { "Effect": "Allow", "Action": ["s3:AbortMultipartUpload", "s3:GetObject", "s3:PutObject"], "Resource": "arn:aws:s3:::BUCKET/eks/*" } ] } ``` ```bash theme={null} aws iam create-role --role-name ryvn-audit-firehose-to-s3 \ --assume-role-policy-document file://firehose-trust.json aws iam put-role-policy --role-name ryvn-audit-firehose-to-s3 \ --policy-name s3-archive --policy-document file://firehose-s3.json ``` **Create the Firehose stream.** Leave compression off. CloudWatch Logs already gzips each batch, and compressing again would wrap the files twice. ```bash theme={null} aws firehose create-delivery-stream --region "$REGION" \ --delivery-stream-name ryvn-audit-eks \ --delivery-stream-type DirectPut \ --extended-s3-destination-configuration '{ "RoleARN": "arn:aws:iam::'"$ACCOUNT_ID"':role/ryvn-audit-firehose-to-s3", "BucketARN": "arn:aws:s3:::'"$BUCKET"'", "Prefix": "eks/", "BufferingHints": { "IntervalInSeconds": 300, "SizeInMBs": 5 }, "CompressionFormat": "UNCOMPRESSED" }' ``` IAM changes take a few seconds to propagate. If this fails with an access error, wait and retry. The stream takes a minute or two to become `ACTIVE`: ```bash theme={null} aws firehose describe-delivery-stream --delivery-stream-name ryvn-audit-eks \ --region "$REGION" --query 'DeliveryStreamDescription.DeliveryStreamStatus' ``` **Create the CloudWatch Logs role.** CloudWatch Logs assumes this role to put records into the stream. The trust policy limits it to the EKS log group. Include both ARN forms. Creating the subscription needs the first and delivering logs needs the second. ```json logs-trust.json theme={null} { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Service": "logs.amazonaws.com" }, "Action": "sts:AssumeRole", "Condition": { "ArnLike": { "aws:SourceArn": [ "arn:aws:logs:REGION:ACCOUNT_ID:log-group:/aws/eks/CLUSTER_NAME/cluster", "arn:aws:logs:REGION:ACCOUNT_ID:log-group:/aws/eks/CLUSTER_NAME/cluster:*" ] } } } ] } ``` ```json logs-firehose.json theme={null} { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "firehose:PutRecord", "Resource": "arn:aws:firehose:REGION:ACCOUNT_ID:deliverystream/ryvn-audit-eks" } ] } ``` ```bash theme={null} aws iam create-role --role-name ryvn-audit-logs-to-firehose \ --assume-role-policy-document file://logs-trust.json aws iam put-role-policy --role-name ryvn-audit-logs-to-firehose \ --policy-name firehose-put --policy-document file://logs-firehose.json ``` **Attach the subscription filter.** An empty filter pattern forwards every event in the log group. ```bash theme={null} aws logs put-subscription-filter --region "$REGION" \ --log-group-name "$LOG_GROUP" \ --filter-name ryvn-audit-eks \ --filter-pattern "" \ --destination-arn "arn:aws:firehose:$REGION:$ACCOUNT_ID:deliverystream/ryvn-audit-eks" \ --role-arn "arn:aws:iam::$ACCOUNT_ID:role/ryvn-audit-logs-to-firehose" ``` The log group also holds API server, controller manager, and scheduler diagnostics. CloudWatch subscription filters match on event content, not stream names, so this setup forwards everything and you select audit records downstream. Audit events come from streams named `kube-apiserver-audit-*` and authenticator events from streams named `authenticator-*`. Each S3 object holds gzipped CloudWatch Logs batches that keep the `logGroup`, `logStream`, and event `id` of every record. If the bucket uses a customer-managed KMS key, add `"EncryptionConfiguration": {"KMSEncryptionConfig": {"AWSKMSKeyARN": ""}}` to the Firehose destination and allow the Firehose role to use `kms:GenerateDataKey` and `kms:Decrypt` on the key. ### Delivery timing New logs should arrive under `eks/` and `cloudtrail/AWSLogs/ACCOUNT_ID/` in the archive bucket. This example configures Firehose with a five-minute buffering interval, and CloudTrail typically delivers events within about five minutes. Initial setup and source-log ingestion can add delay. To recover EKS records from before you attached the subscription, run a one-off [CloudWatch Logs export to S3](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/S3Export.html) into a separate prefix. This only works for records still within the log group's retention. *** ## GCP On GCP, Cloud Audit Logs already records Admin Activity for every service, including GKE, by default. You turn on Data Access logs for GKE, then create a log sink that copies the audit logs to a Cloud Storage bucket. This example keeps the archive in the same project as the cluster. ```bash theme={null} export PROJECT_ID=acme-production export CLUSTER_NAME=ryvn-gke-production # See step 1 export CLUSTER_LOCATION=us-central1 # Region or zone, see step 1 export BUCKET=acme-audit-archive # New bucket, see step 3 export BUCKET_LOCATION=us-central1 # Archive bucket region ``` ### Step 1: Find the cluster Ryvn names GKE clusters `ryvn-gke-`. List the clusters to get the exact name and location: ```bash theme={null} gcloud container clusters list --project "$PROJECT_ID" ``` The environment's **Cluster OIDC Issuer URL** output in Ryvn also contains these values, in the form `https://container.googleapis.com/v1/projects/PROJECT/locations/LOCATION/clusters/CLUSTER`. Use the location exactly as shown. A zonal cluster has a zone such as `us-central1-a`, and the sink filter in step 4 won't match if you use the region instead. ### Step 2: Turn on GKE Data Access audit logs Admin Activity logs cover changes, such as creating a deployment. Kubernetes reads, such as `kubectl get secrets`, are Data Access logs, and Google turns them off by default. Turn on all three Data Access types for the Kubernetes Engine API: 1. In the Google Cloud console, open **IAM & Admin > Audit Logs** for the project. 2. Select **Kubernetes Engine API**. 3. Check **Admin Read**, **Data Read**, and **Data Write**, then click **Save**. The setting applies to every GKE cluster in the project. If your organization manages audit settings at the folder or organization level, make the change there instead. A project can add audit logging on top of its parent's settings but can't turn off what the parent requires. Review any exempted principals on the same screen, because exemptions leave gaps in the record. If you manage IAM with Terraform, use `google_project_iam_audit_config` with `service = "container.googleapis.com"`. That resource is authoritative for the service, so import any existing configuration first. ### Step 3: Prepare the Cloud Storage bucket Create a bucket in the cluster's project, outside the resources managed by Ryvn: ```bash theme={null} gcloud storage buckets create "gs://$BUCKET" --project "$PROJECT_ID" \ --location "$BUCKET_LOCATION" --uniform-bucket-level-access --public-access-prevention ``` Set a retention policy or lifecycle rules for your retention period. Use a bucket lock if you need the retention policy to be permanent. ### Step 4: Create a log sink Save this filter to `audit-filter.txt`, replacing the three cluster values. It routes: * All Admin Activity, System Event, and Policy Denied audit logs in the project. * GKE management API Data Access logs, such as calls to describe the cluster. * Kubernetes API Data Access logs for this cluster only. ```text audit-filter.txt theme={null} log_id("cloudaudit.googleapis.com/activity") OR log_id("cloudaudit.googleapis.com/system_event") OR log_id("cloudaudit.googleapis.com/policy") OR ( log_id("cloudaudit.googleapis.com/data_access") AND ( protoPayload.serviceName="container.googleapis.com" OR ( protoPayload.serviceName="k8s.io" AND resource.type="k8s_cluster" AND resource.labels.project_id="PROJECT_ID" AND resource.labels.cluster_name="CLUSTER_NAME" AND resource.labels.location="CLUSTER_LOCATION" ) ) ) ``` Create the sink, then give its writer identity routing permission on the archive project and permission to create objects in the bucket. Your credentials must allow both IAM grants: ```bash theme={null} gcloud logging sinks create ryvn-audit "storage.googleapis.com/$BUCKET" \ --project "$PROJECT_ID" --log-filter="$(cat audit-filter.txt)" WRITER=$(gcloud logging sinks describe ryvn-audit --project "$PROJECT_ID" --format='value(writerIdentity)') gcloud projects add-iam-policy-binding "$PROJECT_ID" \ --member="$WRITER" --role=roles/logging.logWriter gcloud storage buckets add-iam-policy-binding "gs://$BUCKET" \ --member="$WRITER" --role=roles/storage.objectCreator ``` For an archive in a separate security project, create the bucket there and grant `roles/logging.logWriter` on that project instead. Keep the sink in the source project and the Object Creator grant on the archive bucket. If your organization already has an aggregated sink at the folder or organization level, check whether it covers this project before adding another. Two sinks with overlapping filters write duplicate records. ### Delivery timing New matching logs should arrive in the archive bucket in folders such as `cloudaudit.googleapis.com/activity` and `cloudaudit.googleapis.com/data_access`. Cloud Storage receives logs in hourly batches, and the first files can take two to three hours to appear. *** ## Azure On Azure you add diagnostic settings that copy logs to a storage account: 1. One on the AKS cluster for Kubernetes audit logs. 2. One on the subscription for the Activity Log. 3. Optionally, one on the Entra ID tenant for sign-in and directory audit logs. This example uses the standard Ryvn AKS cluster name and creates the archive account in the same subscription and region, in a separate resource group. ```bash theme={null} export SUBSCRIPTION_ID=00000000-0000-0000-0000-000000000000 export ENVIRONMENT=production # Ryvn environment name export LOCATION=eastus # Region of the AKS cluster export STORAGE_ID=/subscriptions/$SUBSCRIPTION_ID/resourceGroups/acme-security/providers/Microsoft.Storage/storageAccounts/acmeauditarchive az account set --subscription "$SUBSCRIPTION_ID" ``` The person running these commands needs Monitoring Contributor on the cluster and subscription, plus `Microsoft.Storage/storageAccounts/listkeys/action` on the storage account. Scope the second permission to the storage account only, because it exposes the account keys. Creating the archive also requires permission to create the resource group and storage account; an authorized administrator can prepare these instead. ### Step 1: Find the cluster The standard Ryvn cluster name is `-aks`, in the resource group `ryvn-rg-`. Get its resource ID: ```bash theme={null} AKS_ID=$(az aks show --resource-group "ryvn-rg-$ENVIRONMENT" --name "${ENVIRONMENT}-aks" --query id -o tsv) ``` Ryvn already adds a diagnostic setting named `ryvn-control-plane-logs` that sends control-plane and audit logs to a Log Analytics workspace in `ryvn-rg-`. Leave it in place. That workspace keeps logs for 30 days by default and is deleted with the environment, so it isn't an archive. Azure allows five diagnostic settings per resource, so you can add your own alongside it. ### Step 2: Prepare the storage account Use a Standard general-purpose v2 storage account in the same region as the cluster. Put it in a resource group you own, not in `ryvn-rg-`, which Ryvn deletes with the environment. ```bash theme={null} az group create --name acme-security --location "$LOCATION" az storage account create --name acmeauditarchive --resource-group acme-security \ --location "$LOCATION" --sku Standard_LRS --kind StorageV2 \ --min-tls-version TLS1_2 --allow-blob-public-access false ``` If you restrict network access to the account, allow trusted Microsoft services to bypass the firewall so Azure Monitor can write to it. Set retention with a lifecycle management policy on the account, and add an immutability policy if you need tamper resistance. Diagnostic settings used to have their own retention option, but Azure has deprecated it. ### Step 3: Archive AKS audit logs ```bash theme={null} az monitor diagnostic-settings create --name ryvn-audit-archive \ --resource "$AKS_ID" \ --storage-account "$STORAGE_ID" \ --logs '[{"category":"kube-audit","enabled":true},{"category":"guard","enabled":true}]' ``` Use `kube-audit`, not `kube-audit-admin`. The admin category drops `get` and `list` requests, so you'd lose the record of who read what. The `guard` category records Entra ID authentication and Azure RBAC decisions for the cluster. ### Step 4: Archive the subscription Activity Log ```bash theme={null} az monitor diagnostic-settings subscription create --name ryvn-audit-archive \ --location "$LOCATION" \ --storage-account "$STORAGE_ID" \ --logs '[{"category":"Administrative","enabled":true},{"category":"Security","enabled":true},{"category":"ServiceHealth","enabled":true},{"category":"Alert","enabled":true},{"category":"Recommendation","enabled":true},{"category":"Policy","enabled":true},{"category":"Autoscale","enabled":true},{"category":"ResourceHealth","enabled":true}]' ``` The Activity Log records create, update, and delete operations on Azure resources, including failures and role assignments. Unlike CloudTrail on AWS, it doesn't record most read operations. ### Step 5 (optional): Archive Entra ID logs Kubernetes audit events show which identity made a request. Entra ID sign-in logs show how that identity got its token. A tenant administrator with the Security Administrator role sets this up once per tenant: 1. In the Microsoft Entra admin center, open **Monitoring & health > Diagnostic settings** and click **Add diagnostic setting**. 2. Select the categories you need. `AuditLogs` covers directory changes. `SignInLogs`, `NonInteractiveUserSignInLogs`, `ServicePrincipalSignInLogs`, and `ManagedIdentitySignInLogs` cover different kinds of callers. Some categories require an Entra ID P1 or P2 license. 3. Select **Archive to a storage account**, choose the storage account, and save. These logs cover the whole tenant, not only the Ryvn environment. ### Delivery timing New logs should arrive in the storage account under `insights-logs-kube-audit`, `insights-logs-guard`, and `insights-activity-logs`. Allow up to 90 minutes for a new diagnostic setting to begin delivery. Containers only appear after the first matching event. Optional Entra ID exports can take up to three days to begin delivery. *** ## What these logs don't cover These steps collect control-plane and cloud API audit logs. They don't collect: * Application logs from your services. See [Logs](/docs/observability/logs). * Network flow logs, DNS queries, or node operating system logs. * Data-plane access to cloud services, such as individual S3 object reads or Azure Storage blob reads. Each service has its own logging for that. * Commands typed inside a shell. A Kubernetes audit event for `kubectl exec` records the command used to start the session, but not what someone typed afterward. Audit records contain identities, resource names, and request details. Ryvn's log redaction settings don't apply to them, so restrict who can read the archive. ## Removing the export When you retire an environment, remove the export for its cluster but keep the archive: * **AWS.** Delete the subscription filter, the Firehose stream, and the two IAM roles. Keep the trail if it covers other workloads in the account. * **GCP.** Delete the sink if it only served this project, or update its filter if other clusters in the project still need it. * **Azure.** Delete the `ryvn-audit-archive` diagnostic setting on the cluster before you delete the environment. Azure doesn't always remove diagnostic settings with their resource, and a cluster recreated with the same name would pick up the old setting. Remove the subscription and Entra ID settings only if nothing else in the subscription or tenant needs them. Objects already in the archive stay there under the archive's own retention policy.