> ## Documentation Index
> Fetch the complete documentation index at: https://ryvn.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# LGTM access over Tailscale

> Configure private access to Grafana, Loki, and Mimir through your tailnet.

A dedicated LGTM installation can expose Grafana, Loki, and Mimir to people on your tailnet. Ryvn creates three HTTPS Ingresses and configures Grafana's login URL. Your tailnet administrator controls who can reach them.

Collectors continue to send logs and metrics through the environment's internal gateway. Enabling Tailscale does not change their connection URLs.

## Before you enable access

You need:

* A dedicated `observability-infra` blueprint installation with **Stack Exposure** set to **Private**. Use a blueprint and `tailscale-resources` chart release that supports the inputs below. If the inputs are missing, contact your Ryvn administrator to update the releases.
* A running Tailscale Kubernetes Operator in the same environment, with the ProxyGroup and ProxyClass CRDs and the `tailscale` IngressClass. HA ingress requires Operator 1.84.0 or later; use a current stable operator with matching proxy images, CRDs, and Kubernetes permissions.
* Access to the tailnet's DNS settings, trust credentials, and access policy, or an administrator who can update them.
* A Tailscale client connected to that tailnet on each user's device.

> [!WARNING]
>
> Switching **Stack Exposure** from Public to Private disables public endpoints and changes collector URLs to the environment's internal domain. First verify private connectivity, DNS, and a trusted HTTPS certificate on the internal gateway. When using private links, configure Private Service Publisher in the LGTM environment and Private Service Consumer in each collector environment before switching exposure.

If the operator is not installed, follow the [operator setup instructions](https://ryvn.ai/docs/guides/tailscale-database-access.md#step-2-create-tailscale-operator-credentials). LGTM uses the existing operator; it does not install another one.

## 1. Configure the tailnet

### DNS and operator credentials

Enable [MagicDNS](https://tailscale.com/kb/1081/magicdns) and [HTTPS certificates](https://tailscale.com/kb/1153/enabling-https). Copy the tailnet's DNS suffix, such as `example.ts.net`. This becomes `tailnetDomain`; it is specific to your tailnet.

The operator's OAuth or workload identity credential needs write access to **General/Services**, **Devices/Core**, and **Keys/Auth Keys**, with the appropriate operator tag. Ensure the operator owns the tags used for proxy devices. An operator that already exposes a database or the Kubernetes API may still lack Services permission. See [Tailscale's credential setup](https://tailscale.com/docs/kubernetes-operator/install-operator).

### Service advertisement and user access

Allow the proxy devices to advertise the new Tailscale Services, then grant your users access to those Services on TCP 443. These are separate permissions.

The following fragment assumes the default `tag:k8s` tag and an existing `group:observability`. It grants that group access to the three named Services. Merge it into your existing policy and replace the group, tags, and `prod-lgtm` prefix with your own values.

```json theme={null}
{
  "autoApprovers": {
    "services": {
      "tag:k8s": ["tag:k8s"]
    }
  },
  "grants": [
    {
      "src": ["group:observability"],
      "dst": [
        "svc:prod-lgtm-grafana",
        "svc:prod-lgtm-loki",
        "svc:prod-lgtm-mimir"
      ],
      "ip": ["tcp:443"]
    }
  ]
}
```

In `autoApprovers.services`, the key is a **Service tag** and the values identify the **proxy devices** allowed to advertise it. The example approves advertisements for all Services tagged `tag:k8s`. Adapt its scope to your policy.

LGTM leaves tags to the operator's configured defaults. If you reuse a ProxyGroup with different device tags, use those device tags on the right-hand side of the advertisement rule. Existing access rules for proxy devices alone do not establish access to HA Services. See [Tailscale's ingress permissions](https://tailscale.com/docs/kubernetes-operator/reference/rbac).

## 2. Choose the proxy configuration

| Configuration | Blueprint inputs | Resource ownership |
| - | - | - |
| Create a group and class | Leave both fields blank | Ryvn creates a two-replica ingress group and a class that prefers different nodes. |
| Create a group using your class | Set `tailnetProxyClass` | Ryvn creates the group; you manage the existing class. |
| Reuse your ingress group | Set `tailnetProxyGroup` | Ryvn creates only the Ingresses; you manage the group and its class. |

Use `tailnetProxyClass` if your proxies require custom node selection, tolerations, or security settings. The generated class does not inherit your operator's default class. Configure replica spreading in your class and provide at least two eligible nodes for node-failure resilience.

An existing ProxyGroup must be in the same cluster with `spec.type: ingress`. A Kubernetes API or egress group cannot be used. When `tailnetProxyGroup` is set, `tailnetProxyClass` is ignored; the group owner controls its replicas, scheduling, and lifecycle.

> [!WARNING]
>
> Switching from a blueprint-created group to an external group removes the old managed group and class. Plan a maintenance window. Do not set `tailnetProxyGroup` to the blueprint-created group's name to try to transfer ownership.

## 3. Update the LGTM installation

In the installation's **Inputs**, enable **Expose to Tailnet**, enter the tailnet domain and a unique hostname prefix, and select your proxy configuration.

For the CLI, save this as `lgtm-tailnet.patch.yaml`. Replace the domain and prefix. This example creates a group with the generated class.

```yaml theme={null}
spec:
  inputs:
    - name: exposure
      value: private
    - name: exposeToTailnet
      value: true
    - name: tailnetDomain
      value: example.ts.net
    - name: tailnetHostnamePrefix
      value: prod-lgtm
    - name: tailnetProxyGroup
      value: ""
    - name: tailnetProxyClass
      value: ""
```

To use your class, set `tailnetProxyClass` to its name. To reuse a group, set `tailnetProxyGroup` to its name instead. Then apply the patch to your existing dedicated installation:

```bash theme={null}
ryvn update blueprint-installation lgtm -e production \
  --patch-file lgtm-tailnet.patch.yaml
```

The hostname prefix must contain at most 53 lowercase letters, digits, or hyphens, starting and ending with a letter or digit. The three resulting Service names must be unused on your tailnet. Keep the installation's existing `domain` input; private collector endpoints derive from the environment's internal domain.

### Schedule LGTM on a dedicated node pool

On AWS, Azure, and GCP, set **Node Group** (`nodeGroup`) and **Pod Tolerations**
(`tolerations`) to match your environment's node pool. Pod Tolerations accepts a
YAML list. For a pool labeled `ryvn.app/node-group-name: observability` and tainted
`dedicated=observability:NoSchedule`, add these entries to the installation inputs:

```yaml theme={null}
- name: nodeGroup
  value: observability
- name: tolerations
  value: |
    - key: dedicated
      operator: Equal
      value: observability
      effect: NoSchedule
```

These inputs apply to Loki, Mimir, their caches, Grafana, the auth-proxy, the
Grafana bootstrap job, and the managed Grafana database's user helper. They do
not create the node pool. Loki's default ingester placement requires at least three
schedulable nodes in that pool. Tailscale proxies use their ProxyClass's scheduling
settings separately. Blank inputs retain the chart defaults.

If you previously edited child installations directly, first make both the LGTM
and Postgres blueprint releases with these inputs available in the installation's
release channel. Set the inputs, compare the proposed child configuration with
the running configuration, and preserve any other overrides before restoring
blueprint management of excluded children. Removing an exclusion allows the
blueprint to replace that child's configuration.

## 4. Verify access and login

After the installation updates, use its `grafanaUrl`, `lokiTailnetUrl`, and `mimirTailnetUrl` outputs. For the example above, Grafana is `https://prod-lgtm-grafana.example.ts.net`.

From an authorized device on the tailnet:

1. Open Grafana and complete sign-in with your Ryvn account. Ryvn updates Grafana's canonical URL and registers `/login/generic_oauth` on that hostname with its managed OIDC application. You do not need to register a separate application. Your browser must also reach your Ryvn identity provider.
2. Query Loki and Mimir with the installation's existing API credentials. Tailnet access does not replace their tenant authentication.
3. Confirm new logs and metrics still arrive through the private collector path.

Have your cluster administrator confirm the three Ingresses have HTTPS addresses and the ingress ProxyGroup's replicas are ready. For high availability, confirm placement on separate nodes and test access during a proxy restart.

## Troubleshooting

| Symptom | Check |
| - | - |
| Proxy pods stay Pending | The group's class has the required node selectors and tolerations, and eligible nodes have capacity. |
| No tailnet address or Service | The `tailscale` IngressClass exists, the group is an ingress group, and the operator credential has Services write permission. Inspect operator errors. |
| Service exists but is unreachable | Service advertisement is approved, the client has a grant to the `svc:` destination, and its Tailscale client is connected. |
| Hostname or certificate fails | `tailnetDomain` matches the operator's tailnet, MagicDNS and HTTPS are enabled, and the Service names are unique. |
| Grafana login redirects incorrectly | The LGTM update completed, `grafanaUrl` matches the browser URL, and the managed OIDC callback update succeeded. |
| Collectors stop after switching to Private | The internal gateway, private connectivity, DNS, and HTTPS certificate are ready in every collector environment. |
