nfsv4-idmapd-uid-mapping - SKILL.md Agent Skill

name: nfsv4-idmapd-uid-mapping description: | Fix for all file UIDs showing as 65534 (nobody) inside Kubernetes containers when using NFS volumes from TrueNAS/FreeBSD. Use when: (1) ls -lan inside a container shows all files owned by 65534:65534 despite correct ownership on the NFS server, (2) PostgreSQL fails with "data directory has wrong ownership", (3) chown inside containers returns "Invalid argument" on NFS volumes, (4) services that check file ownership (PostgreSQL, MySQL) crash on startup, (5) the same NFS mount shows correct UIDs on the host but 65534 inside containers, (6) NFSv4.2 appears in container mount output even though host mounts use NFSv3. Root cause: Kubernetes inline NFS volumes auto-negotiate NFSv4.2 (not NFSv3), and NFSv4 idmapd fails to map UIDs when domains don't match or users don't exist on the server. author: Claude Code version: 1.0.0 date: 2026-03-01

NFSv4 idmapd UID Mapping — All Files Show as nobody (65534)

Problem

All files on NFS volumes appear owned by UID 65534 (nobody:nogroup) inside Kubernetes containers, even though ls -lan on the NFS server shows the correct UIDs (e.g., 999, 472). This breaks any service that checks file ownership: PostgreSQL refuses to start ("data directory has wrong ownership"), MySQL's entrypoint chown fails with "Invalid argument", and any chown inside the container returns EINVAL.

Context / Trigger Conditions

TrueNAS CORE (FreeBSD) or TrueNAS SCALE as NFS server
NFSv4 enabled on the NFS server (v4: true in TrueNAS NFS config)
Kubernetes using inline NFS volumes (not PV/PVC with mount options)
Key symptom: mount inside the container shows type nfs4 (vers=4.2,...) even though existing kubelet mounts on the host show vers=3
Key symptom: Same NFS path mounted directly on the host shows correct UIDs, but inside any container shows 65534

Root Cause

Kubernetes inline NFS volumes don't support mountOptions. When kubelet mounts NFS for a new pod, the Linux NFS client auto-negotiates the highest available version — NFSv4.2 if the server supports it.

NFSv4 uses idmapd for UID translation: the server translates UID→username (e.g., 999→postgres@domain), sends the username string over the wire, and the client translates it back to a local UID. This fails when:

Domain mismatch: Server domain (from hostname) differs from client domain
- TrueNAS: viktorbarzin.me (from truenas.viktorbarzin.me)
- K8s nodes: viktorbarzin.lan (from k8s-node4.viktorbarzin.lan)
- When domains don't match, ALL UIDs fall back to nobody (65534)
Unknown UIDs: Even with matching domains, if the NFS server has no local user for UID 999 (common for container UIDs), idmapd maps it to nobody

Why existing mounts work: Older kubelet mounts (established before NFSv4 was enabled, or when the NFS client defaulted to v3) continue using NFSv3 with direct numeric UID passthrough. Only NEW mounts negotiate NFSv4.2.

Solution

Fix on TrueNAS (no NFS restart required):

# 1. Enable NFSv3-style numeric UID passthrough for NFSv4
midclt call nfs.update '{"v4_v3owner": true, "v4_domain": "viktorbarzin.lan"}'

# 2. Restart nfsuserd with the correct domain (NOT nfsd — that would crash the cluster)
killall nfsuserd
nfsuserd -domain viktorbarzin.lan -force

Clear caches on all K8s nodes:

for node in k8s-node1 k8s-node2 k8s-node3 k8s-node4; do
  ssh wizard@$node "sudo nfsidmap -c && sudo keyctl clear @u"
done

Key settings explained:

v4_v3owner = true: Makes NFSv4 use numeric UID passthrough like NFSv3, completely bypassing the username-based idmapd translation. This is the critical fix.
v4_domain: Should match the K8s nodes' DNS domain (check with hostname -d on a node)
nfsuserd -domain <domain> -force: FreeBSD daemon that handles NFSv4 user mapping. The -force flag is required if it thinks it's already running.

Verification

# Run a test pod and check UIDs
kubectl run nfs-test --rm -it --restart=Never --image=alpine \
  --overrides='{"spec":{"containers":[{"name":"test","image":"alpine",
  "command":["sh","-c","ls -lan /data | head -5"],
  "volumeMounts":[{"name":"nfs","mountPath":"/data"}]}],
  "volumes":[{"name":"nfs","nfs":{"server":"10.0.10.15","path":"/mnt/main/some-path"}}]}}'

# Should show actual UIDs (e.g., 999, 472) instead of 65534

Debugging Steps

If you're not sure whether this is the issue:

# 1. Check mount type INSIDE a container (not on the host!)
kubectl exec <pod> -- mount | grep nfs
# If it shows "type nfs4" with "vers=4.2" — this is the issue

# 2. Compare UIDs: host vs container
# On host (via kubelet mount path):
sudo ls -lan /var/lib/kubelet/pods/<pod-uid>/volumes/kubernetes.io~nfs/<vol>/
# Inside container:
kubectl exec <pod> -- ls -lan /mount-path/

# 3. Check TrueNAS NFS config
midclt call nfs.config  # Look for v4: true, v4_v3owner, v4_domain

# 4. Check nfsuserd is running with the right domain
ps aux | grep nfsuserd  # On TrueNAS

Notes

NEVER restart NFS (nfsd) on TrueNAS — it causes mount failures across ALL pods cluster-wide. Only restart nfsuserd (the ID mapping daemon).
Existing NFSv3 mounts continue working fine. The issue only affects NEW mounts.
The v4_v3owner setting is persistent across TrueNAS reboots (stored in middleware config).
The nfsuserd restart is NOT persistent — TrueNAS may restart it without the -domain flag after a reboot. The v4_domain setting in the middleware config should handle this, but verify after any TrueNAS restart.
On Linux NFS servers (not FreeBSD/TrueNAS), the equivalent fix is setting Domain in /etc/idmapd.conf on both server and all clients.