splunk-platform-pki-setup

name: splunk-platform-pki-setup description: >- Render, preflight, apply, validate, rotate, and inventory private or public PKI for Splunk Enterprise TLS surfaces: Splunk Web, splunkd REST, S2S, HEC, KV Store, indexer clusters, SHC, License Manager, Deployment Server, Monitoring Console, Federated Search, heavy forwarders, Universal Forwarders, Edge Processor, SAML SP signing, LDAPS trust, and CLI CA trust. Covers CSR handoffs, internal CA rendering, FIPS mode, TLS policy presets, KV Store EKU enforcement, default-cert refusal, SAN-aware leaf certs, mTLS, replication-port TLS, and delegated rotation runbooks. Use when the user asks to build Splunk PKI, mint certs, prepare third-party CA CSRs, replace default certs, configure mTLS, fix KV Store cert validation, encrypt replication traffic, configure SAML/LDAPS trust, or rotate Splunk TLS certificates.

Splunk Platform PKI Setup

This skill owns the full TLS / PKI lifecycle for a self-managed Splunk Enterprise deployment. It runs in either of two modes:

• Private PKI — the skill renders scripts that build an internal Root CA (and optional Intermediate), then mint per-component server / client certificates with the right basicConstraints, keyUsage, and extendedKeyUsage (including the dual serverAuth
  • clientAuth EKU that KV Store 7.0+ requires), with per-host SANs.
• Public PKI — the skill renders per-host CSRs + openssl.cnf and a handoff Markdown for the operator's third-party CA (HashiCorp Vault PKI, ACME / cert-manager / Let's Encrypt, Microsoft AD CS, EJBCA, or any commercial CA). It installs and validates the returned signed PEMs but never embeds CA credentials.

It is render-first: the default phase produces a reviewable directory of CA scripts, CSR templates, install / verify scripts, per-role distribution payloads (cluster bundle, SHC deployer bundle, standalone, forwarder fleet, Edge Processor placeholders), rotation runbooks, and operator handoff Markdown. It refuses to apply changes until the operator passes --accept-pki-rotation.

Read this first — what this skill does NOT do

• It does not talk to a CA. Public-PKI mode renders CSRs and a handoff Markdown; the operator submits to Vault / ACME / AD CS / EJBCA / commercial CA out of band.
• It does not implement rolling restart or cluster bundle apply. Both are delegated to skills/splunk-indexer-cluster-setup (matches the repo precedent set by pass4SymmKey rotation, which is also operator-orchestrated).
• It does not configure Splunk Web HSTS / CSP / browser security headers. Splunk Web has no customHttpHeaders; those headers come from the reverse proxy and are owned by skills/splunk-enterprise-public-exposure-hardening.
• It does not force TLS 1.3. Splunk's TLS-protocol-version doc lists tls1.2 as the maximum supported version; the skill defaults to and refuses anything below tls1.2.
• It does not build the FIPS-validated OpenSSL module. The operator owns the FIPS module; the skill flips SPLUNK_FIPS_VERSION in splunk-launch.conf.
• It does not issue certificates for Splunk Cloud. It refuses and emits the Universal Forwarder Credentials Package handoff. Splunk Cloud's ACS does not currently expose a self-service BYOC endpoint for HEC custom-domain certificates; operators open a Splunk Support ticket or deploy an inputs.conf-in-app instead.
• It does not generate Java keystores / truststores (JKS / PKCS#12). Splunk DB Connect uses those and is intentionally out of scope.
• It does not own Splunk SOAR PKI, Splunk Mobile / Secure Gateway certs, IdP-side configuration, HSM integration for the CA private key, or CRL / OCSP responder hosting. Each is referenced where relevant but operator-driven.
• It does not certify compliance (PCI / HIPAA / FedRAMP / SOC 2 / DISA STIG). It renders STIG-aligned configs (--tls-policy stig) and cites NIST controls in references/fips-and-common-criteria.md but does not attest.

Architecture the skill assumes

• Private mode builds an internal root/intermediate CA and signs the role-specific leaf certificates.
• Public mode renders CSRs and operator handoffs for Vault PKI, ACME, AD CS, EJBCA, or a commercial CA.
• Rendered outputs become cluster-bundle drop-ins, SHC deployer apps, standalone overlays, and UF fleet overlays.
• Cluster bundle apply and rolling restart remain delegated to splunk-indexer-cluster-setup; SHC app push remains delegated to splunk-agent-management-setup.

Agent behavior — credentials

Never paste secrets into chat or pass them on argv. The skill consumes file paths for every secret and never embeds secret values in rendered output:

bash skills/shared/scripts/write_secret_file.sh /tmp/splunk_admin_password
bash skills/shared/scripts/write_secret_file.sh /tmp/splunk_idxc_secret
bash skills/shared/scripts/write_secret_file.sh /tmp/pki_root_ca_key_password
bash skills/shared/scripts/write_secret_file.sh /tmp/pki_intermediate_ca_key_password
bash skills/shared/scripts/write_secret_file.sh /tmp/pki_leaf_key_password
bash skills/shared/scripts/write_secret_file.sh /tmp/pki_saml_sp_key_password

Pass them in via --admin-password-file, --idxc-secret-file, --ca-key-password-file, --intermediate-ca-key-password-file, --leaf-key-password-file, --saml-sp-key-password-file.

The rendered pki/install/install-leaf.sh script accepts a separate --ssl-password-file PATH flag — this is the plaintext leaf-key passphrase the operator copies to each target host. install-leaf.sh writes it verbatim to the sslPassword line of the per-host overlay ($SPLUNK_HOME/etc/system/local/server.conf | web.conf | inputs.conf | outputs.conf) and on first restart Splunk encrypts it with splunk.secret. In typical deployments --ssl-password-file and --leaf-key-password-file reference the same plaintext file (the leaf key's passphrase, which is what Splunk needs to read the key). Omit --ssl-password-file when the leaf key is unencrypted (e.g. PKCS#8 nocrypt for Edge Processor).

For non-secret values (FQDNs, SANs, role inventory, validity days, key algorithm, mTLS surfaces, FIPS mode, TLS preset) use template.example.

Quick start

Render a Private PKI for a 3-peer indexer cluster + 3-member SHC with default Splunk-modern algorithms, mTLS on S2S + HEC, hostname validation everywhere, and the splunkd cert distributed through the cluster bundle:

bash skills/splunk-platform-pki-setup/scripts/setup.sh \
  --phase render \
  --mode private \
  --target indexer-cluster,shc,license-manager,deployment-server,monitoring-console \
  --cm-fqdn cm01.example.com \
  --peer-hosts idx01.example.com,idx02.example.com,idx03.example.com \
  --shc-deployer-fqdn deployer01.example.com \
  --shc-members sh01.example.com,sh02.example.com,sh03.example.com \
  --lm-fqdn lm01.example.com \
  --ds-fqdn ds01.example.com \
  --mc-fqdn mc01.example.com \
  --enable-mtls s2s,hec \
  --tls-policy splunk-modern \
  --include-intermediate-ca true

Render a Public PKI for the same cluster with a HashiCorp Vault PKI operator handoff and the SAML SP signing cert:

bash skills/splunk-platform-pki-setup/scripts/setup.sh \
  --phase render \
  --mode public \
  --target indexer-cluster,shc,license-manager,saml-sp \
  --cm-fqdn cm01.example.com \
  --peer-hosts idx01.example.com,idx02.example.com,idx03.example.com \
  --shc-deployer-fqdn deployer01.example.com \
  --shc-members sh01.example.com,sh02.example.com,sh03.example.com \
  --lm-fqdn lm01.example.com \
  --saml-sp true \
  --public-ca-name vault \
  --leaf-days 397

Render a FIPS 140-3 Private PKI with the indexer-cluster replication port encrypted (atomic migration of [replication_port://9887] to [replication_port-ssl://9887]):

bash skills/splunk-platform-pki-setup/scripts/setup.sh \
  --phase render \
  --mode private \
  --target indexer-cluster \
  --cm-fqdn cm01.example.com \
  --peer-hosts idx01.example.com,idx02.example.com,idx03.example.com \
  --fips-mode 140-3 \
  --tls-policy fips-140-3 \
  --encrypt-replication-port true \
  --key-algorithm rsa-2048 \
  --include-intermediate-ca true

Render the Edge Processor cert pair (RSA-2048 by default; pass --key-algorithm ecdsa-p256 for ECDSA):

bash skills/splunk-platform-pki-setup/scripts/setup.sh \
  --phase render \
  --mode private \
  --target edge-processor \
  --include-edge-processor true \
  --ep-fqdn ep01.example.com \
  --ep-data-source-fqdn datasource01.example.com \
  --key-format pkcs8

Run preflight against a live host (read-only checks; refuses to apply):

bash skills/splunk-platform-pki-setup/scripts/setup.sh \
  --phase preflight \
  --mode private \
  --target indexer-cluster,shc \
  --cm-fqdn cm01.example.com \
  --admin-password-file /tmp/splunk_admin_password

Inventory live cert posture (read-only; never writes):

bash skills/splunk-platform-pki-setup/scripts/setup.sh \
  --phase inventory \
  --target all \
  --admin-password-file /tmp/splunk_admin_password

Apply rendered certs to a search head (mutates Splunk; requires the explicit accept flag):

bash skills/splunk-platform-pki-setup/scripts/setup.sh \
  --phase apply \
  --mode private \
  --target shc \
  --shc-deployer-fqdn deployer01.example.com \
  --accept-pki-rotation \
  --admin-password-file /tmp/splunk_admin_password \
  --leaf-key-password-file /tmp/pki_leaf_key_password

Validate live state post-apply:

bash skills/splunk-platform-pki-setup/scripts/validate.sh \
  --target indexer-cluster,shc \
  --cm-fqdn cm01.example.com \
  --admin-password-file /tmp/splunk_admin_password

What it renders

Under the project root in splunk-platform-pki-rendered/:

• pki/private-ca/ — only when --mode private: create-root-ca.sh, create-intermediate-ca.sh, sign-server-cert.sh, sign-client-cert.sh, sign-saml-sp.sh, plus openssl-*.cnf files with the documented basicConstraints / keyUsage / extendedKeyUsage extensions, and a README.md that walks the operator through CA generation. Uses $SPLUNK_HOME/bin/splunk cmd openssl genpkey/req/x509 per Splunk's documented workflow so the same OpenSSL build that Splunk uses signs and verifies.
• pki/csr-templates/<role>-<host>.cnf + generate-csr.sh — emitted in both modes; per-host CSR config with SANs and EKU.
• pki/install/install-leaf.sh, verify-leaf.sh, kv-store-eku-check.sh, align-cli-trust.sh, install-fips-launch-conf.sh, prepare-key.sh — cert install + verify per host. kv-store-eku-check.sh runs the documented splunk cmd openssl verify -x509_strict check from the KV Store custom-cert prep doc and refuses to declare a host ready unless the verification returns OK.
• pki/distribute/cluster-bundle/master-apps/000_pki_trust/local/ — cluster-bundle drop-in: server.conf (with [replication_port-ssl://9887] if --encrypt-replication-port=true), inputs.conf ([splunktcp-ssl:9997] + [SSL]).
• pki/distribute/shc-deployer/shcluster/apps/000_pki_trust/local/ — SHC deployer drop-in: server.conf, web.conf, inputs.conf.
• pki/distribute/standalone/000_pki_trust/local/ — for non-clustered roles (LM, DS, MC, single SH, HF): server.conf, web.conf, inputs.conf, outputs.conf, authentication.conf, deploymentclient.conf, splunk-launch.conf (when FIPS), system-files/ldap.conf (when LDAPS).
• pki/distribute/forwarder-fleet/<group>/{outputs-overlay.conf,server-overlay.conf} — UF / HF outputs overlay with clientCert / sslVerifyServerCert=true / sslVerifyServerName=true and per-indexer [tcpout-server://host:port] SAN overrides.
• pki/distribute/edge-processor/ — only when --include-edge-processor=true: 5-file PEM placeholders (ca_cert.pem.example, edge_server_cert.pem.example, edge_server_key.pem.example, data_source_client_cert.pem.example, data_source_client_key.pem.example) + upload-via-rest.sh.example for the EP REST upload and README.md for the EP UI walkthrough.
• pki/distribute/saml-sp/ — only when --saml-sp=true: sp-signing.crt, sp-signing.key.placeholder, README.md for re-uploading IdP metadata after rotation.
• pki/rotate/{plan-rotation.md, rotate-leaf-host.sh, swap-trust-anchor.sh, swap-replication-port-to-ssl.sh, expire-watch.sh} — rotation helpers with the delegated rolling-restart runbook.
• handoff/ — Markdown checklists for Vault PKI, ACME / cert-manager, Microsoft AD CS, EJBCA, Splunk Cloud UFCP / BYOC, FIPS migration, Edge Processor upload, post-install monitoring (SSL Cert Checker Splunkbase 3172, /server/health/splunkd REST endpoint, CIM Certificates data model), and the operator checklist.
• preflight.sh, validate.sh, inventory.sh, README.md, metadata.json.

Phases

• render (default) — produce the reviewable rendered tree. No Splunk REST calls; safe to run anywhere.
• preflight — render then run the live preflight checks: cert directory permissions, default-cert refusal, KV-Store EKU verification (splunk cmd openssl verify -x509_strict must return OK), splunk.secret SHA-256 parity across cluster members, FIPS posture (refuses mid-Phase-1 / Phase-2 migration), hostname-validation gating, TLS protocol floor check (sslVersions = tls1.2), per-host splunk btool server list sslConfig snapshot, replication-port mode (cleartext vs SSL), [shclustering] pass4SymmKey presence reminder. Refuses to mark the deployment ready when any check fails.
• apply — render then run the per-role pki/install/install-leaf.sh + align-cli-trust.sh + install-fips-launch-conf.sh if FIPS. Requires --accept-pki-rotation (a single-flag acknowledgement that the operator is about to swap serving certs and trigger downstream restart).
• rotate — render then emit a rotation runbook (pki/rotate/plan-rotation.md) describing the full delegated order. Does NOT exec the rolling restart itself (delegate pattern, see "Rotation ownership" below).
• validate — render then run the live validation probes: REST + openssl s_client -connect per surface, KV Store handshake check, splunk show-decrypted round-trip on sslPassword, SAML SP signing cert exposed in IdP-metadata endpoint.
• inventory — read-only: collects splunk btool server list sslConfig, web list sslConfig, inputs list http SSL, dumps PEM expiry catalogue and emits pki/inventory/<host>.json. No Splunk write operations. No --accept-… required.
• all — render + preflight + apply + validate, gated by --accept-pki-rotation.

Apply guard — --accept-pki-rotation

The skill refuses to run apply or all without --accept-pki-rotation. This is a single-flag acknowledgement that:

• The new cert chain has been verified (verify-leaf.sh returned OK).
• A rolling restart of the indexer cluster and SHC will follow (delegated to splunk-indexer-cluster-setup --phase rolling-restart).
• The SAML / LDAPS / Edge Processor / Splunk Cloud handoffs (where applicable) will be completed.
• The operator has a rollback plan (the previous PEM directory is preserved).

The render and preflight phases never need this flag.

Rotation ownership — delegate

The skill emits pki/rotate/plan-rotation.md describing the order of operations:

1. Stage the new trust anchor in the cluster bundle and SHC deployer apps (already rendered).
2. Run bash skills/splunk-indexer-cluster-setup/scripts/setup.sh --phase bundle-validate then --phase bundle-apply.
3. Run bash skills/splunk-indexer-cluster-setup/scripts/setup.sh --phase rolling-restart --rolling-restart-mode searchable to roll the indexer cluster.
4. Stage new leaf certs on each non-clustered role (LM, DS, MC, single SH, HF) and run splunk restart.
5. On the SHC deployer, run splunk apply shcluster-bundle -target https://captain01.example.com:8089 and let the SHC roll its members.
6. Roll the forwarder fleet via bash skills/splunk-agent-management-setup/scripts/setup.sh.
7. Run this skill's validate phase to confirm.

Every step is a single shell line that the operator runs. No tight coupling, no duplicated rolling-restart logic. This matches the repo precedent set by splunk-indexer-cluster-setup ("Cluster pass4SymmKey rotation… does not orchestrate that rolling rotation").

Cross-skill handoff matrix

The skill consumes — does not duplicate — these. When you also use one of the adjacent skills below, run THIS skill's render + preflight first to establish the cert paths, then layer the adjacent skill's behaviour on top.

TLS algorithm presets

Pick with --tls-policy {splunk-modern|fips-140-3|stig}. Keep the detailed cipher and key constraints in references/algorithm-presets.md; the renderer also consumes references/algorithm-policy.json.

TLS protocol floor

Defaults to sslVersions = tls1.2 and sslVersionsForClient = tls1.2. Read references/tls-protocol-policy.md for the upstream support table and the guarded --allow-deprecated-tls path.

FIPS mode

--fips-mode {none|140-2|140-3} — when set, the renderer emits a splunk-launch.conf overlay with SPLUNK_FIPS_VERSION = 140-3 (or 140-2). Read references/fips-and-common-criteria.md before apply; the skill refuses mid-migration states.

Validity-day defaults

Override with --root-ca-days, --intermediate-ca-days, and --leaf-days.

Key format / permissions

Default private keys are encrypted PKCS#1 PEM. Use --key-format pkcs8 for Edge Processor or DB Connect compatibility. The install and validation details live in references/key-format-and-permissions.md.

mTLS surfaces

Opt-in via --enable-mtls {none|s2s|hec|splunkd|all}. Default is s2s,hec. Read references/mtls-and-hostname-validation.md before enabling splunkd mTLS because it can break operator tooling that does not present a client cert.

References

Read reference.md before any apply. Topical deep dives (each anchored to a specific upstream Splunk doc captured in references/authoritative-sources.md):

• references/component-cert-matrix.md
• references/private-pki-workflow.md
• references/public-pki-workflow.md
• references/handoff-vault-pki.md
• references/handoff-acme-cert-manager.md
• references/handoff-microsoft-adcs.md
• references/handoff-ejbca.md
• references/kv-store-eku-requirements.md
• references/mtls-and-hostname-validation.md
• references/replication-port-tls.md
• references/saml-signing-certs.md
• references/ldaps-trust.md
• references/edge-processor-pki.md
• references/cli-trust-cacert-alignment.md
• references/tls-protocol-policy.md
• references/algorithm-presets.md
  • algorithm-policy.json (machine-readable companion consumed by renderer + preflight)
• references/fips-and-common-criteria.md
• references/key-format-and-permissions.md
• references/rotation-runbook.md
• references/post-install-monitoring.md
• references/splunk-cloud-ufcp-handoff.md
• references/troubleshooting.md
• references/authoritative-sources.md