nic-structure

star 5.0k

NIC architecture, resource processing pipeline, template systems, and key type definitions. Use when exploring the codebase, understanding data flow, debugging config generation, or working on controller logic.

nginx By nginx schedule Updated 5/18/2026
play_arrow Run Skill in Manus View GitHub

name: nic-structure description: 'NIC architecture, resource processing pipeline, template systems, and key type definitions. Use when exploring the codebase, understanding data flow, debugging config generation, or working on controller logic.'

NIC Architecture and Structure

Repository Layout

cmd/nginx-ingress/              Main binary entry point
pkg/apis/configuration/v1/
  types.go                      CRD struct definitions (source of truth)
  zz_generated.deepcopy.go      Auto-generated DeepCopy (never edit)
pkg/apis/configuration/validation/
  policy.go                     ValidatePolicy entry point
  virtualserver.go              VirtualServer/VSR validation
pkg/client/                     Auto-generated typed clients, informers, listers
internal/k8s/
  controller.go                 Informer setup, sync loop, task dispatch
  policy.go                     syncPolicy handler
  handlers.go                   Event handler factories
  configuration.go              In-memory resource state
  secrets/                      Secret store and validation
  policies/policy_refs.go       Policy reference conversion
internal/configs/
  configurator.go               Orchestrator: merge config, render, write, reload
  virtualserver.go              VirtualServer -> version2 config generation
  ingress.go                    Ingress -> version1 config generation
  transportserver.go            TransportServer -> version2 stream config generation
  policy.go                     generatePolicies() dispatcher + add*Config() methods
  annotations.go                Annotation constants + parseAnnotations()
  config_params.go              ConfigParams struct + defaults
  configmaps.go                 ConfigMap -> ConfigParams merge
  dos.go                        DoS protection config generation
  common.go                     Shared config utilities
  warnings.go                   Warning accumulation types
  validation_results.go         validationResults type (isError + warnings)
  commonhelpers/                Shared template helper functions (v1 + v2)
  oidc/                         OIDC config files (openid_connect.js, oidc_common.conf)
  njs/                          NJS scripts (apikey_auth.js)
  version1/                     Ingress template structs + .tmpl files
    __snapshots__/              Snapshot golden files
  version2/                     VirtualServer/TS template structs + .tmpl files
    __snapshots__/              Snapshot golden files
internal/nginx/                 NGINX process manager, reload, rollback, version detection
internal/metrics/               Prometheus metrics collectors and listeners
internal/telemetry/             Usage telemetry collection and export
internal/certmanager/           cert-manager integration controller
internal/externaldns/           ExternalDNS integration controller
charts/nginx-ingress/           Helm chart (values.yaml, schema, templates)
charts/tests/                   Helm snapshot tests (terratest + go-snaps)
tests/suite/                    Python integration tests (pytest)
tests/data/                     Test YAML manifests by feature
config/crd/bases/               Generated CRD YAML (from controller-gen)
deploy/                         Pre-built CRD YAML bundles (crds.yaml, crds-nap-*.yaml)
hack/                           update-codegen.sh, verify-codegen.sh

Architectural Layers

Each layer has a strict ownership boundary. Identify the correct layer before placing any change.

Layer Package(s) Owns
Data model pkg/apis/configuration/v1/ CRD struct definitions, generated DeepCopy
Validation pkg/apis/configuration/validation/, internal/k8s/validation.go CRD field validation (kubebuilder markers), Ingress annotation validation
Controller internal/k8s/ Event handling, in-memory state, secret resolution, sync handlers, status updates
Config generation internal/configs/, version1/, version2/ Extended resource → NGINX config struct → template render → file write
Process management internal/nginx/ NGINX process lifecycle, reload, rollback

Layer crossing rules — violations cause architectural drift:

  • Config generation (internal/configs/) must NOT call the k8s API or access SecretStore directly — it receives pre-resolved SecretReference{Path} via extended resources.
  • Controller (internal/k8s/) must NOT generate NGINX config text or render templates.
  • Data model (types.go) must NOT import internal/configs or internal/k8s.
  • Validation layer must NOT trigger NGINX reloads or update k8s status.

Resource Processing Pipeline

kubectl apply -f resource.yaml
  -> K8s API Server persists resource
  -> Informer detects Add/Update/Delete event
      [handlers.go: createXxxHandlers(); IsSupportedSecretType() gates secret events]
  -> Event handler enqueues task onto syncQueue
      [controller.go: AddSyncQueue()]
  -> Controller dispatches task
      [controller.go: sync() -> syncVirtualServer() / syncIngress() / syncSecret() / syncPolicy() / ...]
  -> Build / update in-memory state, returning []ResourceChange
      [configuration.go: AddOrUpdateVirtualServer() / AddOrUpdateIngress()]
      Validation (CRD fields):          pkg/apis/configuration/validation/
      Validation (Ingress annotations): internal/k8s/validation.go
  -> Find affected resources (fans out when a secret or policy changes)
      [configuration.go: FindResourcesForSecret() / FindResourcesForPolicy()]
  -> Resolve secret references  <-- controller layer resolves; configurator only consumes paths
      [controller.go: createVirtualServerEx() / createIngressEx() -> secretStore.GetSecret()]
      [secrets/store.go: GetSecret() lazily writes valid secret to filesystem via SecretFileManager]
  -> Build extended resources
      [controller.go: createVirtualServerEx()   -> VirtualServerEx]
                     [createIngressEx()          -> IngressEx]
                     [createTransportServerEx()  -> TransportServerEx]
  -> Configurator generates NGINX config  [internal/configs/configurator.go: AddOrUpdateVirtualServer()]
      HTTP path:   GenerateVirtualServerConfig() [virtualserver.go]    -> version2.VirtualServerConfig
                   generateNginxCfg()            [ingress.go]          -> version1.IngressNginxConfig
      Stream path: generateTransportServerConfig(...) [transportserver.go] -> *version2.TransportServerConfig
      Policies:    generatePolicies() -> add*Config() -> policiesCfg   [policy.go]
      OSS vs Plus: Configurator.isPlus flag; Plus-only policies = OIDC, WAF
                   Template level: nginx.virtualserver.tmpl vs nginx-plus.virtualserver.tmpl
  -> Template executor renders NGINX config text
      [version1.TemplateExecutor / version2.TemplateExecutorV2;
       TransportServer uses ExecuteTransportServerTemplate(...)]
  -> NginxManager writes files + reloads NGINX
      [internal/nginx/: Manager.CreateConfig() + Manager.Reload()]
  -> Update resource status + emit Kubernetes events  [happens AFTER reload returns]
      [controller.go: updateVirtualServerStatusAndEvents() / updateIngressStatusAndEvents()]
      [k8s/status.go: statusUpdater.UpdateVirtualServerStatus()]
      Startup optimisation: status updates deferred to pendingVSStatus slices during !isNginxReady;
      flushed in background via flushPendingStatusesAsync() after first reload.

Secret Store

The secret store (internal/k8s/secrets/) sits entirely in the controller layer and uses a two-phase model to avoid writing unreferenced files to the NGINX filesystem.

Phase 1 — in-memory validation (SecretStore.AddOrUpdateSecret()): Validates the secret via ValidateSecret() and stores SecretReference{Secret, Error} in memory. Does not write to disk unless a filesystem path already exists for that secret.

Phase 2 — lazy filesystem write (SecretStore.GetSecret()): On first reference during createVirtualServerEx() / createIngressEx(), materializes supported secrets under /etc/nginx/secrets/ via SecretFileManager (implemented by Configurator). Filenames are derived from <namespace>-<secretName> rather than a literal path. Some secret types create multiple files (for example, CA secrets), while OIDC and API key secrets are not written to disk, so their Path is empty. Returns SecretReference{Path, Error}.

Supported secret types (internal/k8s/secrets/validation.go):

Constant Kubernetes type Used for
— kubernetes.io/tls TLS server certs
SecretTypeCA nginx.org/ca CA cert (mTLS / upstream trust)
SecretTypeJWK nginx.org/jwk JWT validation keys
SecretTypeOIDC nginx.org/oidc OIDC client secret
SecretTypeHtpasswd nginx.org/htpasswd HTTP Basic auth
SecretTypeAPIKey nginx.org/apikey API key auth
SecretTypeLicense nginx.com/license NGINX Plus license

Special secrets (defaultServer TLS, wildcard TLS, license, mgmt client cert, mgmt trusted CA): handled by handleSpecialSecretUpdate() in the controller, which triggers an NGINX reload directly — independent of any resource re-sync.

Key invariant: The controller resolves secrets before they reach config generation. VirtualServerEx.SecretRefs / IngressEx.SecretRefs carry map[string]*secrets.SecretReference, and internal/configs/ may consume the pre-resolved data on those references, including .Path, .Secret.Type, and .Secret.Data. Do not add SecretStore.GetSecret() calls or any direct Kubernetes API access inside internal/configs/.

Two Template Systems

Pipeline Resources Package Templates
Version 1 Ingress internal/configs/version1/ nginx.ingress.tmpl, nginx-plus.ingress.tmpl
Version 2 VirtualServer, VSR, TS internal/configs/version2/ nginx.virtualserver.tmpl, nginx-plus.virtualserver.tmpl
  • Version 1: IngressNginxConfig with multiple Server blocks per config
  • Version 2: VirtualServerConfig with single Server block per config
  • Main templates (nginx.tmpl, nginx-plus.tmpl) produce global nginx.conf
  • Both share generatePolicies() in internal/configs/policy.go

Policy System

Policies are mutually exclusive: each Policy CR has exactly ONE non-nil field in PolicySpec.

Types: AccessControl, RateLimit, JWTAuth, ExternalAuth, BasicAuth, IngressMTLS, EgressMTLS, OIDC, WAF, APIKey, Cache, CORS.

Application levels (VirtualServer):

  • spec.policies -- server-level (all routes unless overridden)
  • route.policies -- route-level (overrides spec-level)
  • subroute.policies -- VirtualServerRoute subroute-level

Ingress: Policies referenced via IngressEx.Policies map. Annotations are Ingress-only, never on VS/VSR.

Key Types

policiesCfg (internal/configs/policy.go): Aggregation struct holding resolved policies per context (Allow/Deny slices, RateLimit, JWTAuth, ExternalAuth, BasicAuth, IngressMTLS, EgressMTLS, OIDC, APIKey, WAF, Cache, CORSHeaders/CORSMap, Context, BundleValidator, ErrorReturn).

version2.VirtualServerConfig: Top-level struct with HTTP-level directives (Maps, LimitReqZones, CacheZones) and a single Server block.

version2.Location: Per-route struct with all policy fields (Allow, Deny, LimitReqs, JWTAuth, Cache, CORSEnabled, AddHeaders).

version1.IngressNginxConfig: Top-level Ingress struct with multiple Server blocks plus Maps, CORSHeaders, LimitReqZones.

ConfigParams (config_params.go): ~125 fields for tunable NGINX params. Flow: defaults -> ConfigMap -> Ingress annotations.

CRD Struct Pattern

// +kubebuilder:resource:shortName=pol
// +kubebuilder:subresource:status
// +kubebuilder:storageversion
type Policy struct {
    metav1.TypeMeta   `json:",inline"`
    metav1.ObjectMeta `json:"metadata"`
    Spec              PolicySpec   `json:"spec"`
    Status            PolicyStatus `json:"status"`
}
  • Types: PascalCase singular. Spec/Status: <CRD>Spec, <CRD>Status. Lists: <CRD>List.
  • Short names: vs, vsr, ts, gc, pol. API group: k8s.nginx.org/v1.

Kubebuilder Markers

Marker Purpose
+kubebuilder:validation:Required Field must be present
+kubebuilder:validation:Optional Field is optional
+kubebuilder:validation:Pattern= `regex` Regex validation
+kubebuilder:validation:Minimum=N Numeric minimum
+kubebuilder:default=value Default value
+kubebuilder:validation:XValidation:rule="CEL" Cross-field CEL validation

Error Handling

  • Warnings: map[runtime.Object][]string in internal/configs/warnings.go
  • validationResults: isError bool + warnings []string. When isError = true, policy dispatcher returns ErrorReturn: {Code: 500}
  • Validation errors: Kubernetes field.ErrorList from k8s.io/apimachinery/pkg/util/validation/field
Install via CLI
npx skills add https://github.com/nginx/kubernetes-ingress --skill nic-structure
Repository Details
star Stars 5,034
call_split Forks 2,045
navigation Branch main
article Path SKILL.md
Occupations
Information Security Analysts
More from Creator
nginx
nginx Explore all skills →