observability

name: observability description: Guide for working with observability features including rate limiting, visualization, alerting, and request tracing.

Observability Features

Use this skill when working with the observability features: rate limiting, telemetry visualization, alerting, and request tracing.

Feature Documentation

Comprehensive documentation is available in architecture/features/observability/:

1. Rate Limiting

Overview

Rate limiting uses @nestjs/throttler with a custom RateLimitGuard for per-endpoint configuration.

Key Files

• src/server/modules/rate-limit/rate-limiter.guard.ts - Custom guard
• src/server/modules/rate-limit/rate-limit.config.ts - Per-endpoint rules
• src/server/modules/rate-limit/rate-limit.service.ts - Metrics tracking

Adding New Rate Limit Rules

// rate-limit.config.ts
export const rateLimitRules: RateLimitRule[] = [
  {
    pattern: '/api/your-endpoint',
    limit: 30,
    ttlSeconds: 60,
  },
];

Testing Rate Limits

// Integration test
it('should rate limit after threshold', async () => {
  for (let i = 0; i < limit; i++) {
    await request(app.getHttpServer()).get('/api/endpoint').expect(200);
  }
  await request(app.getHttpServer()).get('/api/endpoint').expect(429);
});

2. Visualization Components

Overview

Telemetry visualization uses Recharts for charts and SSE for real-time updates.

Key Components

Testing Recharts

Use the mockRecharts helper to mock Recharts components:

import { mockRecharts } from 'ui/test-utils/mockRecharts';

jest.mock('recharts', () => mockRecharts);

it('should render chart', async () => {
  render(<TracesContainer />);
  await waitFor(() => {
    // ResponsiveContainer renders as a div with specific class
    expect(document.querySelector('.recharts-responsive-container')).toBeInTheDocument();
  });
});

Testing SSE Streams

Use MockEventSource for SSE testing:

import { MockEventSource } from 'ui/test-utils/mockEventSource';

beforeEach(() => {
  global.EventSource = MockEventSource as unknown as typeof EventSource;
});

it('should handle SSE messages', async () => {
  render(<TracesContainer />);

  // Simulate SSE message
  MockEventSource.simulateMessage('traces', {
    type: 'trace.created',
    data: { id: 1, path: '/api/test' },
  });

  await waitFor(() => {
    expect(screen.getByText('/api/test')).toBeInTheDocument();
  });
});

3. Alerting System

Overview

Alerts are triggered when metrics exceed thresholds. Alert channels implement the IAlertChannel interface.

Key Files

• src/server/modules/traces/trace-alert.service.ts - Core alerting logic
• src/server/modules/traces/channels/*.channel.ts - Alert channels
• src/shared/types/trace.types.ts - Alert type definitions

Adding New Alert Channels

1. Create channel class implementing IAlertChannel:

// src/server/modules/traces/channels/slack.channel.ts
import { Injectable } from '@nestjs/common';
import { IAlertChannel, AlertPayload } from '../alert.types';

@Injectable()
export class SlackChannel implements IAlertChannel {
  readonly name = 'slack';

  async send(payload: AlertPayload): Promise<void> {
    // Implement Slack webhook call
  }

  isEnabled(): boolean {
    return !!process.env.SLACK_WEBHOOK_URL;
  }
}

2. Register in TracesModule:

providers: [
  AlertService,
  LogChannel,
  SlackChannel, // Add new channel
],

3. Inject in AlertService:

constructor(
  private readonly logChannel: LogChannel,
  private readonly slackChannel: SlackChannel,
) {
  this.channels = [logChannel, slackChannel];
}

4. Request Tracing

Overview

Request tracing captures timing, status, and path information for all API requests.

Key Files

• src/server/shared/interceptors/tracing.interceptor.ts - Captures request traces
• src/server/modules/traces/trace.service.ts - Trace storage and queries
• src/server/modules/traces/events.ts - Event name constants
• src/shared/types/trace.types.ts - Trace type definitions

Trace Data Flow

Request → TracingInterceptor → TraceService.recordTrace()
        → EventEmitter.emit(TRACE_EVENTS.TRACE_CREATED) → SSE to UI

5. Testing Patterns

Cron Job Testing

Use cronTestUtils for testing scheduled tasks:

import {
  simulateCronExecution,
  parseCronExpression,
} from 'server/test-utils/cronTestUtils';

it('should validate cron expression', () => {
  const expression = '0 0 * * *'; // Daily at midnight
  const result = parseCronExpression(expression);
  expect(result.hour).toBe(0);
  expect(result.minute).toBe(0);
});

it('should run cleanup on schedule', async () => {
  await simulateCronExecution(service, 'runCleanup');
  expect(mockRepo.delete).toHaveBeenCalled();
});

Integration Test Patterns

describe('Traces Integration', () => {
  let app: INestApplication;
  let dataSource: DataSource;

  beforeAll(async () => {
    const module = await Test.createTestingModule({
      imports: [
        TracesModule,
        TypeOrmModule.forRoot({
          type: 'better-sqlite3',
          database: ':memory:',
          entities: [TraceEntity, AlertEntity],
          synchronize: true,
        }),
      ],
    }).compile();

    app = module.createNestApplication();
    await app.init();
  });

  it('should record traces', async () => {
    // Make request
    await request(app.getHttpServer()).get('/api/traces');

    // Verify trace recorded
    const traces = await request(app.getHttpServer())
      .get('/api/traces')
      .expect(200);
    expect(traces.body.length).toBeGreaterThan(0);
  });
});

6. Environment Variables

See architecture/features/observability/configuration.md for complete reference.

7. Troubleshooting

Charts Not Rendering in Tests

Mock Recharts with the mockRecharts helper:

jest.mock('recharts', () => mockRecharts);

SSE Connection Issues in Tests

Use MockEventSource instead of real EventSource:

global.EventSource = MockEventSource as unknown as typeof EventSource;

Rate Limit Not Applying

Check that:

1. The pattern matches your endpoint path
2. The guard is applied to the controller/route
3. The ThrottlerModule is imported in your module

Alerts Not Firing

Check that:

1. Thresholds are configured correctly
2. Cooldown period hasn't blocked the alert
3. At least one channel is enabled