CoreWeave Observability

Community-contributed. Not affiliated with, endorsed by, or sponsored by CoreWeave, Inc. CoreWeave is a registered trademark of CoreWeave, Inc.

Overview

CoreWeave runs GPU-intensive workloads on Kubernetes where hardware failures, memory exhaustion, and underutilization directly impact cost and reliability. Observability must cover DCGM GPU metrics, Kubernetes pod health, inference latency, and job completion rates. Proactive monitoring prevents wasted spend on idle GPUs and catches OOM conditions before they cascade.

Prerequisites

• A metrics backend receiving Kubernetes and DCGM exporter metrics.
• A named dashboard and on-call owner for the namespace or service.
• Log and trace redaction rules that exclude prompts, model outputs, tokens, and credentials.

Instructions

1. Tag metrics with bounded values such as namespace, model family, and status; do not use request IDs, prompts, or user identifiers as labels.
2. Build dashboards for utilization, memory, queue depth, latency, error rate, and restart rate, then set alert thresholds from a measured baseline.
3. Route critical alerts to the responsible on-call team and link a runbook that includes a safe scale-down or rollback action.
4. Test one alert in a non-production namespace and verify that the receipt contains only operational metadata, not workload data.

Key Metrics

Instrumentation

async function trackInference(model: string, fn: () => Promise<any>) {
  const start = Date.now();
  try {
    const result = await fn();
    metrics.record('coreweave.inference.latency', Date.now() - start, { model, status: 'ok' });
    metrics.increment('coreweave.inference.completed', { model });
    return result;
  } catch (err) {
    metrics.increment('coreweave.inference.errors', { model, error: err.code });
    throw err;
  }
}

Health Check Dashboard

async function coreweaveHealth(): Promise<Record<string, string>> {
  const gpu = await queryPrometheus('avg(DCGM_FI_DEV_GPU_UTIL)');
  const mem = await queryPrometheus('avg(DCGM_FI_DEV_FB_USED/(DCGM_FI_DEV_FB_USED+DCGM_FI_DEV_FB_FREE))');
  const pods = await queryPrometheus('kube_deployment_status_replicas_available{namespace="inference"}');
  return {
    gpu_utilization: gpu > 20 ? 'healthy' : 'underutilized',
    gpu_memory: mem < 0.9 ? 'healthy' : 'critical',
    inference_pods: pods > 0 ? 'healthy' : 'down',
  };
}

Alerting Rules

const alerts = [
  { metric: 'DCGM_FI_DEV_GPU_UTIL', condition: 'avg < 20', window: '30m', severity: 'warning' },
  { metric: 'gpu_memory_pct', condition: '> 0.95', window: '5m', severity: 'critical' },
  { metric: 'inference_latency_p99', condition: '> 500ms', window: '10m', severity: 'warning' },
  { metric: 'pod_restart_count', condition: '> 3', window: '15m', severity: 'critical' },
];

Structured Logging

function logGpuEvent(event: string, node: string, data: Record<string, any>) {
  console.log(JSON.stringify({
    service: 'coreweave', event, node,
    gpu_model: data.gpu_model, utilization: data.util,
    memory_pct: data.memPct, temperature: data.temp,
    timestamp: new Date().toISOString(),
  }));
}

Error Handling

Output

• A bounded-label GPU and workload dashboard with actionable alert rules.
• A redacted event trail linking an alert to the namespace, model family, severity, and response owner.
• A tested incident path for capacity, memory, latency, and node-health failures.

Examples

Use a non-production workload to verify the alert route without disrupting a live service:

kubectl -n inference-staging scale deployment/summarizer --replicas=0
kubectl -n inference-staging get pods --watch
# Confirm the unavailable-replica alert reaches the test route, then restore it.
kubectl -n inference-staging scale deployment/summarizer --replicas=1

Record the alert ID and restoration time, not request or model content. Escalate a node or memory alert through the runbook before deleting pods or changing quotas.

Resources

• CoreWeave Observability

Next Steps

For incident response, see coreweave-incident-runbook.