Observability and Debugging
How to see what production is doing. Sentry is the primary error surface, Dokploy and the NestJS process hold the logs, and the reliability layer doubles as a set of debugging signals.
This page is for engineers and ops staff debugging the live SEKEN platform. It covers where errors and logs surface, how production health is monitored, the required Sentry environment variables, and how the three reliability patterns (IdempotencyGuard, OutboxProcessor, CircuitBreaker) tell you what is happening when something looks wrong.
Sentry across the three runtimes​
Sentry is wired and live across all three SEKEN runtimes. Errors are PII-scrubbed at the source before any event leaves the app: authorization tokens, session tokens, KTP and NIK values, phone numbers, email (partially masked), and URL query params are stripped before the envelope is sent.
| Surface | Integration | DSN env var |
|---|---|---|
| React web | @sentry/react, initialised in src/lib/sentry.ts | VITE_SENTRY_DSN (set in Dokploy frontend env) |
| NestJS backend | @sentry/node, initialised at app bootstrap | SENTRY_DSN_SERVER (set in Dokploy backend env) |
| Flutter mobile | sentry_flutter, initialised in mobile/lib/core/observability/sentry_setup.dart | SENTRY_DSN_MOBILE (passed via --dart-define at build time) |
All three report into one Sentry organisation, one project per surface: javascript-react for web, node (or nestjs) for backend, flutter for mobile. The organisation is on Sentry's EU region, which is why the web CSP must allow the *.ingest.sentry.io wildcard (not only the bare ingest.sentry.io) for envelope POSTs to leave the browser.
Canonical source: the Observability section of wiki/products/marketplace-status.md in the brain.
Reading Sentry​
- Web errors land in the
javascript-reactproject. Edge errors land indeno. Mobile crashes land influtter, split byos.nameso iOS and Android do not cross-contaminate. - Test triggers. On web, append
?sentry_test=1to any page URL to throw a verification exception and confirm the pipeline end to end. On the backend,GET /__sentry-testfires a test event and is gated by a token in staging only. - Release tagging. Events carry the deploy git SHA as their release (web via
@sentry/vite-plugin, backend via the CI deploy workflow). If you seerelease: unknownon a backend event, the SHA did not propagate on that deploy and release tracking will be broken for it.
Backend: why caught errors were once invisible (PS-133)​
This issue was diagnosed when the backend ran as Deno Edge Functions: caught errors in Hono handlers were logged locally but never reached Sentry because only uncaught exceptions hit app.onError(). PS-133 (2026-05-02) instrumented roughly 700 catch blocks.
The NestJS backend uses @sentry/node's exception filter integration, which captures both caught and uncaught exceptions. If SENTRY_DSN_SERVER is unset at startup, the SDK logs a warning and error reporting becomes a no-op — an empty Sentry project usually means the DSN env var is missing, not that nothing went wrong. See the Error Log for worked examples of backend Sentry issues driving fixes.
Mobile: compile-time DSN gotcha​
Mobile reads its DSN via String.fromEnvironment('SENTRY_DSN_MOBILE'), which is a compile-time constant. It must be passed with --dart-define at build time. A value placed in a runtime dotenv file is ignored, and a missing build flag makes the SDK boot in disabled mode silently, so a release build can ship with zero crash reporting and look fine. If Sentry init itself fails, reportInitFailureDirect() sends a direct HTTP envelope so even the init failure is visible. Symbol upload (iOS dSYM, Android ProGuard) runs at build time; the Android ProGuard mapping upload has a known caveat tracked in the brain as PS-75.
The web DSN is a required environment variable​
The web Sentry DSN is a required, pre-launch-critical environment variable. It is tracked in the launch blockers and the architecture map under the label SENTRY_DSN_WEB. The concrete variable is VITE_SENTRY_DSN, set in Dokploy frontend env and wired to production in PS-36 together with the CSP allowlist fix. For the full list of Sentry DSN names and where each is set, see Environment Variables.
Where logs live and how to read them​
Sentry is for exceptions. Everything else lives in the platform's own logs.
| Log | What is in it | Where to read it |
|---|---|---|
| Sentry | Exceptions and error events, all three surfaces | Sentry dashboard, one project per surface |
| NestJS backend logs | Structured JSON log output (logger.log, logger.warn, logger.error). This is where non-exception diagnostics live. | Dokploy → backend container → Logs tab |
| Frontend container logs | nginx access and error logs | Dokploy → frontend container → Logs tab |
webhook_events table | Every inbound webhook with its HMAC result and idempotency outcome | Database, see Integrations |
fin_audit_log and outbox_events tables | Finance state changes, and queued or failed side effects | Database |
During an incident, on-call reads Sentry and the Dokploy container logs together rather than any one in isolation.
Health and uptime monitoring​
Ownership: Ihsan owns knowing production is broken. Rez is informed, not paged, except for payout-flow incidents.
There is no separate uptime pinger today. Health is read from Sentry error volume, Dokploy container logs, and the backend /health endpoint. The current monitor coverage, and its gaps, are documented in workflow/eos-design-2026-04-29.md section 5.5 in the brain:
| Surface | Captured by | Alerting today |
|---|---|---|
| Failed payments | Sentry | No rate-based alert yet |
| Failed payouts | Manual ops review | No automated monitor |
| Auth failures | Sentry | No spike alert |
| Messaging failures | NestJS logger | No external alert |
| Listing publish failures | Sentry | No alert |
| Webhook failures (Xendit, Biteship) | Sentry, webhook_events | No rate-based alert |
| Backend 5xx | Sentry | No alert |
Target state: P0 errors (unhandled exceptions, payment failures) route to an on-call Discord channel that pages Ihsan within 5 minutes, with Rez receiving a daily digest. Rate-based alerting (for example, error rate at or above 3x the 7-day baseline) and the Sentry-to-Discord bridge are documented follow-ups, not yet fully built. Treat the "Alerting today" column as the honest current state.
The reliability layer as a debugging aid​
Three production-hardened patterns wrap every transaction-touching path. Their mechanics are documented in System Architecture and Integrations. This section is about what each one is telling you when something looks wrong. All three live in the NestJS backend under backend/src/infrastructure/reliability/.
IdempotencyGuard: duplicate suppression​
Dedupes retried operations (order creation, all webhooks, all outbound payment calls) by key. When you suspect a duplicate, two invoices or a doubled webhook, a returned cached result instead of a fresh execution is expected behaviour, not a bug: the guard replays the original outcome for a key it has already seen. The inverse failure is a legitimate retry being silently swallowed. If that happens, check whether the idempotency key is too coarse for the operation.
OutboxProcessor: delayed or missing side effects​
Side effects (notifications, fin_audit_log entries, downstream events) are written to the outbox_events table inside the same transaction as the business logic, then a cron drains pending rows every 15 minutes. When a notification or email did not arrive, the database write can succeed while delivery lags or fails, so check outbox_events for pending or repeatedly-failing rows. A side effect that never fired is usually a stuck outbox row, not lost data, and it will retry on the next drain.
CircuitBreaker: fast failures when a partner is down​
Wraps every external call (Xendit, Biteship, Resend, OpenAI, Xero). After a threshold of consecutive failures it opens and fails fast for a cooldown window instead of hanging on timeouts. When a partner integration returns fast, clear errors rather than slow timeouts, suspect an open breaker: the partner is likely down and the breaker is protecting the rest of the system. Confirm the partner's status before treating it as a SEKEN bug. See Integrations for per-adapter thresholds.
webhook_events: the webhook audit trail​
Every inbound webhook is recorded with its HMAC verification result and idempotency outcome. A failed HMAC is rejected and logged. An amount mismatch never auto-progresses: the order is routed to PAYMENT_REVIEW instead. Start here when a payment or shipping webhook did not have the effect you expected.
Rollback and incidents​
When observability shows a regression, the response paths are owned upstream in the brain and summarised here.
Rollback triggers (from workflow/eos-design-2026-04-29.md section 5.3). Any one of: Sentry error rate at or above 5x the 7-day baseline within 30 minutes of a deploy, any payment, payout, or escrow flow returning 5xx for 5 or more consecutive requests, an observed RLS bypass, or either Rez or Ihsan calling a rollback for any reason. Fewer than 15 minutes of degradation is monitored. More than 15 minutes is an incident.
Execution (summary). Revert the merge commit with git revert -m 1 <merge-commit-sha> and push to main, which triggers the redeploy. The full procedure, including schema-migration handling, lives in the brain runbook workflow/runbooks/rollback.md.
Incident learning. Every incident that crosses the threshold gets a durable root-cause record in the brain at workflow/incidents/INC-*.md, plus a detection-gap analysis asking why no monitor caught it sooner. The public record of significant fixes is the Error Log. Step-by-step incident-response procedures are in the Maintenance Guide.
Where to look​
| What you want | Where it lives |
|---|---|
| The Sentry DSN env vars and where each is set | Environment Variables |
| Past incidents and worked Sentry examples | Error Log |
| Reliability-layer mechanics and the webhook pipeline | System Architecture, Integrations |
| Incident-response procedures | Maintenance Guide |
| How production signals feed the autonomy pipeline | Autonomous Engineering |
Next: Changelog