When an upload silently fails

When a file connects into the core but the audit trail stalls, here's how to walk the pipe end to end.

curl returned 200, the file is in /workspace/ (you can sh "ls" to confirm), but recent_uploads[0].status stays pending indefinitely. The upload worked — only the audit trail is stuck. Here's how to walk the pipe.

The path the claim takes

1. R2 fires a PutObject event into the connect0-r2-uploads Cloudflare Queue.
2. The apps/r2-events Worker consumes the message.
3. The Worker parses key → (project_id, target_path) and HMAC-signs a POST to https://api.connect0.ai/v1/internal/r2-upload-event.
4. apps/api verifies the HMAC, finds the matching upload_link row, flips claimed_at.

A pending-forever upload means one of those steps is silently failing.

Step 1: confirm R2 actually saw the bytes

sh "ls -la /workspace/<your-target-path>"

If the file is there with non-zero size, R2 wrote the object. Skip to step 2. If the file is missing, the PUT didn't actually complete — curl --upload-file might have returned 200 against a proxy that swallowed errors. Re-try.

Step 2: tail the Worker

(Operator-only — needs wrangler auth against the Cloudflare account.)

pnpm exec wrangler tail r2-events

Then trigger another small upload. Within a few seconds you should see a log line like claim_rejected or just the absence of any output.

• No output at all. R2 isn't publishing to the queue. Check the bucket event-notification rule: pnpm exec wrangler r2 bucket notification list connect0-sandbox-fs. If the rule is missing, re-create it (see the runbook).
• queue_handler_error. The Worker hit something unexpected — read the full message. Common: wrangler hasn't deployed the latest version, secret missing.
• claim_rejected { status: 401 }. HMAC mismatch — the Worker's R2_EVENTS_HMAC_SECRET and apps/api's R2_EVENTS_HMAC_SECRET (from Secret Manager) don't agree. Re-seed both with the same value.
• claim_rejected { status: 500 }. apps/api can't reach Postgres or the route is misconfigured. Check apps/api logs.

Step 3: tail apps/api

gcloud run services logs read prod-api --project=c0-p-workloads --limit=50

Look for /v1/internal/r2-upload-event entries. A healthy one returns 204. A 401 means HMAC drift. A 400 means body parse error (the Worker payload shape changed — unlikely without a recent deploy).

Step 4: query the table directly

SELECT id, target_path, issued_at, claimed_at, expires_at
FROM upload_link
WHERE project_id = '<uuid>'
ORDER BY issued_at DESC
LIMIT 5;

If claimed_at is null but bytes_received is set: somewhere a partial update is happening. Shouldn't be possible — file a bug.

Common root causes

• HMAC secret drift. Most common. After rotating the secret, you have to bind the new version to both the Worker (wrangler secret put) and apps/api (re-deploy after updating Secret Manager).
• Event rule was never created. R2 events are off by default; wrangler r2 bucket notification create is a one-time step that's easy to forget on a new deploy.
• Worker was redeployed without the secret. wrangler deploy doesn't preserve secrets across major version changes; check wrangler secret list.
• sh "tee"-written files. Files written outside an upload-link issuance still produce R2 events; apps/api can't find a matching link row and ignores them. Looks like "the event went nowhere" but is by design.

Next steps

• What recent_uploads.status means.
• Audit: who uploaded what, and when.

Where this fits

This is the Connect everything pillar of connect0 — the core that connects everything your company runs on. This is the debugging path for when the connect-in step succeeds but the core's audit ledger lags behind. Start building → · All guides →