03 — Deployment & Management¶
Deploy target¶
| Service | Platform | URL (production) | URL (staging) |
|---|---|---|---|
billing-worker |
Cloudflare Workers (Paid plan) | https://billing-worker.adventive.com |
https://billing-worker-staging.adventive.com |
| PDF storage | Cloudflare R2 | r2://adventive-invoices/{acct_id}/{inv_uuid}.pdf |
r2://adventive-invoices-staging/... |
| Stripe webhook endpoint (prod) | Stripe Dashboard → Developers → Webhooks | https://billing-worker.adventive.com/webhook/stripe |
https://billing-worker-staging.adventive.com/webhook/stripe |
The billing Worker runs on the existing Cloudflare Workers Paid plan alongside adventive-admin-api. It is a distinct Worker with a distinct responsibility: webhook ingestion, PDF rendering, Mailgun delivery, and scheduled reconciliation. It does not serve operator UI traffic.
Environments¶
| Env | Worker name | Stripe mode | Mailgun domain | Promotion rule |
|---|---|---|---|---|
| Local dev | wrangler dev |
Stripe test mode | Mailgun sandbox | No deploy; local only |
| Staging | adventive-billing-worker-staging |
Stripe test mode | Mailgun sandbox | PR merge to staging branch triggers auto-deploy |
| Production | adventive-billing-worker |
Stripe live mode | Mailgun production | PR merge to main after staging validation |
Stripe test mode and live mode use separate webhook endpoints. A staging Stripe account or the Stripe test-mode clock is used for end-to-end invoice lifecycle testing without affecting live billing.
Build & deploy pipeline¶
GitHub Actions workflow — billing Worker¶
# .github/workflows/deploy-billing-worker.yml
name: Deploy billing Worker
on:
push:
branches: [main, staging]
jobs:
deploy:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Install dependencies
run: npm ci
- name: Run tests
run: npm test
- name: Deploy to staging
if: github.ref == 'refs/heads/staging'
run: npx wrangler deploy --env staging
env:
CLOUDFLARE_API_TOKEN: ${{ secrets.CLOUDFLARE_API_TOKEN }}
CLOUDFLARE_ACCOUNT_ID: ${{ secrets.CLOUDFLARE_ACCOUNT_ID }}
- name: Deploy to production
if: github.ref == 'refs/heads/main'
run: npx wrangler deploy --env production
env:
CLOUDFLARE_API_TOKEN: ${{ secrets.CLOUDFLARE_API_TOKEN }}
CLOUDFLARE_ACCOUNT_ID: ${{ secrets.CLOUDFLARE_ACCOUNT_ID }}
Manual deploy (hotfix or first deploy)¶
Stripe webhook registration¶
Register webhook endpoints in the Stripe Dashboard (Developers → Webhooks) or via Stripe CLI. Register both staging and production endpoints separately against their respective Stripe modes.
# Register production endpoint (Stripe CLI)
stripe listen --forward-to https://billing-worker.adventive.com/webhook/stripe
# Or register permanently in Stripe Dashboard:
# - URL: https://billing-worker.adventive.com/webhook/stripe
# - Events: invoice.finalized, invoice.paid, invoice.payment_failed,
# customer.subscription.updated, customer.subscription.deleted
The webhook signing secret (STRIPE_WEBHOOK_SECRET) is generated by Stripe when the endpoint is registered. Copy it immediately and set as a Wrangler secret.
Secrets & configuration¶
All secrets are managed via Wrangler secrets. Never commit secret values to Git.
| Secret name | Description | Owner | Rotation |
|---|---|---|---|
STRIPE_SECRET_KEY |
Stripe restricted key (billing Worker only — read/write on Invoices, Customers, Subscriptions, Usage Records) | Jeffrey | On Stripe security event or key rotation schedule |
STRIPE_WEBHOOK_SECRET |
Stripe webhook signing secret for Stripe-Signature header verification |
Jeffrey | When Stripe webhook endpoint is re-registered |
MAILGUN_API_KEY |
Mailgun API key for billing email send | Jeffrey | On Mailgun security event |
MAILGUN_DOMAIN |
Mailgun sending domain (notify.adventive.com) |
Jeffrey | If domain changes |
BILLING_BCC_ADDRESS |
HubSpot BCC address (2658568@bcc.hubspot.com) |
Jeffrey | If HubSpot CRM account changes |
R2_INVOICE_BUCKET |
R2 bucket name for PDF storage | Patrick | If bucket is renamed |
DB_HYPERDRIVE_CONNECTION |
Hyperdrive connection string (for reconciliation read of billing_invoice) |
Patrick | On DB credential rotation |
Setting / rotating secrets¶
# Set a secret (prompts for value — never pass on command line)
wrangler secret put STRIPE_SECRET_KEY --env production
wrangler secret put STRIPE_SECRET_KEY --env staging
# List current secrets (names only, not values)
wrangler secret list --env production
# Rotate: re-run the same put command with the new value
# No Worker redeploy needed — secrets are fetched per-request
wrangler.toml (non-secret configuration)¶
name = "adventive-billing-worker"
main = "src/index.ts"
compatibility_date = "2024-01-01"
[[r2_buckets]]
binding = "INVOICES"
bucket_name = "adventive-invoices"
[env.staging]
name = "adventive-billing-worker-staging"
[[env.staging.r2_buckets]]
binding = "INVOICES"
bucket_name = "adventive-invoices-staging"
[triggers]
crons = ["0 6 * * *"] # Daily reconciliation at 06:00 UTC
[env.staging.triggers]
crons = ["0 6 * * *"]
Stripe webhook registration — per-phase checklist¶
| Phase | Action | Stripe mode |
|---|---|---|
| B.1 | Register staging webhook endpoint; configure invoice.finalized listener |
Test mode |
| B.3 | Register production webhook endpoint; configure invoice.finalized, invoice.paid, invoice.payment_failed |
Live mode |
| B.5 | Add customer.subscription.updated, customer.subscription.deleted events |
Live mode |
| B.9 (optional) | Add customer.subscription.trial_will_end if trial periods used |
Live mode |
Cohort-cutover runbook¶
The migration proceeds cohort-by-cohort, one billing cycle per cohort, with zero-drift reconciliation as the gate to proceed.
Recommended cohort order¶
| Cohort | Pricing model | Estimated customers | Gate |
|---|---|---|---|
| 1 | Flat subscription only — no metered impressions | ~20–40 | One billing cycle; reconciliation drift = 0 |
| 2 | Flat subscription + tiered impressions (ST/RM/CV) | ~100–150 | One billing cycle; reconciliation drift = 0 |
| 3 | Custom arrangements / managed-services-heavy | Remaining | Manual review per account before migration |
Pre-cohort checklist (before each cohort cutover)¶
□ Stripe Subscription created for each customer in cohort
□ Flat Price configured (matches current account_plan_sub.sub_rate)
□ Metered Prices configured (match current account_plan_usage tiers)
□ Metering signal tested: POST /usage for each account in cohort;
confirmed usage records appear in Stripe subscription
□ Webhook delivery confirmed: test-mode invoice.finalized received
and processed by staging billing Worker
□ PDF rendered successfully for representative accounts in cohort
□ Mailgun test send successful: PDF attached, BCC present,
billing-noreply@notify.adventive.com as sender
□ Acodei picks up test-mode Stripe events and creates QuickBooks entries
□ Historical invoices for cohort exported/accessible (14-year retention)
□ Rollback plan confirmed: if drift found, cohort returns to legacy rollup
for the following billing cycle
Dual-run reconciliation¶
During each cohort's first billing cycle under Stripe:
1. billing_service still runs for this cohort (parallel run)
2. Billing Worker also processes invoice.finalized for the cohort
3. Daily reconciliation Cron Trigger compares: Stripe invoice total vs. legacy billing_invoice total per account
4. Acceptable drift: $0.00 (zero tolerance — any drift triggers investigation before next cycle)
5. After one clean cycle, cohort is formally migrated; billing_service stops processing that cohort
Post-cohort sign-off¶
□ Reconciliation report: zero drift for full billing cycle
□ All invoices delivered via Mailgun (check delivery logs)
□ No operator rollback requests during cycle
□ Stripe payment collection confirmed (not double-charged)
□ QuickBooks entries match Stripe invoices (Acodei confirmation)
□ Jeffrey + Patrick sign-off → proceed to next cohort
Observability¶
Logs¶
# Real-time Worker tail log (production)
wrangler tail adventive-billing-worker --env production --format pretty
# Filter for webhook events
wrangler tail adventive-billing-worker --env production --format pretty \
--search "invoice.finalized"
# Filter for errors
wrangler tail adventive-billing-worker --env production --format pretty \
--search "ERROR"
Metrics — Workers Analytics Engine¶
The billing Worker emits structured datapoints for:
| Event | Dimensions | When |
|---|---|---|
webhook_received |
event_type, idempotency_key |
On every Stripe webhook |
pdf_rendered |
account_id, inv_uuid, duration_ms |
After successful Browser Rendering |
pdf_render_failed |
account_id, inv_uuid, error |
On Browser Rendering failure |
email_sent |
account_id, template, mailgun_id |
After successful Mailgun send |
email_failed |
account_id, template, error |
On Mailgun failure |
reconcile_drift |
account_id, stripe_total, legacy_total, delta |
When drift detected |
reconcile_clean |
cohort, accounts_checked |
When daily reconciliation finds no drift |
Dashboards¶
- Workers Analytics Engine: Cloudflare Dashboard → Workers & Pages → adventive-billing-worker → Analytics
- Stripe webhook delivery: Stripe Dashboard → Developers → Webhooks → click endpoint → Event deliveries tab
- Mailgun delivery: Mailgun Dashboard → Logs → filter by domain
notify.adventive.com - Real-time tail:
wrangler tail(see above)
Alerts to configure¶
| Alert | Condition | Channel |
|---|---|---|
| Webhook delivery failure | Stripe marks endpoint as failing (>3 consecutive failures) | Stripe Dashboard email alert → Slack #alerts |
| PDF render failure | pdf_render_failed event in Analytics Engine |
Cloudflare notification → Slack #alerts |
| Reconciliation drift | reconcile_drift event with delta != 0 |
Slack #billing-alerts (direct DM to @jeffrey + @patrick) |
| Worker error rate > 1% | Workers error rate dashboard threshold | Cloudflare notification |
Rollback strategy per phase¶
| Phase | Rollback mechanism | Time to rollback | Notes |
|---|---|---|---|
| B.1–B.2 (build, test mode only) | Delete staging Worker | < 5 min | No production impact |
| B.3 (first production webhook) | Disable Stripe webhook endpoint in Dashboard → stop new events firing | < 2 min | Legacy billing_service continues unaffected |
| B.4–B.5 (PDF + metering) | Disable webhook endpoint; billing_service resumes full ownership | < 2 min | Same as B.3 |
| B.6–B.7 (cohort cutover) | Move affected cohort back to legacy billing_service for next cycle | Next billing cycle | No data migration; shared DB |
| B.8 (legacy retirement) | Rollback = recovery from DB backup + restart billing_service | Hours | CRITICAL: B.8 is the point of no return. Extra scrutiny required before retiring legacy code. |
| B.9 (Customer Portal) | Disable Customer Portal in Stripe Dashboard | < 5 min | No billing impact |
Cost model¶
| Component | Billing basis | Estimated monthly cost |
|---|---|---|
| Cloudflare Workers (billing-worker) | ~5,000 webhook requests/month + cron triggers | < $1/month on Paid plan (included in base $5) |
| Cloudflare Browser Rendering | ~200 renders/month at $5/1,000 renders | ~$1/month |
| Cloudflare R2 | ~200 PDFs × ~200KB average = 40MB storage; ~200 GET requests/month | < $1/month |
| Mailgun | Existing usage (invoice emails already sent via Mailgun) | No incremental cost |
| Stripe Billing | Stripe charges % fees on payment volume, not on Billing feature usage | No incremental SaaS fee for using Billing vs. Invoicing |
| Legacy billing_service (EC2) | Eliminated at B.8 | Cost savings — EC2 instance retirement candidate |
Total incremental cost (billing Worker + R2 + Browser Rendering): approximately $2–3/month above baseline Cloudflare Paid plan.