Running a Renox app in production

Your app works on your own computer. This guide is about what comes next: keeping it running well on a real server, where real people use it, and fixing things when they go wrong.

It covers the settings for slow or broken services, the proxy in front of the app, health checks, backups, and how to recover jobs and webhooks that failed. To set up the server itself, see the deploy/README.md file that rnx make:deploy writes for you.

In this guide

• The app's commands: what you can run on the server
• Timeouts and Behind a reverse proxy
• Several servers and /health
• When a dependency fails: what the app does when something breaks
• Failed jobs and Failed webhook calls
• Backups, Deploys and migrations and Deploys without refused connections
• Sessions, Scheduled tasks and housekeeping and Maintenance mode
• All settings
• Logs, Error reports, Error pages and the Debug inspector

#Words you'll meet

#The app's commands

Your app is also its own command-line tool. my-app help lists its commands. While you develop, rnx <command> runs one through cargo run for you.

Run without a command, the app runs serve. That starts three things at once: the web server, the queue workers and the scheduler.

You can add your own commands (App::command, App::typed_command). They sit next to these, but they can't use the same names.

#Timeouts

Sometimes a service the app needs stops answering. Without a time limit, every request would wait for it, and requests would pile up until the app is stuck. With timeouts, requests fail quickly instead, and the app keeps going.

All timeouts are in seconds. You set them in .env. (Every setting, with a comment, is in the .env.example that rnx new writes; see also All settings below.)

"(fixed)" means you can't change it with a setting.

Jobs have their own limit: Job::TIMEOUT (60 s by default). A job that takes longer is stopped and tried again later.

#Size limits

Two more settings decide how big things may get:

#Behind a reverse proxy

On a server, the app doesn't usually face the internet by itself. Instead:

• the app listens only on 127.0.0.1 (APP_HOST=127.0.0.1), which means "this machine only";
• a reverse proxy such as Caddy or nginx faces the internet. It handles TLS (the https:// part) and passes each request on to the app.

Here is a whole Caddy setup for that:

# Caddyfile
shop.example.com {
    reverse_proxy 127.0.0.1:3000
}

It says: requests for shop.example.com go to the app on port 3000 of this machine. Caddy also gets and renews the HTTPS certificate by itself.

#TRUSTED_PROXIES: the visitor's real address

Behind a proxy, every connection seems to come from the proxy's address. The proxy writes the visitor's real address in a header called X-Forwarded-For (or the standard Forwarded header; the app reads X-Forwarded-For first, then Forwarded). Set TRUSTED_PROXIES so the app believes that header when it comes from your proxy:

TRUSTED_PROXIES=127.0.0.1          # Caddy or nginx on the same machine
TRUSTED_PROXIES=10.0.0.0/8         # a load balancer on a private network
TRUSTED_PROXIES=*                  # whoever connects (a platform whose proxy IPs you can't list)

Pick the one line that matches your setup. More about the value:

• it is a comma-separated list of addresses and CIDR ranges (TRUSTED_PROXIES=127.0.0.1,10.0.0.0/8);
• with a list, the app walks the forwarded addresses from the nearest hop back, skipping the proxies you listed, and takes the first address that isn't one of them;
• * trusts only the last hop: the address the connecting proxy added;
• * together with addresses (*,10.0.0.1), a bad address or a bad prefix stops the app at boot with an error.

Without this setting:

• Routes::throttle and named limiters (throttle_by) count every guest as one person;
• the login lock counts per email only;
• logs show the proxy's address instead of the visitor's.

In a handler, ClientIp gives you the same address the app uses.

#TRUSTED_HOSTS: the names your app answers to

Set TRUSTED_HOSTS to the host names the app serves (example.com,*.example.com). Requests for any other Host then get a 400 ("bad request").

Why it matters: links in mails, like a password reset link, are built from the host of the request. This setting stops an attacker from making your app send a mail whose link points to their own site.

Two exceptions:

• APP_URL's host is always allowed;
• /health answers for any host, because load balancers check it by IP address.

#Headers the proxy must keep

Routes for other hosts (Routes::domain("admin.example.com", …)) are picked by the Host header, so the proxy must pass it on. Caddy does. With nginx, add proxy_set_header Host $host.

An app with App::detect_locale() (which picks the language from the browser) answers with Vary: Accept-Language. A CDN or caching proxy in front must keep that header. Otherwise it could show one visitor's language to another.

#The notification bell's stream

The notification bell (Auth::new().notifications()) keeps a Server-Sent Events stream open on each page: a request that stays open so the server can push news to the browser. It lives at /notifications/stream. Each stream lasts at most five minutes; then the browser opens a new one.

Caddy passes it through as it is. nginx buffers responses (it waits to collect them before sending), which would hold the news back. Turn that off for this address:

location /notifications/stream {
    proxy_pass http://127.0.0.1:3000;
    proxy_buffering off;
    proxy_read_timeout 10m;
}

This sends /notifications/stream to the app without buffering, and lets a stream stay open for up to ten minutes.

Each open stream asks the database for news every 15 seconds (two small queries for its user). It also checks at once when something changes in the same process. When the app shuts down gracefully, it ends them all.

The app's own live events (state.broadcast, see mail.md) use the same streams but aren't stored: they reach only the pages connected to the process that sends them. With several servers, or with workers in a separate queue:work process, pages connected elsewhere miss them. Keep broadcasts for "look again" hints, and store a database notification for anything someone must see.

#APP_URL

Also set APP_URL to the public https:// address, so links in mails point to the right place.

An https:// APP_URL does two more things:

• it marks cookies Secure, so browsers send them over HTTPS only: the session, the app's SetCookies and the maintenance bypass;
• with APP_ENV=production as well, every response sends HSTS (Strict-Transport-Security), which tells browsers to always use HTTPS for your site.

#Several servers

When one server isn't enough, several app servers can share one PostgreSQL database. Renox makes sure they don't get in each other's way:

• the queue hands each job to one worker only (FOR UPDATE SKIP LOCKED);
• each scheduled run is claimed once, so it doesn't run on every server;
• migrations take turns.

Set CACHE_STORE=database as well. Then these live in the cache table, so they work across servers: the cache, Routes::throttle and throttle_by limits, the login lock and cache locks (state.cache.lock).

Sessions live in their encrypted cookie, or in the sessions table with SESSION_DRIVER=database (see Sessions). Either way, any server can answer any request. You don't need "sticky sessions" (sending a visitor always to the same server).

Uploaded files must be on shared storage (STORAGE_DISK=s3, see File storage on S3), or on the one server that has the disk.

#File storage on S3

S3 storage needs renox's s3 feature (features = ["s3"] in Cargo.toml). It works with AWS S3 and with services that speak the same language, such as Cloudflare R2 or MinIO. The settings:

A disk added with App::disk("backups", |config| StorageConfig::from_env(config, "BACKUPS")) reads its own settings, starting with the prefix you give:

• BACKUPS_DISK (local or s3, default local), BACKUPS_PATH (a local disk's folder, default STORAGE_PATH/backups), BACKUPS_BUCKET and BACKUPS_URL;
• BACKUPS_REGION, BACKUPS_ENDPOINT, BACKUPS_ACCESS_KEY_ID and BACKUPS_SECRET_ACCESS_KEY, which fall back to the S3_* settings when unset.

#/health

GET /health is a small address for machines that check on your app: load balancers, uptime monitors and container health checks.

• 200 with "status": "ok" when the database answered within 2 seconds.
• 503 with "status": "error", and the reason in "database", when it didn't.

The JSON answer also has:

• queue.pending and queue.failed: how many jobs are waiting and how many failed. Set up an alert for when failed grows.
• maintenance: whether the app is in maintenance mode.

Maintenance mode doesn't change the status code. That's on purpose: the load balancer keeps sending visitors to the app, and they see the maintenance page.

#When a dependency fails

What does the app do when something it needs breaks? The table below answers that.

How each row is checked:

• the database and panic rows are checked on every change by the chaos test (tests/chaos/run.sh, run in CI on SQLite and PostgreSQL). It breaks things on purpose and watches what happens;
• the SMTP and killed-process rows are covered by the integration tests (crates/renox/tests/it/background_resilience.rs);
• the SIGTERM row is how serve shuts down, and has no automated test yet.

A panic is when Rust code crashes in the middle of running.

#Failed jobs

There are two ways to look after failed jobs:

• in the browser: /_renox/queue (the renox::queue::Dashboard module, behind the view-queue-dashboard gate) shows the queue and the failed jobs, with retry and forget buttons;
• in a shell: the commands below do the same.

For monitoring, state.queue.stats() has the numbers: ready jobs, the oldest wait, and jobs done and failed in the last hour.

#When a job counts as failed

A job is tried several times (Job::MAX_ATTEMPTS, 3 by default). If every attempt fails, it moves to the failed_jobs table, with its error.

Some jobs go there after the first attempt, because trying again can't help:

• a job whose error was made with Error::permanent;
• a job whose saved data (its payload) can no longer be read.

#The commands

my-app queue:failed          # list them, with their errors
my-app queue:retry 12        # put one back on the queue with fresh attempts
my-app queue:retry all
my-app queue:flush           # delete them all
my-app queue:forget 12       # delete one
my-app queue:prune-failed --hours 168    # delete those older than a week (the default)
my-app queue:prune-batches --hours 24    # delete finished batches

What a retry does:

• queue:retry runs the same payload again;
• a job that was part of a chain carries on with the chain;
• a job that was part of a batch counts in its batch again.

A job's failed hook runs once, when the job fails for good (not again on queue:retry).

To delete old failed jobs on a schedule, call the functions behind the prune commands from a task (see Scheduled tasks and housekeeping).

#Workers in their own process

By default, serve runs the workers too. To run them in a separate process, start my-app queue:work and set QUEUE_WORKERS=0 for serve.

queue:work takes these options:

• --queue a,b: only these queues, and the first one listed is emptied first. Without it, every queue, in the order jobs arrived;
• --workers N: how many jobs run at once (1 by default);
• --once: run what is waiting now, then exit.

See queue.md.

#Failed webhook calls

Every webhook call your app accepts is saved in the webhook_calls table before it is handled. The provider got its 200 ("OK"), so it won't send the call again. If handling it fails, it's up to you to try again.

my-app webhook:failed        # calls whose handler failed, with the error
my-app webhook:retry 7       # process one again, e.g. after a fix

Each call has a status:

• received until its handler is done;
• processed when the handler succeeded;
• failed when it failed. A handler that panics also marks its call failed.

The handler runs as a queued job with five attempts. Each new attempt waits 30 s times the attempt number. So a failed call may still succeed on a later attempt. After the fifth, it stays failed until you run webhook:retry.

The saved payload is the exact body the provider sent, so its signature can be checked again.

The renox-billing crate's webhooks (Stripe, Xendit) are saved under the provider billing, with event ids like stripe:evt_…. Retrying one is safe: applying an event twice changes nothing, and an event older than what a subscription already shows is ignored.

#Backups

A backup is only useful if it has everything you need to start again. Here is what to keep.

SQLite. The database is storage/app.db. While the app runs, it also has two helper files next to it: -wal and -shm.

• For a continuous backup (copied all the time, as it changes), use Litestream (see deploy/README.md).
• For a one-off copy while the app runs: sqlite3 storage/app.db ".backup backup.db".

PostgreSQL. Use your provider's backups or pg_dump.

Files. Uploads live in STORAGE_PATH/app (or the S3 bucket). Each disk added with App::disk lives in STORAGE_PATH/<name> (or its own bucket). Back those directories up too.

Keys. Keep .env's APP_KEY with the backups. It's the app's secret key, and a lot depends on it. Without it:

• sessions end, so everyone is logged out;
• signed links stop working (signed_url, the local disk's temporary_urls, email verification links);
• the app's encrypted cookies (SetCookie::encrypted, read with Cookies::get_encrypted) read as missing;
• maintenance bypass cookies stop letting people through.

#Deploys and migrations

When you deploy, you usually run migrate first. Here is how it behaves:

• migrate runs all pending migrations as one batch (a group it can undo together). Several servers can run it during the same deploy: they take turns, and the later ones find nothing left to do.
• Each migration runs in a transaction, so a failed one leaves nothing behind. Migrations with CREATE INDEX CONCURRENTLY or a -- renox:no-transaction line run without one. Keep those to a single change.
• On PostgreSQL, DATABASE_STATEMENT_TIMEOUT (30 s) applies to migrations too. For a slow one, run DATABASE_STATEMENT_TIMEOUT=0 my-app migrate.
• migrate:status shows each migration's batch. It also flags migrations that already ran but whose file was edited or deleted since.

• Restart the app after migrate. (The Dockerfile and deploy/README.md already do: migrate, then start.) Migrations run inside the app itself (tests, migrate:fresh) are safe: connections opened before them are dropped. But a server that keeps running while another process migrates may keep connections that read the old schema, until it restarts.
• migrate:rollback undoes the last batch only if every migration in it has a .down.sql. Otherwise it undoes nothing.

#Deploys without refused connections

#The problem

A restart has a short gap. The old process stops (after finishing its running jobs), and the new one isn't listening yet. During that gap, the port is closed, and visitors get "connection refused".

#The fix: systemd socket activation

With socket activation, systemd holds the port instead of the app. While the app restarts, systemd keeps new connections waiting in a line. So a deploy only makes visitors wait a moment; nobody is turned away.

rnx make:deploy writes the file for this, deploy/<app>.socket. To turn it on:

sudo cp deploy/shop.socket /etc/systemd/system/ && sudo systemctl daemon-reload
sudo systemctl stop shop                  # it holds the port; the socket takes it over
sudo systemctl enable --now shop.socket
sudo systemctl start shop
sudo systemctl restart shop               # each deploy: migrate, then take over the socket

Step by step: copy the socket file where systemd looks and reload it; stop the app; switch on the socket (now and at every boot); start the app again. The last line is what you run at each deploy.

The app then uses the socket systemd passes to it (LISTEN_FDS) instead of opening APP_HOST:APP_PORT itself. Keep the socket's ListenStream at that same address.

Does it help? Measured on a laptop with Renox's hello example, sending requests one after another through three restarts: without the socket, 280 of 1,288 requests were refused; with it, all 1,350 got an answer.

#Migrations both versions accept

During a deploy, the old code runs against the new migrations for a moment (and for longer with several servers). So write migrations that both the old and the new version accept:

• add columns as nullable or with a default. Fill them in (backfill) later, in a job or a command;
• rename or drop a column over two deploys: first stop reading it, then drop it;
• add an index with CREATE INDEX CONCURRENTLY on PostgreSQL (see above).

#When a restart must not pause at all

Sometimes even a short wait is too much (a slow start, a cache that needs to warm up). Then run two copies of the app on two ports behind Caddy, and restart them one at a time. deploy/README.md has the Caddyfile for it (lb_policy first with health_uri /health).

The two copies can share a SQLite file on one machine. Set CACHE_STORE=database (and SESSION_DRIVER=database for large sessions), so both see the same rate limits, locks and sessions.

#Sessions

A session is what the app remembers about a visitor between pages: that they're logged in, flashed messages, old form input and the CSRF token (which protects forms).

#The default: in the cookie

By default, the whole session is encrypted into its cookie. Good: there's nothing to store on the server. But:

• browsers drop cookies over about 4 KB;
• a copied cookie stays valid until it expires or the user logs out (Renox then refuses it).

#SESSION_DRIVER=database

SESSION_DRIVER=database keeps sessions in the sessions table, and only an id in the cookie:

• there's no size limit (a large cart, long old input);
• each login and logout gives the session a new id and deletes the old row, so a copied cookie stops working at once;
• the table has user_id, so you can, say, show users their sessions or end them from an admin page;
• rows are keyed by the id's SHA-256 (a fingerprint of it), so someone with a copy of the table alone can't take over a session.

Each request reads its row. It writes the row only when the session changed (or once a minute, to keep it from expiring).

Switching from cookie to database logs nobody out: a cookie written by the cookie driver is read and moved into the table. Switching back, from database to cookie, ends every session: the cookie then holds only an id, which the cookie driver can't use, so everyone has to log in again.

Expired rows are deleted now and then by requests, and by my-app session:prune (Session::prune_expired(&db)).

In tests (APP_ENV=testing), database sessions are kept in memory instead of the sessions table, so TestApp's session helpers keep working.

#Scheduled tasks and housekeeping

The scheduler runs tasks at set times, like a timer. It runs inside serve (or alone with schedule:work).

Its times follow APP_TIMEZONE. That can be UTC, a fixed offset like +07:00, or an IANA name like Asia/Jakarta (with daylight saving handled). A task can also set its own timezone.

my-app schedule:list          # each task's next run, in its time zone
my-app schedule:run backup    # run one task now, e.g. to check it after a deploy

#Tables that keep growing

Some tables grow until something cleans them up (prunes them). You can run the commands by hand, or call the functions behind them from a scheduled task. (A task runs Rust code, not commands, so it calls the function.)

Each function returns how many rows it deleted. Where the command takes --hours or --days, the function takes a std::time::Duration.

For example, a daily task at 3 in the morning that deletes failed jobs older than a week: s.daily_at("03:00", "prune-failed", |state| async move { state.queue.prune_failed(Duration::from_secs(7 * 86_400)).await.map(drop) }).

#Maintenance mode

Maintenance mode shows visitors a "back soon" page while you work on the app.

my-app down --secret let-me-in --retry 60   # 503 page with Retry-After; /let-me-in lets you through
my-app up

• down turns it on. Visitors get a 503 page. --retry 60 tells browsers and bots to try again in 60 seconds (the Retry-After header). --secret let-me-in means that opening /let-me-in lets you through, so you can check the site.
• up turns it off.

The switch is a small file in STORAGE_PATH, so every process that shares that directory sees it. /health and every webhook route keep working, so payment providers' calls are still saved.

#All settings

Every setting is read from the environment or from .env, when the app starts. The full list, each with a comment and its default, is the .env.example that rnx new writes (its source is crates/renox-cli/stubs/env.stub). This guide covers the ones for running in production. The others, in short:

Your app's own settings need no code in Renox: state.config.var("NAME") reads any of them.

#Logs

Logs are the app's diary: what it did, and what went wrong.

Where they go. Logs go to stdout (the terminal's normal output).

How much. Set the level with RUST_LOG (for example RUST_LOG=info,sqlx=warn). Without it:

• serve, queue:work and schedule:work log at info (info,renox=debug with APP_DEBUG);
• every other command logs only warnings.

JSON logs. LOG_FORMAT=json writes one JSON object per line. Log tools like Loki, Datadog, CloudWatch or jq read that easily. The request's fields are in span:

{"timestamp":"…","level":"ERROR","fields":{"message":"request failed","error":"…"},
 "target":"renox_core::error","span":{"id":"k3J9x2mQpL0aB7cDe4Fg","ip":"203.0.113.9","method":"POST","uri":"/checkout","name":"request"}}

This line says: a POST to /checkout from 203.0.113.9 failed, and its request id is k3J9x2mQpL0aB7cDe4Fg.

Log to a file. LOG_FILE=storage/logs/app.log appends to that file instead (without colors). If the file can't be opened (a missing directory, no permission), the app says so on stderr and logs to stdout. Rotate the file with logrotate's copytruncate, or leave logs on stdout for journald or Docker to collect.

Request ids. Each request gets an id:

• the X-Request-Id a proxy sent, kept when it's 8 to 64 characters of A-Z a-z 0-9 . _ -;
• or a new one.

The id is in every log line of the request, in the response's X-Request-Id header, in error reports, and in handlers as the RequestId extractor. A visitor who tells you the id points you at the exact log lines.

Errors and jobs.

• Errors are logged with their cause. With APP_DEBUG=false, visitors see only the error page.
• Job attempts log job done, job failed, will retry and job failed for good.

#Error reports

Logs only help if someone reads them. App::report sends you every error a person should look at. It runs in the background, after the response has been sent, for:

• a request that answered 500 (with its method, path, request id, IP and user id);
• a job that failed for good (source is its name and id, like send-invoice #42);
• a scheduled task that failed or panicked.

use renox::report::ErrorReport;

App::new().report(|report: ErrorReport, state: AppState| async move {
    // Sentry, Honeybadger, a Slack webhook… The report serializes to JSON.
    // Here: if the app has an ERROR_WEBHOOK setting, post the report there as JSON.
    if let Some(url) = state.config.var("ERROR_WEBHOOK") {
        // The `let _ =` ignores a failed send: a broken reporter must not cause new errors.
        let _ = state.http.post(url).json(&report).send().await;
    }
})

This reporter reads an ERROR_WEBHOOK address from the app's settings, and posts each report there as JSON.

You can add several reporters. One that fails or panics doesn't stop the others. 4xx errors (a 404, a failed validation) aren't reported: those are the visitor's mistakes, not the app's.

#Error pages

When something goes wrong, visitors see an error page. rnx new writes resources/views/errors/default.html for you. It extends the app's layout, so an error page keeps the navigation bar and the signed-in user's menu.

The template gets:

• the same globals as any page (auth, request, t(), route()…);
• status, reason and detail. detail is the message of an abort(…); for a 500, it's the cause, but only with APP_DEBUG;
• with APP_DEBUG only: debug, request_line (the request that failed) and template (where a template failed), for a developer box on the page.

Want a special page for one status? Write errors/404.html (or any status); it wins for that status.

If the app's error page itself fails to render, Renox shows its own. So a mistake in the layout can't hide the original error. Clients that ask for JSON get JSON.

#Debug inspector

While you develop, Renox can show you what each request did. With APP_DEBUG=true and APP_ENV=local, /_renox/debug lists the last 50 requests (newest first), with:

• their status and time;
• the view they rendered;
• their SQL.

A statement run three or more times in one request is flagged as a likely N+1 (a query run once per row in a loop, instead of once for all rows).

The page is never there in production or in tests.

Mail sent while you develop is at /_renox/mail.