How NEXUS AI keeps your Postgres alive across host reboots
Five things can fail on host reboot and silently take your data offline. This is the engineering deep dive on how NEXUS AI fixes all five, with code pointers.
By Platform Super Admin • • AI Deployments
# How NEXUS AI keeps your Postgres alive across host reboots **Published:** May 17, 2026 **Category:** Engineering · Reliability · Postgres **Reading time:** 10 minutes **Author:** NEXUS AI Team --- "What happens to my data if the host reboots?" is the question every production platform owes a real answer to. Most answer it with marketing copy. This post is the engineering answer for NEXUS AI, with the actual mechanics. If you run stateful apps (Postgres, MySQL, Mongo, Redis, persistent volumes, S3 buckets) on a single-host Docker platform, five things can fail on reboot and silently take your data offline. The combination of those five failures was the reason most Docker-based deploy platforms historically said "use cloud DBs instead." NEXUS AI fixes all five. For the broader story, see [Your AI app is generated. Now how do you deploy it?](https://nexusai.run/blog/your-ai-app-is-generated-now-how-do-you-deploy-it). This post is the deep dive on the reliability layer. --- ## The reboot problem in one paragraph When the host reboots, the Docker daemon comes back, and containers with `restart: unless-stopped` should come back too. In practice, they often do not, because the orchestrator (the thing that originally ran `docker compose up`) has lost its on-disk project context (compose file, build context, environment variables). When a user then clicks "Stop" and "Start", most platforms rebuild from scratch and lose the wiring to the database, buckets, and volumes that were attached to the original deployment. The DB row says `RUNNING`. Reality says otherwise. Data does not survive. The fix is to make the orchestrator survive reboots too, and to make "start" a real start, not a rebuild. --- ## The five-layer failure most platforms have ### Layer 1: ephemeral compose workdirs A Docker-based platform generates a `docker-compose.yml` per deployment, writes it to a workdir on disk, and runs `docker compose up -d` from that directory. The most common default is to put the workdir under `/tmp/`. On most Linux distros `/tmp` is wiped on reboot (tmpfs by default on Ubuntu 22.04+, systemd-tmpfiles on others). The workdir is gone. The platform now has no way to drive `docker compose stop`, `start`, `up`, or `down` against the original project. ### Layer 2: systemd PrivateTmp wipes /tmp on every backend restart Production deployments of the platform's backend usually run as a systemd unit with hardening: `ProtectSystem=strict`, `ProtectHome=true`, `PrivateTmp=true`. The `PrivateTmp=true` directive gives the service its own private `/tmp` namespace. Every backend restart (a code update, a crash, a reboot) gets a fresh empty `/tmp`. Workdirs are wiped not just on host reboot, but on every backend restart. ### Layer 3: stop = destroy Most platforms implement "Stop" as `docker compose down`, which removes containers, networks, and (with `-v`) volumes. Or as `docker stop` plus `docker rm`. After "Stop" there is no container left to start. ### Layer 4: start = rebuild Because Stop removed everything, Start is forced to rebuild the image, recreate the container, reattach networks, remount volumes, and reinject environment variables. If the build path does not know about all the things the original deploy attached (the bucket env vars, the persistent volume mounts, the linked DB services), Start brings the app container back without them. The app starts. The database it depended on is gone. Worse, the app's first write to its now-missing Postgres dependency takes down the whole stack. ### Layer 5: no boot reconciliation When the backend comes back online after a reboot, it does not check whether the deployments the DB believes are `RUNNING` are actually running. Containers that failed to restart (network ordering issues, transient daemon errors, manual `docker rm`) stay dead until a human notices and clicks "Start", which then triggers layer 4. Each layer alone is recoverable. Together they mean a host reboot, a backend restart, or a user clicking "Stop, Start" can wipe a Postgres deployment's working state. --- ## How NEXUS AI fixes each layer ### Fix 1: persistent compose workdirs Compose workdirs live at `/var/lib/nexus/deployments/<deployment-id>/`, never in `/tmp`. The backend's systemd unit declares: ```ini StateDirectory=nexus/deployments nexus/docker StateDirectoryMode=0750 ``` systemd auto-creates `/var/lib/nexus/deployments` and `/var/lib/nexus/docker` on first start and chowns them to the `nexus` user. The directory survives host reboots, backend restarts, and `systemd-tmpfiles` runs. The base path is overridable via `NEXUS_DEPLOYMENTS_DIR` for non-standard installs. ### Fix 2: PrivateTmp does not matter Because workdirs are not in `/tmp`, `PrivateTmp=true` no longer wipes them. The backend keeps `PrivateTmp=true` on for security (process isolation), but the workdirs live where they need to. `ReadWritePaths=/var/lib/nexus` is added to the unit so the otherwise read-only filesystem allows writes to the persistent directory. ### Fix 3: stop is a real stop `stopContainer` has two modes: - **Soft stop (default).** `docker compose stop` for compose deployments or `container.stop()` for single containers. Containers are preserved. Networks are preserved. Volumes are preserved. The workdir is preserved. The port reservation is preserved. Status flips to `STOPPED`. `destroyedAt` stays null. - **Hard destroy (`options.removeVolumes: true`).** `docker compose down -v` plus network cleanup, volume cleanup, workdir deletion, port release. This is the path the Delete endpoint and auto-destroy job take. The user-facing "Stop" button uses the soft path. "Delete" uses the hard path. Both are tested in production. ### Fix 4: start is a real start `startContainer` does the minimum work needed to bring an existing stack back online: - **For compose deployments:** 1. Ensure the external Traefik network exists (idempotent). 2. Find the workdir. If missing, rehydrate it from the DB. 3. Run `docker compose up -d` (without `--build`). The cached image `deploy-<id>:latest` is reused. External volumes reattach. Networks reattach. Environment variables are read from the persisted `.env`. 4. Refresh container IDs in the DB (compose may have recreated stopped containers with new IDs). - **For single-container deployments:** 1. `container.start()` on the existing container ID. 2. If the container is gone (HTTP 404 from the Docker daemon), fall back to a full rebuild. The rehydration step (point 2 above) is the load-bearing part. It writes `docker-compose.yml` from `deployment.composeFile` in the DB, writes `Dockerfile` from `deployment.dockerfile`, and reconstructs the `.env` file from `deployment.port` plus the `hostPort` values stored on each `DeploymentService` row. No source code is needed because the cached image is reused. ### Fix 5: boot reconciliation On backend startup (five seconds after the API begins serving), the backend runs: ```ts dockerService.reconcileRunningDeployments() ``` That method: 1. Queries every deployment with status `RUNNING` or `BUILDING` and provider `LOCAL_DOCKER`. 2. For each, calls `docker.inspect()` on the recorded `containerId`. 3. If the container is not running (or returns 404), classifies it as needing recovery. 4. For compose deployments, calls the same `startComposeDeployment` helper used by the user-facing Start endpoint (rehydrate workdir if needed, `docker compose up -d`). 5. For single-container deployments, calls `container.start()`, falling back to rebuild on 404. 6. Logs `checked=N recovered=N failed=N` so an operator can see what the platform did on startup. The job runs non-blocking so the API serves traffic immediately even if Docker is slow to respond on a freshly booted host. Failures on individual deployments are logged but do not block reconciliation of the rest. --- ## What survives a host reboot After a clean reboot of the host running NEXUS AI: | Resource | Survives | How | | ----------------------------------- | -------- | ---------------------------------------------------------------- | | App container | ✓ | `restart: unless-stopped` on the compose service | | Postgres / MySQL / Mongo / Redis | ✓ | Same restart policy, plus per-deployment data volume | | Database data | ✓ | Persistent Docker volumes (`postgres-data-<id>`, etc.) | | Org-scoped volumes | ✓ | Named Docker volumes (`nexus-vol-<id>`), declared external | | S3 bucket data | ✓ | MinIO server with its own volume; bucket data on disk | | Bucket credentials | ✓ | Persisted in DB (encrypted), reinjected on container start | | Public URL | ✓ | Traefik external network preserved; subdomain stored in DB | | Logs (last 24h) | ✓ | Docker logging driver writes to host disk | | Audit log | ✓ | Postgres-backed | | Deployment status accuracy | ✓ | Boot reconciliation reconciles DB state with Docker state | What does not automatically survive a reboot, by design: - Logs older than your retention window (configurable per plan). - Backup files stored only in `/tmp/nexus-backups/` (they are copied to persistent storage by the backup job; the temporary working copy may go). --- ## What survives a manual Stop then Start After a user clicks Stop, then Start (the operation that historically broke most platforms): | Resource | Survives | How | | ----------------------------------- | -------- | ---------------------------------------------------------------- | | App container | ✓ | Soft stop preserves the container; Start runs `compose start` | | Postgres / Redis / etc. | ✓ | Same | | Database data | ✓ | Volumes are not touched by soft stop | | Volumes (org-scoped) | ✓ | External; ignored by stop semantics | | Buckets and credentials | ✓ | Bucket env vars are stored in the compose YAML in the DB | | Workdir on disk | ✓ | Soft stop does not delete the workdir | | Port reservation | ✓ | Soft stop does not release the port | | Container IDs in DB | ✓ | If compose recreates anything, IDs are refreshed on Start | This is the change most users notice first. The "Stop, Start" round-trip is now safe. --- ## What requires a redeploy Some configuration changes are baked into the compose YAML at deploy time. Picking these up after a deploy requires a redeploy, not just a restart: - New environment variables added after the original deploy. - New bucket attachments (the `S3_*` env vars are baked in). - New volume attachments (mount points are baked into the compose YAML). - Changes to the start command, framework, or Dockerfile. For those, run `nexus deploy redeploy <deployment-id> --wait`. The redeploy rebuilds the image and recreates the containers with the new configuration. Data on persistent volumes survives the redeploy. --- ## Code pointers For readers who want to see the implementation, the relevant code lives in: - `backend/src/services/dockerService.ts` - `stopContainer()`. Soft and hard stop modes. - `startContainer()`. Orchestrates compose vs single-container start. - `startComposeDeployment()`. The rehydrate plus `compose up -d` flow. - `startSingleContainerDeployment()`. `container.start()` with rebuild fallback. - `rehydrateComposeWorkdir()`. Recreates workdir from DB state. - `reconcileRunningDeployments()`. Boot-time recovery. - `getDeploymentWorkDir()` and `ensureDeploymentWorkDir()`. Persistent path helpers. - `backend/src/index.ts`. Calls `reconcileRunningDeployments()` 5 seconds after API startup. - `systemd/nexus-backend.service`. Sets `StateDirectory=nexus/deployments nexus/docker`, `ReadWritePaths=/var/lib/nexus`, and `Environment=DOCKER_CONFIG=/var/lib/nexus/docker`. --- ## What this changes about the platform's reliability story Before the fix, the honest answer to "does my Postgres survive a host reboot" was "usually, but not if your backend restarted at the wrong moment, and not if anyone clicked Stop, and not if `/tmp` got cleaned." That is the same answer most single-host Docker platforms give if you press them. After the fix, the answer is "yes, full stop." Containers come back via `restart: unless-stopped`. The orchestrator survives because its workdirs are in `/var/lib`. Stop is a stop, not a destroy. Start is a start, not a rebuild. Boot reconciliation catches the edge cases where Docker did not bring something back on its own. That changes what NEXUS AI can credibly host. Stateful workloads (Postgres, Mongo, Redis, file-backed apps) are first-class, not "use a cloud DB instead." The cost of running a stateful AI app on the platform is now bounded by Docker, the kernel, and the disk, not by an orchestration layer that loses its mind on every reboot. --- ## FAQ **Will my data survive a host crash, not just a clean reboot?** For unclean shutdowns (power loss, kernel panic), Postgres recovers from its WAL on next start, the same as any Postgres install. The reboot recovery path described in this post brings the container back up; Postgres handles its own crash recovery from there. The same applies to MySQL and Mongo via their own journaling. Redis recovers from its AOF or RDB snapshot. **What if Docker itself crashes and restarts?** Same answer as a host reboot. `restart: unless-stopped` brings containers back when the daemon comes back. Boot reconciliation catches anything that did not restart cleanly. **Does this work if I am running the backend in development without systemd?** Yes. The persistent workdir helper falls back to `/tmp/deploy-full-<id>/` if `/var/lib/nexus/deployments` is not writable. In dev that is fine because the backend usually stays running across deploys. **What about replica counts after restart?** Replica count preservation across restart is a known limitation in the current release. `composeService` does not yet emit `deploy.replicas` in the generated YAML, so a restart brings 1 replica back regardless of prior scaling. Fix is on the roadmap. Manual workaround: run `nexus deploy scale <id> <n>` after restart. **What if the cached image was pruned (`docker image prune -a`)?** Restart needs the cached `deploy-<id>:latest` image. If it was pruned and the build context is not available, `docker compose up -d` will try to build, fail, and the platform surfaces a clear error directing you to redeploy. Don't run `docker image prune -a` on a NEXUS AI host without intent. **Are backups affected by any of this?** Backups are taken on demand or on schedule. They are not part of the restart path. A reboot does not trigger or invalidate backups. Existing backups remain downloadable. New backups can be taken before, during, or after a reboot. **Can I trigger reconciliation manually?** Not yet via a public endpoint. The reconciliation runs automatically 5 seconds after backend startup. If you need to trigger it manually (e.g., after a manual `docker rm`), restart the backend (`sudo systemctl restart nexus-backend`) and the reconciliation will run on the next boot. --- The reliability layer is the part of a platform users never think about until it fails. Building it as a first-class feature is the reason we can credibly host the database under your AI-generated app, not just the app itself. [Start free.](https://nexusai.run/register)