How NEXUS AI keeps your Postgres alive across host reboots

Five things can fail on host reboot and silently take your data offline. This is the engineering deep dive on how NEXUS AI fixes all five, with code pointers.

By NEXUS AI Team • May 17, 2026 • AI Deployments

Published: May 17, 2026 Category: Engineering · Reliability · Postgres Reading time: 10 minutes Author: NEXUS AI Team

"What happens to my data if the host reboots?" is the question every production platform owes a real answer to. Most answer it with marketing copy. This post is the engineering answer for NEXUS AI, with the actual mechanics.

If you run stateful apps (Postgres, MySQL, Mongo, Redis, persistent volumes, S3 buckets) on a single-host Docker platform, five things can fail on reboot and silently take your data offline. The combination of those five failures was the reason most Docker-based deploy platforms historically said "use cloud DBs instead." NEXUS AI fixes all five.

For the broader story, see Your AI app is generated. Now how do you deploy it?. This post is the deep dive on the reliability layer.

The reboot problem in one paragraph

When the host reboots, the Docker daemon comes back, and containers with restart: unless-stopped should come back too. In practice, they often do not, because the orchestrator (the thing that originally ran docker compose up) has lost its on-disk project context (compose file, build context, environment variables). When a user then clicks "Stop" and "Start", most platforms rebuild from scratch and lose the wiring to the database, buckets, and volumes that were attached to the original deployment. The DB row says RUNNING. Reality says otherwise. Data does not survive.

The fix is to make the orchestrator survive reboots too, and to make "start" a real start, not a rebuild.

The five-layer failure most platforms have

Layer 1: ephemeral compose workdirs

A Docker-based platform generates a docker-compose.yml per deployment, writes it to a workdir on disk, and runs docker compose up -d from that directory. The most common default is to put the workdir under /tmp/. On most Linux distros /tmp is wiped on reboot (tmpfs by default on Ubuntu 22.04+, systemd-tmpfiles on others). The workdir is gone. The platform now has no way to drive docker compose stop, start, up, or down against the original project.

Layer 2: systemd PrivateTmp wipes /tmp on every backend restart

Production deployments of the platform's backend usually run as a systemd unit with hardening: ProtectSystem=strict, ProtectHome=true, PrivateTmp=true. The PrivateTmp=true directive gives the service its own private /tmp namespace. Every backend restart (a code update, a crash, a reboot) gets a fresh empty /tmp. Workdirs are wiped not just on host reboot, but on every backend restart.

Layer 3: stop = destroy

Most platforms implement "Stop" as docker compose down, which removes containers, networks, and (with -v) volumes. Or as docker stop plus docker rm. After "Stop" there is no container left to start.

Layer 4: start = rebuild

Because Stop removed everything, Start is forced to rebuild the image, recreate the container, reattach networks, remount volumes, and reinject environment variables. If the build path does not know about all the things the original deploy attached (the bucket env vars, the persistent volume mounts, the linked DB services), Start brings the app container back without them. The app starts. The database it depended on is gone. Worse, the app's first write to its now-missing Postgres dependency takes down the whole stack.

Layer 5: no boot reconciliation

When the backend comes back online after a reboot, it does not check whether the deployments the DB believes are RUNNING are actually running. Containers that failed to restart (network ordering issues, transient daemon errors, manual docker rm) stay dead until a human notices and clicks "Start", which then triggers layer 4.

Each layer alone is recoverable. Together they mean a host reboot, a backend restart, or a user clicking "Stop, Start" can wipe a Postgres deployment's working state.

How NEXUS AI fixes each layer

Fix 1: persistent compose workdirs

Compose workdirs live at /var/lib/nexus/deployments/<deployment-id>/, never in /tmp. The backend's systemd unit declares:

StateDirectory=nexus/deployments nexus/docker
StateDirectoryMode=0750

systemd auto-creates /var/lib/nexus/deployments and /var/lib/nexus/docker on first start and chowns them to the nexus user. The directory survives host reboots, backend restarts, and systemd-tmpfiles runs.

The base path is overridable via NEXUS_DEPLOYMENTS_DIR for non-standard installs.

Fix 2: PrivateTmp does not matter

Because workdirs are not in /tmp, PrivateTmp=true no longer wipes them. The backend keeps PrivateTmp=true on for security (process isolation), but the workdirs live where they need to.

ReadWritePaths=/var/lib/nexus is added to the unit so the otherwise read-only filesystem allows writes to the persistent directory.

Fix 3: stop is a real stop

stopContainer has two modes:

• Soft stop (default). docker compose stop for compose deployments or container.stop() for single containers. Containers are preserved. Networks are preserved. Volumes are preserved. The workdir is preserved. The port reservation is preserved. Status flips to STOPPED. destroyedAt stays null.
• Hard destroy (options.removeVolumes: true). docker compose down -v plus network cleanup, volume cleanup, workdir deletion, port release. This is the path the Delete endpoint and auto-destroy job take.

The user-facing "Stop" button uses the soft path. "Delete" uses the hard path. Both are tested in production.

Fix 4: start is a real start

startContainer does the minimum work needed to bring an existing stack back online:

• For compose deployments:
  1. Ensure the external Traefik network exists (idempotent).
  2. Find the workdir. If missing, rehydrate it from the DB.
  3. Run docker compose up -d (without --build). The cached image deploy-<id>:latest is reused. External volumes reattach. Networks reattach. Environment variables are read from the persisted .env.
  4. Refresh container IDs in the DB (compose may have recreated stopped containers with new IDs).
• For single-container deployments:
  1. container.start() on the existing container ID.
  2. If the container is gone (HTTP 404 from the Docker daemon), fall back to a full rebuild.

The rehydration step (point 2 above) is the load-bearing part. It writes docker-compose.yml from deployment.composeFile in the DB, writes Dockerfile from deployment.dockerfile, and reconstructs the .env file from deployment.port plus the hostPort values stored on each DeploymentService row. No source code is needed because the cached image is reused.

Fix 5: boot reconciliation

On backend startup (five seconds after the API begins serving), the backend runs:

dockerService.reconcileRunningDeployments()

That method:

1. Queries every deployment with status RUNNING or BUILDING and provider LOCAL_DOCKER.
2. For each, calls docker.inspect() on the recorded containerId.
3. If the container is not running (or returns 404), classifies it as needing recovery.
4. For compose deployments, calls the same startComposeDeployment helper used by the user-facing Start endpoint (rehydrate workdir if needed, docker compose up -d).
5. For single-container deployments, calls container.start(), falling back to rebuild on 404.
6. Logs checked=N recovered=N failed=N so an operator can see what the platform did on startup.

The job runs non-blocking so the API serves traffic immediately even if Docker is slow to respond on a freshly booted host. Failures on individual deployments are logged but do not block reconciliation of the rest.

What survives a host reboot

After a clean reboot of the host running NEXUS AI:

What does not automatically survive a reboot, by design:

• Logs older than your retention window (configurable per plan).
• Backup files stored only in /tmp/nexus-backups/ (they are copied to persistent storage by the backup job; the temporary working copy may go).

What survives a manual Stop then Start

After a user clicks Stop, then Start (the operation that historically broke most platforms):

This is the change most users notice first. The "Stop, Start" round-trip is now safe.

What requires a redeploy

Some configuration changes are baked into the compose YAML at deploy time. Picking these up after a deploy requires a redeploy, not just a restart:

• New environment variables added after the original deploy.
• New bucket attachments (the S3_* env vars are baked in).
• New volume attachments (mount points are baked into the compose YAML).
• Changes to the start command, framework, or Dockerfile.

For those, run nexus deploy redeploy <deployment-id> --wait. The redeploy rebuilds the image and recreates the containers with the new configuration. Data on persistent volumes survives the redeploy.

Code pointers

For readers who want to see the implementation, the relevant code lives in:

• backend/src/services/dockerService.ts
  • stopContainer(). Soft and hard stop modes.
  • startContainer(). Orchestrates compose vs single-container start.
  • startComposeDeployment(). The rehydrate plus compose up -d flow.
  • startSingleContainerDeployment(). container.start() with rebuild fallback.
  • rehydrateComposeWorkdir(). Recreates workdir from DB state.
  • reconcileRunningDeployments(). Boot-time recovery.
  • getDeploymentWorkDir() and ensureDeploymentWorkDir(). Persistent path helpers.
• backend/src/index.ts. Calls reconcileRunningDeployments() 5 seconds after API startup.
• systemd/nexus-backend.service. Sets StateDirectory=nexus/deployments nexus/docker, ReadWritePaths=/var/lib/nexus, and Environment=DOCKER_CONFIG=/var/lib/nexus/docker.

What this changes about the platform's reliability story

Before the fix, the honest answer to "does my Postgres survive a host reboot" was "usually, but not if your backend restarted at the wrong moment, and not if anyone clicked Stop, and not if /tmp got cleaned." That is the same answer most single-host Docker platforms give if you press them.

After the fix, the answer is "yes, full stop." Containers come back via restart: unless-stopped. The orchestrator survives because its workdirs are in /var/lib. Stop is a stop, not a destroy. Start is a start, not a rebuild. Boot reconciliation catches the edge cases where Docker did not bring something back on its own.

That changes what NEXUS AI can credibly host. Stateful workloads (Postgres, Mongo, Redis, file-backed apps) are first-class, not "use a cloud DB instead." The cost of running a stateful AI app on the platform is now bounded by Docker, the kernel, and the disk, not by an orchestration layer that loses its mind on every reboot.

FAQ

Will my data survive a host crash, not just a clean reboot? For unclean shutdowns (power loss, kernel panic), Postgres recovers from its WAL on next start, the same as any Postgres install. The reboot recovery path described in this post brings the container back up; Postgres handles its own crash recovery from there. The same applies to MySQL and Mongo via their own journaling. Redis recovers from its AOF or RDB snapshot.

What if Docker itself crashes and restarts? Same answer as a host reboot. restart: unless-stopped brings containers back when the daemon comes back. Boot reconciliation catches anything that did not restart cleanly.

Does this work if I am running the backend in development without systemd? Yes. The persistent workdir helper falls back to /tmp/deploy-full-<id>/ if /var/lib/nexus/deployments is not writable. In dev that is fine because the backend usually stays running across deploys.

What about replica counts after restart? Replica count preservation across restart is a known limitation in the current release. composeService does not yet emit deploy.replicas in the generated YAML, so a restart brings 1 replica back regardless of prior scaling. Fix is on the roadmap. Manual workaround: run nexus deploy scale <id> <n> after restart.

What if the cached image was pruned (docker image prune -a)? Restart needs the cached deploy-<id>:latest image. If it was pruned and the build context is not available, docker compose up -d will try to build, fail, and the platform surfaces a clear error directing you to redeploy. Don't run docker image prune -a on a NEXUS AI host without intent.

Are backups affected by any of this? Backups are taken on demand or on schedule. They are not part of the restart path. A reboot does not trigger or invalidate backups. Existing backups remain downloadable. New backups can be taken before, during, or after a reboot.

Can I trigger reconciliation manually? Not yet via a public endpoint. The reconciliation runs automatically 5 seconds after backend startup. If you need to trigger it manually (e.g., after a manual docker rm), restart the backend (sudo systemctl restart nexus-backend) and the reconciliation will run on the next boot.

The reliability layer is the part of a platform users never think about until it fails. Building it as a first-class feature is the reason we can credibly host the database under your AI-generated app, not just the app itself.

Start free.