Gantry

Direction, not current-state fiction. Watchpost already has the server/agent relationship that makes fleet work natural. Clustering, configuration propagation, distributed Webfleet workers and Cortex remediation are roadmap directions described here so the architecture can converge deliberately rather than as unrelated features.

Machine identity belongs with Watchpost

Watchpost already enrolls machines through Watchpost Agent, receives their telemetry and presents them as operational objects. That makes it the natural place to attach fleet membership and optional Gantry service associations:

prod-api-01
├── Watchpost Agent   paired
├── Cortex            https://cortex.prod-01.example
├── Warden            https://warden.prod-01.example
├── Trestle            running
└── Fleet              production

The association should be explicit. Watchpost does not need to scan the whole machine and claim ownership of every service; an operator can configure or approve discovered relationships.

Fleet groups are targeting, not a new security boundary

A fleet is a convenient collection such as development, production, home lab or regional probes. It lets the operator say “apply this to these machines” without selecting twenty checkboxes every time.

Membership alone should not grant additional machine privileges. Pairing/authentication still establishes which Watchpost server/agent relationship is trusted; the fleet is an organizational and configuration target on top of that relationship.

Configuration bundles

The primary payoff is repeatable setup. Instead of visiting every machine to configure the same Cortex provider or Watchpost policy, define a versioned bundle and target it:

Bundle: Development Cortex defaults
Revision: 8

Includes
✓ provider configuration
✓ credential reference: cortex-main
✓ model defaults
✓ agent defaults

Targets
✓ workstation
✓ nuc
✓ dev-vps

Each target can report applied, pending, failed or drifted. An offline machine keeps the desired revision pending and receives it after reconnecting rather than forcing an operator to remember which host was unavailable during the rollout.

Apply once, managed, and defaults are different

Configuration propagation needs explicit ownership semantics:

Mode	Meaning
Apply once	Push this configuration now, then local administrators may change it freely.
Managed	Watchpost remains authoritative and can report/reconcile drift. Use only for settings you genuinely want centrally controlled.
Default for new machines	Seed newly enrolled members of a fleet, but do not continuously overwrite local changes.

This distinction prevents “central management” from becoming surprising background overwrites.

Secrets are not ordinary configuration

API keys and provider credentials should be write-only in normal fleet UX. A Watchpost server can store an encrypted secret or a reference to one, then deliver it over the authenticated agent channel to selected machines. The UI should show which secret/revision is assigned without displaying the secret value back to every operator.

A stronger design can encrypt a secret specifically for the enrolled target so intercepted fleet data is not a reusable plaintext credential. Rotation then becomes a normal rollout:

Credential: production-ai
Revision: 9
Targets: 17

16 applied
 1 offline — pending

Safe rollout semantics

Fleet changes should be reviewable before they are pushed. Useful controls include dry-run diff, selected machines/fleets, revision history, bounded concurrency and a visible result per target. A failed machine must not turn the whole operation into an ambiguous “mostly worked” state.

Where a configuration can break a service, the agent should apply it transactionally where possible: write/validate new state, restart/reload, verify health, then commit—or restore the previous known-good state.

Cortex on a monitored machine

The first Cortex integration is intentionally simple: a machine can expose Open Cortex ↗. Opening a new tab is deliberate here because Watchpost is the incident evidence and Cortex is the investigation workspace.

The integration can then become progressively more capable without jumping straight to autonomous repair:

Open Cortex
    ↓
Pass incident context
    ↓
Read-only investigation
    ↓
Proposed remediation
    ↓
Human approval
    ↓
Pre-approved automatic policy (only where appropriate)

Statistical anomalies, model confidence and telemetry are evidence. None of them alone should be treated as authorization to make changes.

Watchpost clustering: availability of the control plane

A clustered Watchpost deployment solves a different problem from having many Agents. The monitoring server itself should not be a single point of failure for a serious fleet. A future cluster needs clear answers for:

stable node identity and enrollment;
shared/replicated application state;
leader/lease ownership for work that must happen once;
agent routing and reconnect behaviour;
version/protocol negotiation during rolling upgrades;
health/quorum visibility that does not require reading internal logs.

The exact consensus/replication design should come from the real Watchpost workload rather than a generic Gantry cluster framework designed in advance.

Webfleet clustering vs distributed workers

Webfleet has two independent reasons to use more than one node:

Clustering / HA: multiple Webfleet server nodes keep the control plane, schedules and result store available when one coordinating machine fails.

Distributed observation: workers in different networks/regions run checks because location changes the thing being measured. Melbourne can succeed while New York fails; one region can see stale DNS, broken IPv6, a CDN edge issue or very different latency.

                 Webfleet
                    │
       ┌────────────┼────────────┐
       ▼            ▼            ▼
   Melbourne      London      New York
     182 ms         73 ms       FAIL
     TLS ✓          TLS ✓       DNS ✗

Those worker results should remain individually visible. A global green/red rollup is useful, but it must not erase the evidence that only one region failed.

Watchpost distributed monitoring

Watchpost can also gain value from multiple observation points where the monitored environment crosses sites or networks. That does not mean every Agent becomes a peer server. Agent telemetry, remote checks and Watchpost server clustering are different responsibilities and should remain distinguishable in the UI and protocol.

Shared primitives—only after they are proven

Watchpost and Webfleet are likely to need some of the same low-level machinery: node identity, enrollment, authentication, heartbeats, capabilities, leases, protocol versions and rolling-upgrade compatibility. Trestle clustering may share some pieces too.

The extraction rule is important: implement enough real clustering/distribution in the products to prove what is genuinely common, then extract a Gantry primitive. Do not create a generic distributed-systems framework first and force three products to conform to imagined requirements.

The UX constraint

The simplest useful workflow should remain approximately as simple after an advanced feature is added as it was before.

That means:

one Watchpost server + one Agent is still install → pair → monitor;
one Webfleet instance is still add URL → monitor;
one Cortex host still needs only its local configuration;
fleet membership appears only when there is a fleet;
cluster/quorum controls appear only on clustered deployments;
regional views appear only when distributed workers exist.