This adds the cli proposed in https://github.com/drifting-in-space/plane/pull/199

I'm still working on this (need to add in certs and make the guac-firefox default open up to plane.dev) but I wanted some comments on the cli, the way it works is: ./plane-dev -[lmxg] where -l : linux -m : mac -x : x11 -g : guac

I'd have nice long options, but can't do that without GNU getopts, which isn't on macs by default, so I'd rather not. what'd you guys think?

It looks like currently the container I launch has to listen on port 8080 for incoming connections. It would be nice if the API (DockerExecutableConfig) supported specifiying a different port.

Hi I'm seeing this error when I try to run the sample-config Any workaround? Seems like a rust / docker / M1 Mac issue - https://docs.rs/notify/latest/notify/

$ docker compose up

plane-drone       | Error: Error building cert refresher.
plane-drone       | 
plane-drone       | Caused by:
plane-drone       |     Function not implemented (os error 38)

This is something I've wanted to do to reduce cold start times when we know we have the image locally, and has also been requested by @mycaddev via Discord.

I followed along the example and opened the drop four app in firefox

plane-cli list-dns is giving this below error

2022-11-17T23:16:57.797046Z  INFO async_nats::options: event: connected
Error: NATS Error: Custom { kind: Other, error: "error while processing messages from the stream: 404, Some(\"No Messages\")" }

E.g. https://github.com/drifting-in-space/spawner/runs/8142561122?check_suite_focus=true

@pretentious7 how frequent are these checks emitted? I thought 30s would be a long enough time to wait but maybe not.

Evening :wave:

I stumbled across your project whilst planning to develop something which sounds very similar. I thought I'd let you know my intended use case, because it sounds like you may already be developing the solution I'm looking for.

My plan was to authenticate users using my existing Keycloak OIDC SSO, and then spin up a fresh code-server container.

User story:

• User navigates to a site and is prompted to login with OAuth.
• Once logged in, server automatically spins up a new container.
  • A volume (whose name is based on the user's username) is mapped to the new container for data persistence.
  • Container is exposed on a subdomain via a reverse proxy.
• Server redirects user to the unique subdomain.

A crude diagram in an attempt to illustrate this is below.

I'd be interested to hear if this sort of architecture matches what you're aiming to achieve.

No worries if not! :-)

This adds a flag to the proxy to allow a "passthrough" SocketAddr used when a backend can't be resolved. All unknown hosts will be proxied to this backend, but it will all be TLS terminated with the same certificate, so if this is used to resolve other hostnames a custom certificate will need to be passed in which covers all of them.

Only HTTP is supported for the upstream connection.

This will close #289.

We're using plane in the standalone configuration (single drone, without dnsmasq / firefox). We use a wilcard dns to map *.example.com => 1.2.3.4

In order to manage plane, we have a custom webservice. Currently we need a separate host or port for that, but we would like to host that on the same host and port as the drone. For Example:

• When we visit example.com we reach our custom webservice (e.g. listening on 127.0.0.1:4567)
• When we visit e485cf9a-fc65-4d7c-935e-656572068226.example.com => we reach the launched backend

It would be nice if we could provide custom subdomain to route mapping in the drone.toml as a map. Example:

[proxy]
# IP to listen for connections on.
bind_ip = "0.0.0.0"

[proxy.custom_route]
host = "status.example.com"
route = "http://127.0.0.1:9874"

[proxy.custom_route]
host = "example.com"
route = "http://127.0.0.1:4567"

If I'm correct, I could extend the code here, to make that work: https://github.com/drifting-in-space/plane/blob/dbd550e9b1db33e611ab00730182a28eaf12e7db/drone/src/proxy/service.rs#L203-L236

Hi there, This project looks like the perfect foundation for our use case: We develop a webapp in .NET/Angular and want to spin up a backend (single docker container) for each developer when they click a link in the CI build output of our project (e.g. to Preview a PullRequest).

We want to host plane on our azure infrastructure, but we don't need multiple drones right now. Is there a way to disable the DNS stuff and just create a single drone and a scheduler? We would then create a wildcard dns to map *.test-our-webapp.com => The-One-And-Only-Drone-IP. We really don't want to host multiple services (dns, scheduler, drone) for this, because we need to keep our cloud cost at a minimum....

Thanks in advance

Previously, all DNS records for a given cluster went to the same subject. The cli fetches DNS records using DeliverPolicy::LastPerSubject, so it only gets the last message sent to a subject (so one per cluster). This fixes the issue by adding the backend ID to the subject.

Leaving this as a draft for now because this changes the NATS API between the drone and controller and I haven't thought through a zero-downtime migration path as we try to do for NATS API changes.

Currently, migration requires:

• Re-installing the CLI (cargo install --path=cli in root of the repo)
• Deleting the old stream (nats stream delete dns_record)
• Re-building and running the controller and drone (docker compose build followed by docker compose up)

It could be confusing that we spin up some extra services to get a local demo running that are not needed for a production system. We should have a docker compose file that only runs the controller, DNS server, and NATS jetstream server.

We should also consider an example that doesn't use DNS, as mentioned in #279.

Our current approach to state is to use non-Jetstream NATS and treat it as unreliable and ephemeral. Drones simply rebroadcast all of their current state frequently to keep that state alive in the system. Consumers of that state will cache it for a limited TTL, and then intentionally forget it if it is not rebroadcast again.

This has some maintenance benefits, because even though the system is stateful, it does not have long-persisted state. It does have a few downsides, though:

• As the system state grows, it can become very chatty to have all state rebroadcast frequently. This is especially relevant as we add more state to be used for scheduling.
• There's no system-level consensus on when a drone has become lost, or who is responsible for declaring it lost. As a result, we currently do not handle drone loss.
• A bunch of code that doesn't really have to do with events (i.e. just needs a static view of cluster state) has to deal with the event-based nature of the data.

Now that we have a better understanding of the semantics of Jetstream (including @pretentious7's deep dives), I propose returning to using Jetstream to store Plane's runtime state.

The approach I propose is:

• All state used for scheduling and routing is moved into a single Jetstream stream.
• The stream captures all subjects that match cluster.*.drone.*.sm.> and stores the last message on each subject.
• One-time consumers of this state read a “snapshot” copy by reading the last message on every subject from NATS. Once the snapshot is read, the stream consumer is dropped.
• Long-term consumers of this state (including the DNS server and scheduler) keep a single local copy of a view into all system state. The process has a single global instance of a SystemViewReplica which owns a NATS subscription which processes change events from the drones. These events are applied to the shared data structure. Consumers do not need to be aware of the event-based nature of the data, they just access system state by calling methods of a data structure. All requests are synchronous, since a replica of the entire system state data structure is stored locally.

Tasks:

• [x] Use this model for the CLI client, and broadcasts the appropriate messages. The old messages remain for now. #274
• [ ] Switch to the new model for DNS serving. #285
• [ ] Switch to the new model for scheduling.
• [ ] Remove BackendStateMessage and SetDnsRecord (we will keep SetAcmeDnsRecord for ACME, counter to the previous plans to do all DNS through SetDnsRecord).
• [ ] Handle lost drones appropriately and send update messages on all of their backends.

Consumers of BackendStateMessage will need to change, but it's mostly just a matter of listening on a different subject wildcard as the message has mostly the same information.

This will fix #265, and supersede #145’s proposed changes to SetDnsRecord and BackendStateMessage.

I just tried this with latest pull and install of cli

Created two backends

% plane-cli spawn plane.test ghcr.io/drifting-in-space/demo-image-drop-four
Backend scheduled.
URL: https://64becea6-7e02-472e-bc1e-8319363843d5.plane.test
Drone: 6c12439f-6e48-4a4a-96b1-489be751ed56
Backend ID: 64becea6-7e02-472e-bc1e-8319363843d5

% plane-cli spawn plane.test ghcr.io/drifting-in-space/demo-image-drop-four
Backend scheduled.
URL: https://03b6ca25-b8f1-4519-952d-56d845474434.plane.test
Drone: 6c12439f-6e48-4a4a-96b1-489be751ed56
Backend ID: 03b6ca25-b8f1-4519-952d-56d845474434

List DNS is always showing one or the other - never both

% plane-cli list-dns
Found 1 DNS records:
64becea6-7e02-472e-bc1e-8319363843d5.plane.test	A	172.16.238.3

% plane-cli list-dns
Found 1 DNS records:
03b6ca25-b8f1-4519-952d-56d845474434.plane.test	A	172.16.238.3

% plane-cli list-dns
Found 1 DNS records:
03b6ca25-b8f1-4519-952d-56d845474434.plane.test	A	172.16.238.3

% plane-cli list-dns
Found 1 DNS records:
03b6ca25-b8f1-4519-952d-56d845474434.plane.test	A	172.16.238.3

% plane-cli list-dns
Found 1 DNS records:
64becea6-7e02-472e-bc1e-8319363843d5.plane.test	A	172.16.238.3

We currently generate NATS subjects from message contents, but we don't verify that a message was received at the appropriate subject, so a connected client with access to publish on cluster.foo_bar.* can publish a message about the fizz_boom cluster instead of the foo_bar cluster and it is not checked. This is fine in the current situation where there is only one tenant, but will become necessary when we implement #124.

This will not be useful on all subjects until #145 ensures that all subjects include the cluster, since the cluster is level of isolation we want.

Rather than try to parse the subject, we can just generate the subject upon receiving a message and do a string comparison. This works because subjects are deterministically generated from messages.