spike: discovery server

Author: justinsb

discovery/GEMINI.md

@@ -0,0 +1,63 @@
+# Discovery Service Project
+
+## Overview
+This project implements a public discovery service designed for decentralized, secure peer discovery. The core innovation is the use of **Custom Certificate Authorities (CAs)** to define isolated "Universes". Clients register and discover peers within their own Universe, identified and secured purely by mTLS.
+
+The service emulates a Kubernetes API, allowing interaction via `kubectl`, including support for **Server-Side Apply**.
+
+## Key Concepts
+
+### 1. The "Universe"
+- A **Universe** is an isolated scope for peer discovery.
+- It is cryptographically defined by the **SHA256 hash of the Root CA's Public Key**.
+- Any client possessing a valid certificate signed by a specific CA belongs to that CA's Universe.
+- Different CAs = Different Universes. There is no crossover.
+
+### 2. Authentication & Authorization
+- **Mechanism**: Mutual TLS (mTLS).
+- **Client Identity**: Derived from the **Common Name (CN)** of the leaf certificate.
+- **Universe Context**: Derived from the **Root CA** presented in the TLS handshake.
+- **Requirement**: Clients **MUST** present the full certificate chain (Leaf + Root CA) during the handshake. The server does not maintain a pre-configured trust store for these custom CAs; it uses the presented chain to determine the scope.
+
+### 3. API Resources
+- **DiscoveryEndpoint** (`discovery.kops.k8s.io/v1`): Represents a peer in the discovery network. Can optionally hold OIDC configuration (Issuer URL, JWKS).
+- **Validation**: A client with CN `client1` can only Create/Update a `DiscoveryEndpoint` named `client1`.
+- **Apply Support**: The server supports `PATCH` requests to facilitate `kubectl apply --server-side`.
+
+### 4. OIDC Discovery
+The server acts as an OIDC Discovery Provider for the Universe.
+- **Public Endpoints**:
+  - `/.well-known/openid-configuration`: Returns the OIDC discovery document.
+  - `/openid/v1/jwks`: Returns the JSON Web Key Set (JWKS).
+- **Data Source**: These endpoints serve data uploaded by clients via the `DiscoveryEndpoint` resource.
+
+## Architecture
+
+### Project Structure
+- `cmd/discovery-server/`: Main entry point. Wires up the HTTP server with TLS configuration.
+- `pkg/discovery/`:
+  - `auth.go`: logic for inspecting TLS `PeerCertificates` to extract the Universe ID (CA hash) and Client ID.
+  - `store.go`: In-memory thread-safe storage (`MemoryStore`) mapping Universe IDs to lists of `DiscoveryEndpoint` objects.
+  - `server.go`: HTTP handlers implementing the K8s API emulation for `/apis/discovery.kops.k8s.io/v1`.
+  - `k8s_types.go`: Definitions of `DiscoveryEndpoint`, `DiscoveryEndpointList`, `TypeMeta`, `ObjectMeta` etc.
+
+### Data Model
+- **DiscoveryEndpoint**: The core resource. Contains `Spec.Addresses` and metadata.
+- **Universe**: Contains a map of `DiscoveryEndpoint` objects (keyed by name).
+- **Unified Types**: The API type `DiscoveryEndpoint` is used directly for in-memory storage, ensuring zero conversion overhead.
+
+## Security Model
+- **Trust Delegation**: The server delegates trust to the CA. If you hold the CA key, you control the Universe.
+- **Isolation**: The server ensures that a client presenting a cert chain for `CA_A` cannot read or write data to the Universe defined by `CA_B`.
+- **Ephemeral**: The current implementation uses in-memory storage. Data is lost on restart.
+
+## Building and Running
+
+### Build
+```bash
+go build ./cmd/discovery-server
+```
+
+### Run
+
+See docs/walkthrough.md for instructions on testing functionality.

discovery/README.md

@@ -0,0 +1,42 @@
+# Discovery Service
+
+A public discovery service using mTLS for authentication and "Universe" isolation, emulating a Kubernetes API.
+
+## Concept
+
+- **Universe**: Defined by the SHA256 Fingerprint of a Custom CA Certificate.
+- **Client**: Identified by a Client Certificate signed by that Custom CA.
+- **DiscoveryEndpoint**: The resource type representing a registered client.
+- **Isolation**: Clients can only see `DiscoveryEndpoint` objects signed by the same Custom CA.
+
+## Usage
+
+### Run Server
+
+```bash
+go run ./cmd/discovery-server --tls-cert server.crt --tls-key server.key --listen :8443
+```
+
+(You can generate a self-signed server certificate for testing, see the [walkthrough](docs/walkthrough.md) ).
+
+### Client Requirement
+
+Clients must authenticate using mTLS.
+**Important**: The client MUST provide the full certificate chain, including the Root CA, because the server does not have pre-configured trust stores for these custom universes.
+The server identifies the Universe from the SHA256 hash of the Root CA certificate found in the TLS chain.
+
+### Quick start
+
+See `docs/walkthrough.md` for detailed instructions.
+
+
+## OIDC Discovery
+
+The discovery server also serves OIDC discovery information publicly, allowing external systems (like AWS IAM) to discover the cluster's identity provider configuration.
+
+- `GET /<universe-id>/.well-known/openid-configuration`: Returns the OIDC discovery document.
+- `GET /<universe-id>/openid/v1/jwks`: Returns the JWKS.
+
+This information is populated by clients uploading `DiscoveryEndpoint` resources containing the `oidc` spec.
+
+## Building and Running

discovery/cmd/discovery-server/main.go

@@ -0,0 +1,70 @@
+/*
+Copyright 2025 The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package main
+
+import (
+	"crypto/tls"
+	"flag"
+	"fmt"
+	"log"
+	"net/http"
+	"os"
+
+	"k8s.io/kops/discovery/pkg/discovery"
+)
+
+func main() {
+	certFile := flag.String("tls-cert", "", "Path to server TLS certificate")
+	keyFile := flag.String("tls-key", "", "Path to server TLS key")
+	addr := flag.String("listen", ":8443", "Address to listen on")
+	storageType := flag.String("storage", "memory", "Storage backend (memory, gcs)")
+	flag.Parse()
+
+	if *certFile == "" || *keyFile == "" {
+		fmt.Fprintf(os.Stderr, "Error: --tls-cert and --tls-key are required\n")
+		flag.Usage()
+		os.Exit(1)
+	}
+
+	var store discovery.Store
+
+	switch *storageType {
+	case "memory":
+		store = discovery.NewMemoryStore()
+	default:
+		log.Fatalf("Unknown storage type: %s", *storageType)
+	}
+
+	handler := discovery.NewServer(store)
+
+	tlsConfig := &tls.Config{
+		ClientAuth: tls.RequestClientCert,
+		// We do not set ClientCAs because we accept any CA and use it to define the universe.
+		MinVersion: tls.VersionTLS12,
+	}
+
+	server := &http.Server{
+		Addr:      *addr,
+		Handler:   handler,
+		TLSConfig: tlsConfig,
+	}
+
+	log.Printf("Discovery server listening on %s using %s storage", *addr, *storageType)
+	if err := server.ListenAndServeTLS(*certFile, *keyFile); err != nil {
+		log.Fatalf("Server failed: %v", err)
+	}
+}

discovery/docs/walkthrough.md

@@ -0,0 +1,60 @@
+# Walkthrough of functionality
+
+
+Quick start:
+
+```bash
+# Generate certs, kubeconfig, yaml files
+# Check out the script to better understand how all the pieces fit together!
+./scripts/create-kubeconfig.sh
+
+# Start Server (using generated server certs)
+go run ./cmd/discovery-server --tls-cert walkthrough/server.crt --tls-key walkthrough/server.key &
+
+# Verify server is running and serving the DiscoveryEndpoint resource
+kubectl --kubeconfig=walkthrough/universe1/client1.kubeconfig api-resources
+
+# List DiscoveryEndpoints
+kubectl --kubeconfig=walkthrough/universe1/client1.kubeconfig get discoveryendpoints --all-namespaces
+
+# Register (Apply)
+# The `metadata.name` MUST match the Common Name (CN) of your client certificate (e.g., `client1`), or the server will reject it with 403 Forbidden.
+kubectl --kubeconfig=walkthrough/universe1/client1.kubeconfig apply -f walkthrough/universe1/client1-discoveryendpoint.yaml  --server-side=true --validate=false
+
+# List DiscoveryEndpoints
+kubectl --kubeconfig=walkthrough/universe1/client1.kubeconfig get discoveryendpoints --all-namespaces
+```
+
+
+## Using curl
+
+The kubernetes API is a well-structured REST API, so we don't have to use kubectl.
+
+If you want to test the API with curl, you must include the **Universe ID** in the URL path.
+
+**Export the Universe ID:**
+```bash
+export UNIVERSE_ID=$(openssl x509 -in walkthrough/universe1/ca.crt -noout -fingerprint -sha256 | sed 's/SHA256 Fingerprint=//' | sed 's/://g' | tr '[:upper:]' '[:lower:]')
+echo "UNIVERSE_ID is ${UNIVERSE_ID}"
+```
+
+```bash
+curl --cert walkthrough/universe1/client1-bundle.crt --key walkthrough/universe1/client1.key --cacert walkthrough/server.crt \
+  "https://localhost:8443/${UNIVERSE_ID}/apis/discovery.kops.k8s.io/v1/namespaces/default/discoveryendpoints"
+```
+
+## OIDC discovery
+
+The goal here is to allow anonymous access to OIDC endpoints, so let's verify that:
+
+**Getting the OpenID Configuration**
+```bash
+curl --cacert walkthrough/server.crt  "https://localhost:8443/${UNIVERSE_ID}/.well-known/openid-configuration"
+```
+
+**Getting the JWKS keys**
+```bash
+curl --cacert walkthrough/server.crt  "https://localhost:8443/${UNIVERSE_ID}/openid/v1/jwks"
+```
+
+Note that we do not need a client certificate to get this data.  Data from the DiscoveryEndpoints is published publicly.

discovery/go.mod

@@ -0,0 +1,34 @@
+module k8s.io/kops/discovery
+
+go 1.25.4
+
+require (
+	k8s.io/apimachinery v0.34.3
+	k8s.io/client-go v0.34.3
+	k8s.io/klog/v2 v2.130.1
+)
+
+require (
+	github.com/davecgh/go-spew v1.1.1 // indirect
+	github.com/fxamacker/cbor/v2 v2.9.0 // indirect
+	github.com/go-logr/logr v1.4.3 // indirect
+	github.com/gogo/protobuf v1.3.2 // indirect
+	github.com/json-iterator/go v1.1.12 // indirect
+	github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
+	github.com/modern-go/reflect2 v1.0.3-0.20250322232337-35a7c28c31ee // indirect
+	github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
+	github.com/x448/float16 v0.8.4 // indirect
+	go.yaml.in/yaml/v2 v2.4.2 // indirect
+	golang.org/x/net v0.38.0 // indirect
+	golang.org/x/oauth2 v0.27.0 // indirect
+	golang.org/x/sys v0.31.0 // indirect
+	golang.org/x/term v0.30.0 // indirect
+	golang.org/x/text v0.23.0 // indirect
+	golang.org/x/time v0.9.0 // indirect
+	gopkg.in/inf.v0 v0.9.1 // indirect
+	k8s.io/utils v0.0.0-20250604170112-4c0f3b243397 // indirect
+	sigs.k8s.io/json v0.0.0-20241014173422-cfa47c3a1cc8 // indirect
+	sigs.k8s.io/randfill v1.0.0 // indirect
+	sigs.k8s.io/structured-merge-diff/v6 v6.3.0 // indirect
+	sigs.k8s.io/yaml v1.6.0 // indirect
+)