spike: discovery server

Author: justinsb

discovery/GEMINI.md

@@ -0,0 +1,63 @@
+# Discovery Service Project
+
+## Overview
+This project implements a public discovery service designed for decentralized, secure peer discovery. The core innovation is the use of **Custom Certificate Authorities (CAs)** to define isolated "Universes". Clients register and discover peers within their own Universe, identified and secured purely by mTLS.
+
+The service emulates a Kubernetes API, allowing interaction via `kubectl`, including support for **Server-Side Apply**.
+
+## Key Concepts
+
+### 1. The "Universe"
+- A **Universe** is an isolated scope for peer discovery.
+- It is cryptographically defined by the **SHA256 hash of the Root CA's Public Key**.
+- Any client possessing a valid certificate signed by a specific CA belongs to that CA's Universe.
+- Different CAs = Different Universes. There is no crossover.
+
+### 2. Authentication & Authorization
+- **Mechanism**: Mutual TLS (mTLS).
+- **Client Identity**: Derived from the **Common Name (CN)** of the leaf certificate.
+- **Universe Context**: Derived from the **Root CA** presented in the TLS handshake.
+- **Requirement**: Clients **MUST** present the full certificate chain (Leaf + Root CA) during the handshake. The server does not maintain a pre-configured trust store for these custom CAs; it uses the presented chain to determine the scope.
+
+### 3. API Resources
+- **DiscoveryEndpoint** (`discovery.kops.k8s.io/v1alpha1`): Represents a peer in the discovery network. Can optionally hold OIDC configuration (Issuer URL, JWKS).
+- **Validation**: A client with CN `client1` can only Create/Update a `DiscoveryEndpoint` named `client1`.
+- **Apply Support**: The server supports `PATCH` requests to facilitate `kubectl apply --server-side`.
+
+### 4. OIDC Discovery
+The server acts as an OIDC Discovery Provider for the Universe.
+- **Public Endpoints**:
+  - `/.well-known/openid-configuration`: Returns the OIDC discovery document.
+  - `/openid/v1/jwks`: Returns the JSON Web Key Set (JWKS).
+- **Data Source**: These endpoints serve data uploaded by clients via the `DiscoveryEndpoint` resource.
+
+## Architecture
+
+### Project Structure
+- `cmd/discovery-server/`: Main entry point. Wires up the HTTP server with TLS configuration.
+- `pkg/discovery/`:
+  - `auth.go`: logic for inspecting TLS `PeerCertificates` to extract the Universe ID (CA hash) and Client ID.
+  - `store.go`: In-memory thread-safe storage (`MemoryStore`) mapping Universe IDs to lists of `DiscoveryEndpoint` objects.
+  - `server.go`: HTTP handlers implementing the K8s API emulation for `/apis/discovery.kops.k8s.io/v1alpha1`.
+  - `k8s_types.go`: Definitions of `DiscoveryEndpoint`, `DiscoveryEndpointList`, `TypeMeta`, `ObjectMeta` etc.
+
+### Data Model
+- **DiscoveryEndpoint**: The core resource. Contains `Spec.Addresses` and metadata.
+- **Universe**: Contains a map of `DiscoveryEndpoint` objects (keyed by name).
+- **Unified Types**: The API type `DiscoveryEndpoint` is used directly for in-memory storage, ensuring zero conversion overhead.
+
+## Security Model
+- **Trust Delegation**: The server delegates trust to the CA. If you hold the CA key, you control the Universe.
+- **Isolation**: The server ensures that a client presenting a cert chain for `CA_A` cannot read or write data to the Universe defined by `CA_B`.
+- **Ephemeral**: The current implementation uses in-memory storage. Data is lost on restart.
+
+## Building and Running
+
+### Build
+```bash
+go build ./cmd/discovery-server
+```
+
+### Run
+
+See docs/walkthrough.md for instructions on testing functionality.

discovery/README.md

@@ -0,0 +1,42 @@
+# Discovery Service
+
+A public discovery service using mTLS for authentication and "Universe" isolation, emulating a Kubernetes API.
+
+## Concept
+
+- **Universe**: Defined by the SHA256 Fingerprint of a Custom CA Certificate.
+- **Client**: Identified by a Client Certificate signed by that Custom CA.
+- **DiscoveryEndpoint**: The resource type representing a registered client.
+- **Isolation**: Clients can only see `DiscoveryEndpoint` objects signed by the same Custom CA.
+
+## Usage
+
+### Run Server
+
+```bash
+go run ./cmd/discovery-server --tls-cert server.crt --tls-key server.key --listen :8443
+```
+
+(You can generate a self-signed server certificate for testing, see the [walkthrough](docs/walkthrough.md) ).
+
+### Client Requirement
+
+Clients must authenticate using mTLS.
+**Important**: The client MUST provide the full certificate chain, including the Root CA, because the server does not have pre-configured trust stores for these custom universes.
+The server identifies the Universe from the SHA256 hash of the Root CA certificate found in the TLS chain.
+
+### Quick start
+
+See `docs/walkthrough.md` for detailed instructions.
+
+
+## OIDC Discovery
+
+The discovery server also serves OIDC discovery information publicly, allowing external systems (like AWS IAM) to discover the cluster's identity provider configuration.
+
+- `GET /<universe-id>/.well-known/openid-configuration`: Returns the OIDC discovery document.
+- `GET /<universe-id>/openid/v1/jwks`: Returns the JWKS.
+
+This information is populated by clients uploading `DiscoveryEndpoint` resources containing the `oidc` spec.
+
+## Building and Running

discovery/apis/discovery.kops.k8s.io/v1alpha1/types.go

@@ -0,0 +1,59 @@
+package v1alpha1
+
+import (
+	"k8s.io/apimachinery/pkg/runtime/schema"
+	metav1 "k8s.io/kops/discovery/apis/meta/v1"
+)
+
+var DiscoveryEndpointGVR = schema.GroupVersionResource{
+	Group:    "discovery.kops.k8s.io",
+	Version:  "v1alpha1",
+	Resource: "discoveryendpoints",
+}
+
+var DiscoveryEndpointGVK = schema.GroupVersionKind{
+	Group:   "discovery.kops.k8s.io",
+	Version: "v1alpha1",
+	Kind:    "DiscoveryEndpoint",
+}
+
+// DiscoveryEndpoint represents a registered client in the discovery service.
+type DiscoveryEndpoint struct {
+	metav1.TypeMeta   `json:",inline"`
+	metav1.ObjectMeta `json:"metadata,omitempty"`
+
+	Spec DiscoveryEndpointSpec `json:"spec,omitempty"`
+}
+
+// DiscoveryEndpointSpec corresponds to our internal Node data.
+type DiscoveryEndpointSpec struct {
+	Addresses []string  `json:"addresses,omitempty"`
+	LastSeen  string    `json:"lastSeen,omitempty"`
+	OIDC      *OIDCSpec `json:"oidc,omitempty"`
+}
+
+type OIDCSpec struct {
+	IssuerURL string       `json:"issuerURL,omitempty"`
+	Keys      []JSONWebKey `json:"keys,omitempty"`
+}
+
+type JSONWebKey struct {
+	Use       string `json:"use,omitempty"`
+	KeyType   string `json:"kty,omitempty"`
+	KeyID     string `json:"kid,omitempty"`
+	Algorithm string `json:"alg,omitempty"`
+	N         string `json:"n,omitempty"`
+	E         string `json:"e,omitempty"`
+	// Crv       string `json:"crv,omitempty"`
+	// X         string `json:"x,omitempty"`
+	// Y         string `json:"y,omitempty"`
+}
+
+// DiscoveryEndpointList is a list of DiscoveryEndpoint objects.
+type DiscoveryEndpointList struct {
+	metav1.TypeMeta `json:",inline"`
+	// Standard list metadata.
+	// We implement a minimal subset.
+	Metadata metav1.ListMeta     `json:"metadata,omitempty"`
+	Items    []DiscoveryEndpoint `json:"items"`
+}

discovery/apis/meta/v1/types.go

@@ -0,0 +1,58 @@
+package v1
+
+// TypeMeta describes an individual object in an API response or request
+// with strings representing the type of the object and its API schema version.
+// Structures that are versioned or persisted should inline TypeMeta.
+type TypeMeta struct {
+	Kind       string `json:"kind,omitempty"`
+	APIVersion string `json:"apiVersion,omitempty"`
+}
+
+// ObjectMeta is metadata that all persisted resources must have, which includes all objects
+// users must create.
+type ObjectMeta struct {
+	Name              string            `json:"name,omitempty"`
+	Namespace         string            `json:"namespace,omitempty"`
+	UID               string            `json:"uid,omitempty"`
+	ResourceVersion   string            `json:"resourceVersion,omitempty"`
+	CreationTimestamp string            `json:"creationTimestamp,omitempty"`
+	Labels            map[string]string `json:"labels,omitempty"`
+	Annotations       map[string]string `json:"annotations,omitempty"`
+}
+
+type ListMeta struct {
+	ResourceVersion string `json:"resourceVersion,omitempty"`
+	Continue        string `json:"continue,omitempty"`
+}
+
+// APIResourceList is used for discovery
+type APIResourceList struct {
+	TypeMeta     `json:",inline"`
+	GroupVersion string        `json:"groupVersion"`
+	Resources    []APIResource `json:"resources"`
+}
+
+type APIResource struct {
+	Name         string   `json:"name"`
+	SingularName string   `json:"singularName"`
+	Namespaced   bool     `json:"namespaced"`
+	Kind         string   `json:"kind"`
+	Verbs        []string `json:"verbs"`
+}
+
+// APIGroupList is used for root discovery (GET /apis).
+type APIGroupList struct {
+	TypeMeta `json:",inline"`
+	Groups   []APIGroup `json:"groups"`
+}
+
+type APIGroup struct {
+	Name             string                     `json:"name"`
+	Versions         []GroupVersionForDiscovery `json:"versions"`
+	PreferredVersion GroupVersionForDiscovery   `json:"preferredVersion"`
+}
+
+type GroupVersionForDiscovery struct {
+	GroupVersion string `json:"groupVersion"`
+	Version      string `json:"version"`
+}

discovery/cmd/discovery-server/main.go

@@ -0,0 +1,70 @@
+/*
+Copyright 2025 The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package main
+
+import (
+	"crypto/tls"
+	"flag"
+	"fmt"
+	"log"
+	"net/http"
+	"os"
+
+	"k8s.io/kops/discovery/pkg/discovery"
+)
+
+func main() {
+	certFile := flag.String("tls-cert", "", "Path to server TLS certificate")
+	keyFile := flag.String("tls-key", "", "Path to server TLS key")
+	addr := flag.String("listen", ":8443", "Address to listen on")
+	storageType := flag.String("storage", "memory", "Storage backend (memory, gcs)")
+	flag.Parse()
+
+	if *certFile == "" || *keyFile == "" {
+		fmt.Fprintf(os.Stderr, "Error: --tls-cert and --tls-key are required\n")
+		flag.Usage()
+		os.Exit(1)
+	}
+
+	var store discovery.Store
+
+	switch *storageType {
+	case "memory":
+		store = discovery.NewMemoryStore()
+	default:
+		log.Fatalf("Unknown storage type: %s", *storageType)
+	}
+
+	handler := discovery.NewServer(store)
+
+	tlsConfig := &tls.Config{
+		ClientAuth: tls.RequestClientCert,
+		// We do not set ClientCAs because we accept any CA and use it to define the universe.
+		MinVersion: tls.VersionTLS12,
+	}
+
+	server := &http.Server{
+		Addr:      *addr,
+		Handler:   handler,
+		TLSConfig: tlsConfig,
+	}
+
+	log.Printf("Discovery server listening on %s using %s storage", *addr, *storageType)
+	if err := server.ListenAndServeTLS(*certFile, *keyFile); err != nil {
+		log.Fatalf("Server failed: %v", err)
+	}
+}

discovery/docs/walkthrough.md

@@ -0,0 +1,60 @@
+# Walkthrough of functionality
+
+
+Quick start:
+
+```bash
+# Generate certs, kubeconfig, yaml files
+# Check out the script to better understand how all the pieces fit together!
+./scripts/create-kubeconfig.sh
+
+# Start Server (using generated server certs)
+go run ./cmd/discovery-server --tls-cert walkthrough/server.crt --tls-key walkthrough/server.key &
+
+# Verify server is running and serving the DiscoveryEndpoint resource
+kubectl --kubeconfig=walkthrough/universe1/client1.kubeconfig api-resources
+
+# List DiscoveryEndpoints
+kubectl --kubeconfig=walkthrough/universe1/client1.kubeconfig get discoveryendpoints --all-namespaces
+
+# Register (Apply)
+# The `metadata.name` MUST match the Common Name (CN) of your client certificate (e.g., `client1`), or the server will reject it with 403 Forbidden.
+kubectl --kubeconfig=walkthrough/universe1/client1.kubeconfig apply -f walkthrough/universe1/client1-discoveryendpoint.yaml  --server-side=true --validate=false
+
+# List DiscoveryEndpoints
+kubectl --kubeconfig=walkthrough/universe1/client1.kubeconfig get discoveryendpoints --all-namespaces
+```
+
+
+## Using curl
+
+The kubernetes API is a well-structured REST API, so we don't have to use kubectl.
+
+If you want to test the API with curl, you must include the **Universe ID** in the URL path.
+
+**Export the Universe ID:**
+```bash
+export UNIVERSE_ID=$(openssl x509 -in walkthrough/universe1/ca.crt -noout -fingerprint -sha256 | sed 's/SHA256 Fingerprint=//' | sed 's/://g' | tr '[:upper:]' '[:lower:]')
+echo "UNIVERSE_ID is ${UNIVERSE_ID}"
+```
+
+```bash
+curl --cert walkthrough/universe1/client1-bundle.crt --key walkthrough/universe1/client1.key --cacert walkthrough/server.crt \
+  "https://localhost:8443/${UNIVERSE_ID}/apis/discovery.kops.k8s.io/v1alpha1/namespaces/default/discoveryendpoints"
+```
+
+## OIDC discovery
+
+The goal here is to allow anonymous access to OIDC endpoints, so let's verify that:
+
+**Getting the OpenID Configuration**
+```bash
+curl --cacert walkthrough/server.crt  "https://localhost:8443/${UNIVERSE_ID}/.well-known/openid-configuration"
+```
+
+**Getting the JWKS keys**
+```bash
+curl --cacert walkthrough/server.crt  "https://localhost:8443/${UNIVERSE_ID}/openid/v1/jwks"
+```
+
+Note that we do not need a client certificate to get this data.  Data from the DiscoveryEndpoints is published publicly.

discovery/go.mod

@@ -0,0 +1,34 @@
+module k8s.io/kops/discovery
+
+go 1.25.4
+
+require (
+	k8s.io/apimachinery v0.34.3
+	k8s.io/client-go v0.34.3
+	k8s.io/klog/v2 v2.130.1
+)
+
+require (
+	github.com/davecgh/go-spew v1.1.1 // indirect
+	github.com/fxamacker/cbor/v2 v2.9.0 // indirect
+	github.com/go-logr/logr v1.4.3 // indirect
+	github.com/gogo/protobuf v1.3.2 // indirect
+	github.com/json-iterator/go v1.1.12 // indirect
+	github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
+	github.com/modern-go/reflect2 v1.0.3-0.20250322232337-35a7c28c31ee // indirect
+	github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
+	github.com/x448/float16 v0.8.4 // indirect
+	go.yaml.in/yaml/v2 v2.4.2 // indirect
+	golang.org/x/net v0.38.0 // indirect
+	golang.org/x/oauth2 v0.27.0 // indirect
+	golang.org/x/sys v0.31.0 // indirect
+	golang.org/x/term v0.30.0 // indirect
+	golang.org/x/text v0.23.0 // indirect
+	golang.org/x/time v0.9.0 // indirect
+	gopkg.in/inf.v0 v0.9.1 // indirect
+	k8s.io/utils v0.0.0-20250604170112-4c0f3b243397 // indirect
+	sigs.k8s.io/json v0.0.0-20241014173422-cfa47c3a1cc8 // indirect
+	sigs.k8s.io/randfill v1.0.0 // indirect
+	sigs.k8s.io/structured-merge-diff/v6 v6.3.0 // indirect
+	sigs.k8s.io/yaml v1.6.0 // indirect
+)