USHIFT-6452: WIP: enabling udn on ovn-kubernetes spike

[eslutsky]

enabling user defined network for multi-cluster communication

starting point : Microshift 4.22 build from PR sources, and Multus RPM installed

OVNK patching:

• custom build ovnk image src

• Deployment

  1. applying CRD manifests

      oc apply -f https://raw.githubusercontent.com/ovn-kubernetes/ovn-kubernetes/refs/heads/master/dist/templates/k8s.ovn.org_clusteruserdefinednetworks.yaml.j2
      oc apply -f https://raw.githubusercontent.com/ovn-kubernetes/ovn-kubernetes/refs/heads/master/dist/templates/k8s.ovn.org_userdefinednetworks.yaml.j2
  

  2. update RBAC rules

     	  oc patch clusterrole openshift-ovn-kubernetes-node --type='json' -p='[
        {
          "op": "add", 
          "path": "/rules/-", 
          "value": {
            "apiGroups":["k8s.ovn.org"],
            "resources":["userdefinednetworks","clusteruserdefinednetworks"],
            "verbs":["get","list","watch","patch","update"]
          }
        }
      ]'

  3. create NS

  cat <<EOF | oc apply -f -
  apiVersion: v1
  kind: Namespace
  metadata:
    name: udnspike
    labels:
      k8s.ovn.org/primary-user-defined-network: ""
  EOF
  

  4. create network

  cat <<EOF | oc apply -f -
  apiVersion: k8s.ovn.org/v1
  kind: ClusterUserDefinedNetwork
  metadata:
    name: netext1
  spec:
    namespaceSelector:
      matchExpressions:
      - key: kubernetes.io/metadata.name
        operator: In
        values: ["udnspike"]
    network:
      topology: Layer3
      layer3:
        role: Primary
        subnets:
          - cidr: 10.100.0.0/16
            hostSubnet: 24
  EOF

  5. Create pod in the NS with Primary network

  cat <<EOF | oc apply -f -
  apiVersion: v1
  kind: Pod
  metadata:
    name: network-debug-pod
    namespace: udnspike
  
  spec:
    securityContext:
      runAsNonRoot: true
      seccompProfile:
        type: RuntimeDefault
    containers:
    - name: debug-container
      # Using netshoot instead of the restricted redhat image
      image: docker.io/nicolaka/netshoot:latest
      command: ["/bin/sh", "-c", "sleep infinity"]
      securityContext:
        allowPrivilegeEscalation: false
        capabilities:
          drop:
          - ALL
        runAsUser: 10001    
  EOF
  

[root@i-08482c17698046dd0 ~]# oc get network-attachment-definitions -oyaml -A
apiVersion: v1
items:
- apiVersion: k8s.cni.cncf.io/v1
  kind: NetworkAttachmentDefinition
  metadata:
    annotations:
      k8s.ovn.org/network-id: "1"
      k8s.ovn.org/network-name: cluster_udn_netext1
      k8s.ovn.org/tunnel-keys: '[]'
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"k8s.ovn.org/v1","kind":"ClusterUserDefinedNetwork","metadata":{"annotations":{},"name":"netext1"},"spec":{"namespaceSelector":{"matchExpressions":[{"key":"kubernetes.io/metadata.name","operator":"In","values":["udnspike"]}]},"network":{"layer3":{"role":"Primary","subnets":[{"cidr":"10.100.0.0/16","hostSubnet":24}]},"topology":"Layer3"}}}
    creationTimestamp: "2026-01-20T13:10:38Z"
    finalizers:
    - k8s.ovn.org/user-defined-network-protection
    generation: 1
    labels:
      k8s.ovn.org/user-defined-network: ""
    name: netext1
    namespace: udnspike
    ownerReferences:
    - apiVersion: k8s.ovn.org/v1
      blockOwnerDeletion: true
      controller: true
      kind: ClusterUserDefinedNetwork
      name: netext1
      uid: a87bbaab-5743-4b63-bb33-8c99ff88ae20
    resourceVersion: "909"
    uid: 0dd56527-daab-4f2a-9f3c-d9102f44cfa9
  spec:
    config: '{"cniVersion":"1.0.0","joinSubnet":"100.65.0.0/16,fd99::/64","name":"cluster_udn_netext1","netAttachDefName":"udnspike/netext1","role":"primary","subnets":"10.100.0.0/16/24","topology":"layer3","type":"ovn-k8s-cni-overlay"}'
kind: List
metadata:
  resourceVersion: ""

inside the test pod:

2 IP Addresses while d.g is the UDN (10.100.0.1):
network-debug-pod:/root$ ip -br addr show
lo               UNKNOWN        127.0.0.1/8 ::1/128 
eth0@if13        UP             10.42.0.7/24 fe80::858:aff:fe2a:7/64 
ovn-udn1@if14    UP             10.100.0.3/24 fe80::858:aff:fe64:3/64 


network-debug-pod:/root$ ip route
default via 10.100.0.1 dev ovn-udn1 
10.42.0.0/24 dev eth0 proto kernel scope link src 10.42.0.7 
10.42.0.0/16 via 10.42.0.1 dev eth0 
10.43.0.0/16 via 10.100.0.1 dev ovn-udn1 
10.100.0.0/24 dev ovn-udn1 proto kernel scope link src 10.100.0.3 
10.100.0.0/16 via 10.100.0.1 dev ovn-udn1 
100.64.0.0/16 via 10.42.0.1 dev eth0 
100.65.0.0/16 via 10.100.0.1 dev ovn-udn1 
network-debug-pod:/root$ 

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: eslutsky

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [eslutsky]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[openshift-ci]

@eslutsky: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/ocp-full-conformance-rhel-eus	f75480f	link	true	/test ocp-full-conformance-rhel-eus
ci/prow/ocp-full-conformance-serial-rhel-eus	f75480f	link	true	/test ocp-full-conformance-serial-rhel-eus

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[openshift-ci-robot]

@eslutsky: This pull request references USHIFT-6452 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the spike to target the "4.22.0" version, but no target version was set.

Details

In response to this:

enabling user defined network for multi-cluster communication

starting point : Microshift 4.22 build from PR sources, and Multus RPM installed

OVNK patching:

• custom build ovnk image src

• Deployment

1. applying CRD manifests

    oc apply -f https://raw.githubusercontent.com/ovn-kubernetes/ovn-kubernetes/refs/heads/master/dist/templates/k8s.ovn.org_clusteruserdefinednetworks.yaml.j2
   oc apply -f https://raw.githubusercontent.com/ovn-kubernetes/ovn-kubernetes/refs/heads/master/dist/templates/k8s.ovn.org_userdefinednetworks.yaml.j2

2. update RBAC rules

   	  oc patch clusterrole openshift-ovn-kubernetes-node --type='json' -p='[
     {
       "op": "add", 
       "path": "/rules/-", 
       "value": {
         "apiGroups":["k8s.ovn.org"],
         "resources":["userdefinednetworks","clusteruserdefinednetworks"],
         "verbs":["get","list","watch","patch","update"]
       }
     }
   ]'

3. create NS

cat <<EOF | oc apply -f -
apiVersion: v1
kind: Namespace
metadata:
  name: udnspike
  labels:
    k8s.ovn.org/primary-user-defined-network: ""
EOF

4. create network

cat <<EOF | oc apply -f -
apiVersion: k8s.ovn.org/v1
kind: ClusterUserDefinedNetwork
metadata:
  name: netext1
spec:
  namespaceSelector:
    matchExpressions:
    - key: kubernetes.io/metadata.name
      operator: In
      values: ["udnspike"]
  network:
    topology: Layer3
    layer3:
      role: Primary
      subnets:
        - cidr: 10.100.0.0/16
          hostSubnet: 24
EOF

5. Create pod in the NS with Primary network

cat <<EOF | oc apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: network-debug-pod
  namespace: udnspike

spec:
  securityContext:
    runAsNonRoot: true
    seccompProfile:
      type: RuntimeDefault
  containers:
  - name: debug-container
    # Using netshoot instead of the restricted redhat image
    image: docker.io/nicolaka/netshoot:latest
    command: ["/bin/sh", "-c", "sleep infinity"]
    securityContext:
      allowPrivilegeEscalation: false
      capabilities:
        drop:
        - ALL
      runAsUser: 10001    
EOF

[root@i-08482c17698046dd0 ~]# oc get network-attachment-definitions -oyaml -A
apiVersion: v1
items:
- apiVersion: k8s.cni.cncf.io/v1
 kind: NetworkAttachmentDefinition
 metadata:
   annotations:
     k8s.ovn.org/network-id: "1"
     k8s.ovn.org/network-name: cluster_udn_netext1
     k8s.ovn.org/tunnel-keys: '[]'
     kubectl.kubernetes.io/last-applied-configuration: |
       {"apiVersion":"k8s.ovn.org/v1","kind":"ClusterUserDefinedNetwork","metadata":{"annotations":{},"name":"netext1"},"spec":{"namespaceSelector":{"matchExpressions":[{"key":"kubernetes.io/metadata.name","operator":"In","values":["udnspike"]}]},"network":{"layer3":{"role":"Primary","subnets":[{"cidr":"10.100.0.0/16","hostSubnet":24}]},"topology":"Layer3"}}}
   creationTimestamp: "2026-01-20T13:10:38Z"
   finalizers:
   - k8s.ovn.org/user-defined-network-protection
   generation: 1
   labels:
     k8s.ovn.org/user-defined-network: ""
   name: netext1
   namespace: udnspike
   ownerReferences:
   - apiVersion: k8s.ovn.org/v1
     blockOwnerDeletion: true
     controller: true
     kind: ClusterUserDefinedNetwork
     name: netext1
     uid: a87bbaab-5743-4b63-bb33-8c99ff88ae20
   resourceVersion: "909"
   uid: 0dd56527-daab-4f2a-9f3c-d9102f44cfa9
 spec:
   config: '{"cniVersion":"1.0.0","joinSubnet":"100.65.0.0/16,fd99::/64","name":"cluster_udn_netext1","netAttachDefName":"udnspike/netext1","role":"primary","subnets":"10.100.0.0/16/24","topology":"layer3","type":"ovn-k8s-cni-overlay"}'
kind: List
metadata:
 resourceVersion: ""

inside the test pod:

2 IP Addresses while d.g is the UDN (10.100.0.1):
network-debug-pod:/root$ ip -br addr show
lo               UNKNOWN        127.0.0.1/8 ::1/128 
eth0@if13        UP             10.42.0.7/24 fe80::858:aff:fe2a:7/64 
ovn-udn1@if14    UP             10.100.0.3/24 fe80::858:aff:fe64:3/64 


network-debug-pod:/root$ ip route
default via 10.100.0.1 dev ovn-udn1 
10.42.0.0/24 dev eth0 proto kernel scope link src 10.42.0.7 
10.42.0.0/16 via 10.42.0.1 dev eth0 
10.43.0.0/16 via 10.100.0.1 dev ovn-udn1 
10.100.0.0/24 dev ovn-udn1 proto kernel scope link src 10.100.0.3 
10.100.0.0/16 via 10.100.0.1 dev ovn-udn1 
100.64.0.0/16 via 10.42.0.1 dev eth0 
100.65.0.0/16 via 10.100.0.1 dev ovn-udn1 
network-debug-pod:/root$ 

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.