Use Calico in CKS

[yadvr]

Weave project are looking for maintainers, it may be worth exploring what CNI is widely used and standard/stable for CKS use-case.

For Cilium: #9304

ISSUE TYPE

• Improvement Request

COMPONENT NAME

CKS

[jkacou]

Actually cilium is way more interesting and extensible as a CNI
To have both will be nice, but if I have to choose it will be cilium clearly but I will say it should be something configurable
We have plenty use case where we need to experiment a specific CNI before switch pro on it, so have it configurable will help a lot
BTW do we have any plan about this one please ?

[Awirbl]

With the announcement just made by WeaveWorks, they're going out of business. I think it would be a good idea to take a closer look at an alternative. As a cilium user, I think it would be a good idea to offer this natively in CKS (I already do it via Cloudstack ClusterAPI on my homelab).

[weizhouapache]

With the announcement just made by WeaveWorks, they're going out of business. I think it would be a good idea to take a closer look at an alternative. As a cilium user, I think it would be a good idea to offer this natively in CKS (I already do it via Cloudstack ClusterAPI on my homelab).

We could support multiple CNIs, users can choose the CNI.

Actually Now we can create CKS iso using other CNIs, no one has tested it yet.

[kiranchavala]

@weizhouapache @rohityadavcloud @jkacou

I have tested with Calico CNI the k8s cluster creation working fine.

https://docs.tigera.io/calico/latest/about/

https://github.com/projectcalico/calico/blob/master/manifests/calico.yaml

---

With Flannel CNI the cluster creation was failing

1. Execute the create-kubernetes-binaries-iso.sh script wih the following parametes

./create-kubernetes-binaries-iso.sh ./ 1.29.1 1.2.0 1.26.0 https://raw.githubusercontent.com/flannel-io/flannel/master/Documentation/kube-flannel.yml https://raw.githubusercontent.com/kubernetes/dashboard/v2.7.0/aio/deploy/recommended.yaml setup-1.29.1

2. Register the Kunbernetes ISO

3. Lanch a k8s cluster

4. The cluster fails to come up

root@k2-control-18d9bbfe065:/opt/cloud/bin# kubectl logs -n kube-flannel -f kube-flannel-ds-zxv85
Defaulted container "kube-flannel" out of: kube-flannel, install-cni-plugin (init), install-cni (init)
I0212 05:50:47.071491       1 main.go:209] CLI flags config: {etcdEndpoints:http://127.0.0.1:4001,http://127.0.0.1:2379 etcdPrefix:/coreos.com/network etcdKeyfile: etcdCertfile: etcdCAFile: etcdUsername: etcdPassword: version:false kubeSubnetMgr:true kubeApiUrl: kubeAnnotationPrefix:flannel.alpha.coreos.com kubeConfigFile: iface:[] ifaceRegex:[] ipMasq:true ifaceCanReach: subnetFile:/run/flannel/subnet.env publicIP: publicIPv6: subnetLeaseRenewMargin:60 healthzIP:0.0.0.0 healthzPort:0 iptablesResyncSeconds:5 iptablesForwardRules:true netConfPath:/etc/kube-flannel/net-conf.json setNodeNetworkUnavailable:true}
W0212 05:50:47.071870       1 client_config.go:617] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0212 05:50:47.111138       1 kube.go:139] Waiting 10m0s for node controller to sync
I0212 05:50:47.111565       1 kube.go:461] Starting kube subnet manager
I0212 05:50:48.112838       1 kube.go:146] Node controller sync successful
I0212 05:50:48.113029       1 main.go:229] Created subnet manager: Kubernetes Subnet Manager - k2-node-18d9bc019c5
I0212 05:50:48.113056       1 main.go:232] Installing signal handlers
I0212 05:50:48.114338       1 main.go:540] Found network config - Backend type: vxlan
I0212 05:50:48.114574       1 match.go:210] Determining IP address of default interface
I0212 05:50:48.117010       1 match.go:263] Using interface with name eth0 and address 10.1.1.8
I0212 05:50:48.117128       1 match.go:285] Defaulting external address to interface address (10.1.1.8)
I0212 05:50:48.117695       1 vxlan.go:141] VXLAN config: VNI=1 Port=0 GBP=false Learning=false DirectRouting=false
I0212 05:50:48.124242       1 kube.go:621] List of node(k2-node-18d9bc019c5) annotations: map[string]string{"kubeadm.alpha.kubernetes.io/cri-socket":"unix:///var/run/containerd/containerd.sock", "node.alpha.kubernetes.io/ttl":"0", "volumes.kubernetes.io/controller-managed-attach-detach":"true"}
E0212 05:50:48.127081       1 main.go:332] Error registering network: failed to acquire lease: node "k2-node-18d9bc019c5" pod cidr not assigned
I0212 05:50:48.127652       1 main.go:520] Stopping shutdownHandler..

It would be ideal if we could provide the option of selecting the CNI during the CKS Cluster creation

[weizhouapache]

@weizhouapache @rohityadavcloud @jkacou

I have tested with Calico CNI the k8s cluster creation working fine.

https://docs.tigera.io/calico/latest/about/

https://github.com/projectcalico/calico/blob/master/manifests/calico.yaml

---

With Flannel CNI the cluster creation was failing

1. Execute the create-kubernetes-binaries-iso.sh script wih the following parametes

./create-kubernetes-binaries-iso.sh ./ 1.29.1 1.2.0 1.26.0 https://raw.githubusercontent.com/flannel-io/flannel/master/Documentation/kube-flannel.yml https://raw.githubusercontent.com/kubernetes/dashboard/v2.7.0/aio/deploy/recommended.yaml setup-1.29.1

2. Register the Kunbernetes ISO

3. Lanch a k8s cluster

4. The cluster fails to come up

root@k2-control-18d9bbfe065:/opt/cloud/bin# kubectl logs -n kube-flannel -f kube-flannel-ds-zxv85
Defaulted container "kube-flannel" out of: kube-flannel, install-cni-plugin (init), install-cni (init)
I0212 05:50:47.071491       1 main.go:209] CLI flags config: {etcdEndpoints:http://127.0.0.1:4001,http://127.0.0.1:2379 etcdPrefix:/coreos.com/network etcdKeyfile: etcdCertfile: etcdCAFile: etcdUsername: etcdPassword: version:false kubeSubnetMgr:true kubeApiUrl: kubeAnnotationPrefix:flannel.alpha.coreos.com kubeConfigFile: iface:[] ifaceRegex:[] ipMasq:true ifaceCanReach: subnetFile:/run/flannel/subnet.env publicIP: publicIPv6: subnetLeaseRenewMargin:60 healthzIP:0.0.0.0 healthzPort:0 iptablesResyncSeconds:5 iptablesForwardRules:true netConfPath:/etc/kube-flannel/net-conf.json setNodeNetworkUnavailable:true}
W0212 05:50:47.071870       1 client_config.go:617] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0212 05:50:47.111138       1 kube.go:139] Waiting 10m0s for node controller to sync
I0212 05:50:47.111565       1 kube.go:461] Starting kube subnet manager
I0212 05:50:48.112838       1 kube.go:146] Node controller sync successful
I0212 05:50:48.113029       1 main.go:229] Created subnet manager: Kubernetes Subnet Manager - k2-node-18d9bc019c5
I0212 05:50:48.113056       1 main.go:232] Installing signal handlers
I0212 05:50:48.114338       1 main.go:540] Found network config - Backend type: vxlan
I0212 05:50:48.114574       1 match.go:210] Determining IP address of default interface
I0212 05:50:48.117010       1 match.go:263] Using interface with name eth0 and address 10.1.1.8
I0212 05:50:48.117128       1 match.go:285] Defaulting external address to interface address (10.1.1.8)
I0212 05:50:48.117695       1 vxlan.go:141] VXLAN config: VNI=1 Port=0 GBP=false Learning=false DirectRouting=false
I0212 05:50:48.124242       1 kube.go:621] List of node(k2-node-18d9bc019c5) annotations: map[string]string{"kubeadm.alpha.kubernetes.io/cri-socket":"unix:///var/run/containerd/containerd.sock", "node.alpha.kubernetes.io/ttl":"0", "volumes.kubernetes.io/controller-managed-attach-detach":"true"}
E0212 05:50:48.127081       1 main.go:332] Error registering network: failed to acquire lease: node "k2-node-18d9bc019c5" pod cidr not assigned
I0212 05:50:48.127652       1 main.go:520] Stopping shutdownHandler..

It would be ideal if we could provide the option of selecting the CNI during the CKS Cluster creation

great, thanks @kiranchavala
I remember @Pearl1594 is working /will work on the support for multiple CNIs and other CKS improvements.

[weizhouapache]

@kiranchavala
Can you also test the creation of a nginx service with type: LoadBalancer ?
Just to ensure CCM works.

[kiranchavala]

@weizhouapache Just checked with type : LoadBalancer on Calico CNI and its failing to allocate a public ip address

W0214 10:16:25.316085       1 reflector.go:405] pkg/mod/k8s.io/client-go@v0.18.18/tools/cache/reflector.go:125: watch of *v1.ConfigMap ended with: an error on the server ("unable to decode an event from the watch stream: http2: client connection lost") has prevented the request from succeeding
W0214 10:16:25.316238       1 reflector.go:405] pkg/mod/k8s.io/client-go@v0.18.18/tools/cache/reflector.go:125: watch of *v1.ConfigMap ended with: an error on the server ("unable to decode an event from the watch stream: http2: client connection lost") has prevented the request from succeeding
E0214 10:16:28.208078       1 leaderelection.go:324] error retrieving resource lock kube-system/cloud-controller-manager: Get https://10.96.0.1:443/api/v1/namespaces/kube-system/endpoints/cloud-controller-manager: http2: client connection lost
E0214 10:16:32.220693       1 event.go:316] Could not construct reference to: '&v1.Lease{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cloud-controller-manager", GenerateName:"", Namespace:"kube-system", SelfLink:"", UID:"711a72c5-31dc-4ee0-bbd4-67c988641693", ResourceVersion:"1539", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63843502224, loc:(*time.Location)(0x30c2040)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry{v1.ManagedFieldsEntry{Manager:"cloudstack-ccm", Operation:"Update", APIVersion:"coordination.k8s.io/v1", Time:(*v1.Time)(0xc0001a7f20), FieldsType:"FieldsV1", FieldsV1:(*v1.FieldsV1)(0xc0001a7f60)}}}, Spec:v1.LeaseSpec{HolderIdentity:(*string)(nil), LeaseDurationSeconds:(*int32)(nil), AcquireTime:(*v1.MicroTime)(nil), RenewTime:(*v1.MicroTime)(nil), LeaseTransitions:(*int32)(nil)}}' due to: 'no kind is registered for the type v1.Lease in scheme "pkg/runtime/scheme.go:101"'. Will not report event: 'Normal' 'LeaderElection' 'cloud-controller-manager-574bcb86c-fw2l2_bd39e3c6-084a-41df-bd7a-8c920cac61cb became leader'
I0214 10:16:32.221165       1 leaderelection.go:252] successfully acquired lease kube-system/cloud-controller-manager
I0214 10:16:32.223601       1 node_controller.go:110] Sending events to api server.
I0214 10:16:32.223706       1 controllermanager.go:247] Started "cloud-node"
I0214 10:16:32.226055       1 node_lifecycle_controller.go:78] Sending events to api server
I0214 10:16:32.313288       1 event.go:278] Event(v1.ObjectReference{Kind:"Endpoints", Namespace:"kube-system", Name:"cloud-controller-manager", UID:"7c0b707b-34f5-4a5c-b3e4-6fbc5d9bce32", APIVersion:"v1", ResourceVersion:"1538", FieldPath:""}): type: 'Normal' reason: 'LeaderElection' cloud-controller-manager-574bcb86c-fw2l2_bd39e3c6-084a-41df-bd7a-8c920cac61cb became leader
I0214 10:16:32.315475       1 controllermanager.go:247] Started "cloud-node-lifecycle"
I0214 10:16:32.513923       1 controllermanager.go:247] Started "service"
I0214 10:16:32.514493       1 core.go:101] Will not configure cloud provider routes for allocate-node-cidrs: false, configure-cloud-routes: true.
W0214 10:16:32.514805       1 controllermanager.go:244] Skipping "route"
I0214 10:16:32.516421       1 controller.go:208] Starting service controller
I0214 10:16:32.516452       1 shared_informer.go:223] Waiting for caches to sync for service
I0214 10:16:32.815530       1 shared_informer.go:230] Caches are synced for service
I0214 10:16:32.819667       1 event.go:278] Event(v1.ObjectReference{Kind:"Service", Namespace:"default", Name:"nginx-deployment2", UID:"725a3133-a5f2-4554-8dd7-b1e6138ec260", APIVersion:"v1", ResourceVersion:"1488", FieldPath:""}): type: 'Normal' reason: 'EnsuringLoadBalancer' Ensuring load balancer
E0214 10:16:48.754653       1 leaderelection.go:324] error retrieving resource lock kube-system/cloud-controller-manager: Get https://10.96.0.1:443/api/v1/namespaces/kube-system/endpoints/cloud-controller-manager: context deadline exceeded
E0214 10:16:48.754911       1 event.go:316] Could not construct reference to: '&v1.Lease{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cloud-controller-manager", GenerateName:"", Namespace:"kube-system", SelfLink:"", UID:"711a72c5-31dc-4ee0-bbd4-67c988641693", ResourceVersion:"1554", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63843502224, loc:(*time.Location)(0x30c2040)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry{v1.ManagedFieldsEntry{Manager:"cloudstack-ccm", Operation:"Update", APIVersion:"coordination.k8s.io/v1", Time:(*v1.Time)(0xc00000d960), FieldsType:"FieldsV1", FieldsV1:(*v1.FieldsV1)(0xc00000d980)}}}, Spec:v1.LeaseSpec{HolderIdentity:(*string)(nil), LeaseDurationSeconds:(*int32)(nil), AcquireTime:(*v1.MicroTime)(nil), RenewTime:(*v1.MicroTime)(nil), LeaseTransitions:(*int32)(nil)}}' due to: 'no kind is registered for the type v1.Lease in scheme "pkg/runtime/scheme.go:101"'. Will not report event: 'Normal' 'LeaderElection' 'cloud-controller-manager-574bcb86c-fw2l2_bd39e3c6-084a-41df-bd7a-8c920cac61cb stopped leading'
I0214 10:16:48.755078       1 leaderelection.go:277] failed to renew lease kube-system/cloud-controller-manager: timed out waiting for the condition
F0214 10:16:48.755210       1 controllermanager.go:213] leaderelection lost
 

[weizhouapache]

@weizhouapache Just checked with type : LoadBalancer on Calico CNI and its failing to allocate a public ip address

W0214 10:16:25.316085       1 reflector.go:405] pkg/mod/k8s.io/client-go@v0.18.18/tools/cache/reflector.go:125: watch of *v1.ConfigMap ended with: an error on the server ("unable to decode an event from the watch stream: http2: client connection lost") has prevented the request from succeeding
W0214 10:16:25.316238       1 reflector.go:405] pkg/mod/k8s.io/client-go@v0.18.18/tools/cache/reflector.go:125: watch of *v1.ConfigMap ended with: an error on the server ("unable to decode an event from the watch stream: http2: client connection lost") has prevented the request from succeeding
E0214 10:16:28.208078       1 leaderelection.go:324] error retrieving resource lock kube-system/cloud-controller-manager: Get https://10.96.0.1:443/api/v1/namespaces/kube-system/endpoints/cloud-controller-manager: http2: client connection lost
E0214 10:16:32.220693       1 event.go:316] Could not construct reference to: '&v1.Lease{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cloud-controller-manager", GenerateName:"", Namespace:"kube-system", SelfLink:"", UID:"711a72c5-31dc-4ee0-bbd4-67c988641693", ResourceVersion:"1539", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63843502224, loc:(*time.Location)(0x30c2040)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry{v1.ManagedFieldsEntry{Manager:"cloudstack-ccm", Operation:"Update", APIVersion:"coordination.k8s.io/v1", Time:(*v1.Time)(0xc0001a7f20), FieldsType:"FieldsV1", FieldsV1:(*v1.FieldsV1)(0xc0001a7f60)}}}, Spec:v1.LeaseSpec{HolderIdentity:(*string)(nil), LeaseDurationSeconds:(*int32)(nil), AcquireTime:(*v1.MicroTime)(nil), RenewTime:(*v1.MicroTime)(nil), LeaseTransitions:(*int32)(nil)}}' due to: 'no kind is registered for the type v1.Lease in scheme "pkg/runtime/scheme.go:101"'. Will not report event: 'Normal' 'LeaderElection' 'cloud-controller-manager-574bcb86c-fw2l2_bd39e3c6-084a-41df-bd7a-8c920cac61cb became leader'
I0214 10:16:32.221165       1 leaderelection.go:252] successfully acquired lease kube-system/cloud-controller-manager
I0214 10:16:32.223601       1 node_controller.go:110] Sending events to api server.
I0214 10:16:32.223706       1 controllermanager.go:247] Started "cloud-node"
I0214 10:16:32.226055       1 node_lifecycle_controller.go:78] Sending events to api server
I0214 10:16:32.313288       1 event.go:278] Event(v1.ObjectReference{Kind:"Endpoints", Namespace:"kube-system", Name:"cloud-controller-manager", UID:"7c0b707b-34f5-4a5c-b3e4-6fbc5d9bce32", APIVersion:"v1", ResourceVersion:"1538", FieldPath:""}): type: 'Normal' reason: 'LeaderElection' cloud-controller-manager-574bcb86c-fw2l2_bd39e3c6-084a-41df-bd7a-8c920cac61cb became leader
I0214 10:16:32.315475       1 controllermanager.go:247] Started "cloud-node-lifecycle"
I0214 10:16:32.513923       1 controllermanager.go:247] Started "service"
I0214 10:16:32.514493       1 core.go:101] Will not configure cloud provider routes for allocate-node-cidrs: false, configure-cloud-routes: true.
W0214 10:16:32.514805       1 controllermanager.go:244] Skipping "route"
I0214 10:16:32.516421       1 controller.go:208] Starting service controller
I0214 10:16:32.516452       1 shared_informer.go:223] Waiting for caches to sync for service
I0214 10:16:32.815530       1 shared_informer.go:230] Caches are synced for service
I0214 10:16:32.819667       1 event.go:278] Event(v1.ObjectReference{Kind:"Service", Namespace:"default", Name:"nginx-deployment2", UID:"725a3133-a5f2-4554-8dd7-b1e6138ec260", APIVersion:"v1", ResourceVersion:"1488", FieldPath:""}): type: 'Normal' reason: 'EnsuringLoadBalancer' Ensuring load balancer
E0214 10:16:48.754653       1 leaderelection.go:324] error retrieving resource lock kube-system/cloud-controller-manager: Get https://10.96.0.1:443/api/v1/namespaces/kube-system/endpoints/cloud-controller-manager: context deadline exceeded
E0214 10:16:48.754911       1 event.go:316] Could not construct reference to: '&v1.Lease{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cloud-controller-manager", GenerateName:"", Namespace:"kube-system", SelfLink:"", UID:"711a72c5-31dc-4ee0-bbd4-67c988641693", ResourceVersion:"1554", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63843502224, loc:(*time.Location)(0x30c2040)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry{v1.ManagedFieldsEntry{Manager:"cloudstack-ccm", Operation:"Update", APIVersion:"coordination.k8s.io/v1", Time:(*v1.Time)(0xc00000d960), FieldsType:"FieldsV1", FieldsV1:(*v1.FieldsV1)(0xc00000d980)}}}, Spec:v1.LeaseSpec{HolderIdentity:(*string)(nil), LeaseDurationSeconds:(*int32)(nil), AcquireTime:(*v1.MicroTime)(nil), RenewTime:(*v1.MicroTime)(nil), LeaseTransitions:(*int32)(nil)}}' due to: 'no kind is registered for the type v1.Lease in scheme "pkg/runtime/scheme.go:101"'. Will not report event: 'Normal' 'LeaderElection' 'cloud-controller-manager-574bcb86c-fw2l2_bd39e3c6-084a-41df-bd7a-8c920cac61cb stopped leading'
I0214 10:16:48.755078       1 leaderelection.go:277] failed to renew lease kube-system/cloud-controller-manager: timed out waiting for the condition
F0214 10:16:48.755210       1 controllermanager.go:213] leaderelection lost
 

thanks @kiranchavala
same error as I tested before. could be a bug of CCM, or there are some settings missed for Calico in the control node.