Merge pull request #1756 from XJangel/ha

Add docs for CloudCore HA deployment

build/cloud/ha/02-ha-configmap.yaml.example

+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: cloudcore
+  namespace: kubeedge
+  labels:
+    k8s-app: kubeedge
+    kubeedge: cloudcore
+data:
+  cloudcore.yaml: |
+    apiVersion: cloudcore.config.kubeedge.io/v1alpha1
+    kind: CloudCore
+    kubeAPIConfig:
+      kubeConfig: ""
+      master: ""
+    leaderelection:
+      LeaderElect: true
+      LeaseDuration: 15s
+      RenewDeadline: 10s
+      ResourceLock: endpointsleases
+      ResourceName: cloudcorelease
+      ResourceNamespace: kubeedge
+      RetryPeriod: 2s
+    modules:
+      cloudHub:
+        advertiseAddress:
+         - YOUR CLOUDCORE VIP HERE !!!
+        nodeLimit: 10
+        tlsCAFile: /etc/kubeedge/ca/rootCA.crt
+        tlsCertFile: /etc/kubeedge/certs/server.crt
+        tlsPrivateKeyFile: /etc/kubeedge/certs/server.key
+        unixsocket:
+          address: unix:///var/lib/kubeedge/kubeedge.sock
+          enable: true
+        websocket:
+          address: 0.0.0.0
+          enable: true
+          port: 10000
+      cloudStream:
+         enable: false
+         streamPort: 10003
+         tlsStreamCAFile: /etc/kubeedge/ca/streamCA.crt
+         tlsStreamCertFile: /etc/kubeedge/certs/stream.crt
+         tlsStreamPrivateKeyFile: /etc/kubeedge/certs/stream.key
+         tlsTunnelCAFile: /etc/kubeedge/ca/rootCA.crt
+         tlsTunnelCertFile: /etc/kubeedge/certs/server.crt
+         tlsTunnelPrivateKeyFile: /etc/kubeedge/certs/server.key
+         tunnelPort: 10004

build/cloud/ha/ha-deployment.yaml → build/cloud/ha/03-ha-deployment.yaml.example

@@ -18,6 +18,8 @@ spec:
         k8s-app: kubeedge
         kubeedge: cloudcore
     spec:
+      nodeSelector: # configure the nodeSelector here to directly schedule pods to specific nodes
+        [key]: [value]
       hostNetwork: true
       readinessGates:
         - conditionType: "kubeedge.io/CloudCoreIsLeader"
@@ -39,12 +41,15 @@ spec:
               topologyKey: kubernetes.io/hostname
       containers:
         - name: cloudcore
-          image: kubeedge/cloudcore:v1.3.0
+          image: kubeedge/cloudcore:{tag}
           imagePullPolicy: IfNotPresent
           ports:
             - containerPort: 10000
               name: cloudhub
               protocol: TCP
+            - containerPort: 10002
+              name: certAndReadyz
+              protocol: TCP
           resources:
             limits:
               cpu: 200m
@@ -55,8 +60,6 @@ spec:
           volumeMounts:
             - name: conf
               mountPath: /etc/kubeedge/config
-            - name: certs
-              mountPath: /etc/kubeedge
           env:
             - name: CLOUDCORE_POD_NAME
               valueFrom:
@@ -73,13 +76,4 @@ spec:
         - name: conf
           configMap:
             name: cloudcore
-        - name: certs
-          secret:
-            secretName: cloudcore
-            items:
-              - key: edge.crt
-                path: certs/edge.crt
-              - key: edge.key
-                path: certs/edge.key
-              - key: rootCA.crt
-                path: ca/rootCA.crt
+

build/cloud/ha/README.md

+# The HA of CloudCore(deployed in k8s cluster)
+
+**Note:**
+
+There are several ways to achieve the HA of cloudcore, for example, ingress, keepalived etc. Here we adopt the keepalived. The HA of cloudcore according to ingress will be achieved later.
+
+## Determine the virtual IP of CloudCore
+
+Determine a VIP that the CloudCore service exposed to the edge nodes. Here we recommend `keepalived` to do that. You had better directly schedule pods to specific number of nodes by `nodeSelector` when using  `keepalived`. And you have  to install `keepalived` in each of nodes where CloudCore runs. The configuration of `keepalived` is shown in the end. Here suppose the VIP is 10.10.102.242.
+
+The use of `nodeSelector` is as follow:
+
+```bash
+kubectl label nodes [nodename] [key]=[value]  # label the nodes where the cloudcore will run
+```
+
+modify the term of `nodeselector`:
+
+```yaml
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: cloudcore
+spec:
+  template:
+    spec:
+      nodeSelector: # configure the nodeSelector here!
+        [key]: [value]
+```
+
+## Create k8s resources
+
+The manifests and scripts in `github.com/kubeedge/kubeedge/build/cloud/ha` will be used, so place these files to somewhere you can kubectl with (You have to make some modifications to manifests/scrips to suit your environment.)
+
+First, ensure your k8s cluster can pull cloudcore image. If the image not exist. We can make one, and push to your registry.
+
+```bash
+cd $GOPATH/src/github.com/kubeedge/kubeedge
+make cloudimage
+```
+
+We create k8s resources from the manifests in name order. Before creating, **check the content of each manifest to make sure it meets your environment.**
+
+**Note:** Now the follow manifests don't support `kubectl logs` command yet. If need, you have to make more configuration manually.
+
+### 02-ha-configmap.yaml
+
+Configure the VIP address of CloudCore which is exposed to the edge nodes in the `advertiseAddress`, which will be added to SANs in cert of CloudCore. For example:
+
+```yaml
+modules:
+  cloudHub:
+    advertiseAddress:
+    - 10.10.102.242
+```
+
+**Note:** If you want to reset the CloudCore, run this before creating k8s resources:
+
+```bash
+kubectl delete namespace kubeedge
+```
+
+Then create k8s resources:
+
+```shell
+cd build/cloud/ha
+for resource in $(ls *.yaml); do kubectl create -f $resource; done
+```
+
+## keepalived
+
+The `keepalived` configuration we recommend is as following. You can adjust it according to your needs.
+
+**keepalived.conf:**
+
+- master:
+
+```yaml
+! Configuration File for keepalived
+
+global_defs {
+  router_id lb01
+  vrrp_mcast_group4 224.0.0.19
+}
+# CloudCore
+vrrp_script CloudCore_check {
+  script "/etc/keepalived/check_cloudcore.sh" # the script for health check
+  interval 2
+  weight 2
+  fall 2
+  rise 2
+}
+vrrp_instance CloudCore {
+  state MASTER
+  interface eth0 # based on your host
+  virtual_router_id 167
+  priority 100
+  advert_int 1
+  authentication {
+    auth_type PASS
+    auth_pass 1111
+  }
+  virtual_ipaddress {
+    10.10.102.242/24 # VIP
+  }
+  track_script {
+    CloudCore_check
+ }
+}
+```
+
+- backup:
+
+```yaml
+! Configuration File for keepalived
+
+global_defs {
+  router_id lb02
+  vrrp_mcast_group4 224.0.0.19
+}
+# CloudCore
+vrrp_script CloudCore_check {
+  script "/etc/keepalived/check_cloudcore.sh" # the script for health check
+  interval 2
+  weight 2
+  fall 2
+  rise 2
+}
+vrrp_instance CloudCore {
+  state BACKUP
+  interface eth0 # based on your host
+  virtual_router_id 167
+  priority 99
+  advert_int 1
+  authentication {
+    auth_type PASS
+    auth_pass 1111
+  }
+  virtual_ipaddress {
+    10.10.102.242/24 # VIP
+  }
+  track_script {
+    CloudCore_check
+ }
+}
+```
+
+check_cloudcore.sh:
+
+```shell
+#!/usr/bin/env bash
+http_code=`curl -k -o /dev/null -s -w %{http_code} https://127.0.0.1:10002/readyz`
+if [ $http_code == 200 ]; then
+    exit 0
+else
+    exit 1
+fi
+
+```
+

build/cloud/ha/ha-clusterrole.yaml

-apiVersion: rbac.authorization.k8s.io/v1
-kind: ClusterRole
-metadata:
-  name: cloudcore
-  labels:
-    k8s-app: kubeedge
-    kubeedge: cloudcore
-rules:
-- apiGroups: [""]
-  resources: ["nodes", "nodes/status", "configmaps", "pods", "pods/status", "secrets", "endpoints", "services"]
-  verbs: ["get", "list", "watch", "create", "update"]
-- apiGroups: [""]
-  resources: ["pods"]
-  verbs: ["delete", "patch"]
-- apiGroups: ["coordination.k8s.io"]
-  resources: ["leases"]
-  verbs: ["get", "list", "watch", "create", "update"]
-- apiGroups: ["devices.kubeedge.io"]
-  resources: ["devices", "devicemodels", "devices/status", "devicemodels/status"]
-  verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
-- apiGroups: ["reliablesyncs.kubeedge.io"]
-  resources: ["objectsyncs", "clusterobjectsyncs", "objectsyncs/status", "clusterobjectsyncs/status"]
-  verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]

build/tools/certgen.sh

@@ -49,6 +49,7 @@ genCertAndKey() {
 }
 
 stream() {
+    ensureFolder
     readonly streamsubject=${SUBJECT:-/C=CN/ST=Zhejiang/L=Hangzhou/O=KubeEdge}
     readonly STREAM_KEY_FILE=${certPath}/stream.key
     readonly STREAM_CSR_FILE=${certPath}/stream.csr