Gateway Pod Connections
Verify nodes in the cluster(s) are labeled:
kubectl get nodes --show-labels | grep "avesha/node-type=gateway"
If nodes are properly labeled:
Verify the presence of a slice gateway object on the cluster:
kubectl get slicegw -n avesha-system
If the api key and access token are valid and correct:
Verify the presence of the slice router pod beginning with prefix “vl3-nse-<slice_name>-*” in its name is present and in RUNNING state.
kubectl get po -n avesha-system |grep vl3-nse-<slice name>
vl3-nse-slice-iperf-7f9c696776-58jlf 2/2 Running 0 2d2h
If it is not present or in any state other than RUNNING:
Verify if all pods with prefix “nsm-*” are in RUNNING state.
kubectl get po -n avesha-system |grep nsm
nsm-admission-webhook-6f7d8897c9-p78sv 1/1 Running 0 2d3h
nsm-kernel-forwarder-bh6xd 1/1 Running 0 2d3h
nsm-kernel-forwarder-zvg6p 1/1 Running 0 2d3h
nsmgr-mj6g6 3/3 Running 0 2d3h
nsmgr-nndd8 3/3 Running 0 2d3h
If the vl3 and nsm pods are missing or not in a RUNNING state:
kubectl get pods -n avesha-system | grep controller
kubectl logs <avesha-mesh-controller-manager-xxxx> -c manager -n avesha-system
kubectl get pods -n avesha-system | grep nsm
kubectl logs nsm-* --all-containers -n avesha-system
If vl3 and nsm pod are in a RUNNING state:
Check if the REMOTE SUBNET and the REMOTE CLUSTER fields are populated in the slice gateway object.
kubectl get slicegw -n avesha-system
NAME SUBNET REMOTE SUBNET REMOTE CLUSTER GW STATUS
slice-iperf-s1-6da494a6 10.6.2.0/24 SLICE_GATEWAY_STATUS_CERTS_PENDING
If the REMOTE SUBNET and the REMOTE CLUSTER columns are populated:
Check the GW STATUS field. If it says SLICE_GATEWAY_STATUS_CERTS_PENDING, wait a few minutes for the connection to establish and the status to change to SLICE_GATEWAY_STATUS_REGISTERED.
kubectl get slicegw -n avesha-system
NAME SUBNET REMOTE SUBNET REMOTE CLUSTER GW STATUS
slice-iperf-s1-6da494a6 10.6.2.0/24 10.6.1.0/24 6ace0341-7e80-4dcb-be83-aeb3dd783e46 SLICE_GATEWAY_STATUS_CERTS_PENDING
kubectl get slicegw -n avesha-system
NAME SUBNET REMOTE SUBNET REMOTE CLUSTER GW STATUS
slice-iperf-s1-6da494a6 10.6.2.0/24 10.6.1.0/24 6ace0341-7e80-4dcb-be83-aeb3dd783e46 SLICE_GATEWAY_STATUS_REGISTERED
If the GW STATUS is SLICE_GATEWAY_STATUS_REGISTERED:
Verify gateway pods are present:
There are both Server and Client gateway pods.
Server gateway pods are prefixed with <slice_name>-s1-*
Client gateway pods are prefixed with <slice_name>-c1-s1-*
kubectl get pods -n avesha-system | grep c1
kubectl get pods -n avesha-system | grep s1
kubectl get pods -n avesha-system | grep controller
kubectl logs <avesha-mesh-controller-manager-xxxx> -c manager -n avesha-system
kubectl describe slice -n avesha-system
kubectl describe slicegw -n avesha-system
Identify the cluster that is running the gateway vpn server:
kubectl get pods -n avesha-system | grep s1
kubectl describe slicegw <slice name>-s1* |grep "Slice Gateway Host Type:"
Check if the node port service to reach the VPN server is created on the cluster:
kubectl get svc -n avesha-system |grep svc-<name_of_slice>
kubectl logs <slice gateway pod name> --all-containers -n avesha-system
kubectl logs vl3-nse-<name_of_slice>-<> --all-containers -n avesha-system
If the above validations pass, change the context to the VPN client cluster and follow these steps:
Check if the headless service to reach the remote vpn server is configured:
kubectl get svc -n avesha-system |grep <slice name>
Check if an endpoint has been populated for the service:
kubectl exec -it <slice gateway pod> -c avesha-sidecar -n avesha-system — nslookup <name of the service obtained in previous step>
kubectl get pods -n avesha-system
kubectl logs <avesha-mesh-controller-manager-xxxx> -c manager -n avesha-system
kubectl logs <slice gateway pod name> --all-containers -n avesha-system
kubectl logs vl3-nse-<name_of_slice>-<> --all-containers -n avesha-system
If the service is present:
Check if a tun interface is created in the gateway pod:
kubectl get pods -n avesha-system | grep s1
kubectl exec -it <slice name>-c1-s1-* -c avesha-sidecar -- ip a | grep tun0
If all the validations on the status of the slice gateway and the vpn tunnel pass:
kubectl get pods -n avesha-system
kubectl logs <avesha-mesh-controller-manager-xxxx> -c manager -n avesha-system
kubectl logs <slice gateway pod name> -c avesha-sidecar -n avesha-system
kubectl logs <slice gateway pod name> -c avesha-openvpn-client -n avesha-system
Application Unreachable
Slice Configuration
Annotations
Check if the slice is installed on the cluster:
kubectl get slice <slice name> -n avesha-system
Verify slice annotation is present in the application deployment:
kubectl describe deployment <deployment name> -n <namespace> | grep Annotations -A 5
If the annotation is properly configured:
Review slice configuration for any Error/Warning events:
kubectl describe slice < slice name > -n avesha-system
kubectl get pods -n avesha-system | grep controller
kubectl logs <avesha-mesh-controller-manager-xxxx> -c manager -n avesha-system
kubectl describe slice <name_of_the_slice> -n avesha-system
Webhooks
If there are no errors in the slice configuration:
Check if admission web hooks are present:
kubectl get mutatingwebhookconfigurations
The following web hooks must be present:
avesha-admission-webhook-cfg, nsm-admission-webhook-cfg
Check if the web hook services are installed:
kubectl get svc -n avesha-system
The following services should be present:
avesha-admission-webhook-svc, nsm-admission-webhook-svc
To remediate:
Delete the application deployment
Uninstall the slice
Uninstall avesha-mesh
Re-install avesha-mesh
Verify the webhooks are installed correctly
Re-install the slice
Redeploy the application with the slice annotation
Namespace
Check the namespaceIsolationProfile block in the slice configuration.
kubectl get slice <slice name> -n avesha-system -o=jsonpath="{..status.config.namespaceIsolationProfile.isolationEnabled}"
true
kubectl get slice <slice name> -n avesha-system -o=jsonpath="{..status.config.namespaceIsolationProfile.applicationNamespaces}"
["<cluster name>:<slice name>"]
namespaceIsolationProfile:
isolationEnabled: true
applicationNamespaces:
- “<cluster name>:<namespace>”
Delete the application deployment
Uninstall the slice on all the clusters
Re-install the slice with the application namespace listed in the namespaceIsolationProfile config block of the slice yaml file
Redeploy the application with the slice annotation.
kubectl get pods -n avesha-system | grep controller
kubectl logs <avesha-mesh-controller-manager-xxxx> -c manager -n avesha-system
kubectl get pods -n avesha-system | grep nsm
kubectl logs <nsm-admission-webhook-xxxx> -n avesha-system