No meshnet gRPC connections made / timeout - ceos multi-host lab

[mgisch]

I'm trying to setup a 2 node ceos lab using the topology from the examples (https://github.com/openconfig/kne/blob/main/examples/arista/ceos-150/ceos-150.pb.txt) scaled down to only 4 nodes for now wired in a ring like that example. ceos version is 4.33.1F if that is relevant.

The control-plane node has pod placement enabled.
When I force all pods to place on the same node (either one) everything works.
When they're split, any pods that happen to have both neighbors placed on same node will start but all pods with a neighbor on the other node will be stuck in init state forever. ie - all cross-node links fail to be created and the init-wait image never completes.

Event log contains many of these:
26s Warning FailedCreatePodSandBox pod/r3 (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_r3_ceos-4_a70c8de5-bf1e-446d-9d34-517bf09fa6f6_0(7e6f3ad44bc0105b7320f290222a9b837ce650585d2d5ab659849ba7f0248f1f): error adding pod ceos-4_r3 to CNI network "cbr0": plugin type="meshnet" name="meshnet" failed (add): rpc error: code = Unavailable desc = failed to receive server preface within timeout

In earlier testing that log additionally contained the other nodes address:51111 reference but after a recent reboot it now only shows this generic timeout error; I'm not sure what changed but the symptom is the same either way.

Based on tcpdump tests neither node ever sends a port 51111 packet on the wire.
Port 51111 is allowed in both host firewalls, the symptom is the same with firewall disabled and using nc to test connectivity I can connect to that port on both nodes from each other by both IP and hostname. A tcpdump on loopback shows many packets but nothing on the outside interface.
Something is preventing it from even attempting to make these grpc connections to the other node.

Kubernetes version is 1.28. OS is Oracle Linux 9. Selinux is disabled.
KNE was installed exactly as per these instructions: https://github.com/openconfig/kne/blob/main/docs/multinode.md but on a local bare metal cluster. Meshnet daemonset image says it's version 0.3.2, meshnet binary says it's version 0.3.0. Flannel appears to work (can ssh to pods on remote node)

Any ideas on how to troubleshoot further to make inter-host grpc links work?
Is vxlan mode still supported as alternative? Re-applying meshnet vxlan manifest appeared to change nothing...meshnet still starts in grpc mode.

Thank you.

[chrisy]

Do you have any excerpts from the meshnet daemon logs from both kube nodes? My guess is that they're not discovering each other so never get to the step where they try to talk to each other. I would start by restarting the daemonset and observing those logs.

You mention pod placement is enabled; what do you mean? Have you configured something beyond the defaults? Meshnet daemons specify a nodeAddinity rule that places one meshnet pod per kube node. Is it possible something you have configured interferes with this?

[mgisch]

Hi Chris,

These are the meshnet logs from both nodes after I restarted them and then tried to start up the kne lab.

grpc-fail-meshnet-1.log
grpc-fail-meshnet-2.log

There is one per node running (redacted real hostnames):

# kubectl get pods -n meshnet -o wide
NAME            READY   STATUS    RESTARTS   AGE   IP             NODE                            NOMINATED NODE   READINESS GATES
meshnet-jzj4d   1/1     Running   0          16h   100.73.91.38   node1   <none>           <none>
meshnet-k2w7s   1/1     Running   0          16h   100.73.91.37   node2   <none>           <none>

By enabling pod placement on the control node I just mean this:
kubectl taint node node1 node-role.kubernetes.io/control-plane:NoSchedule-

My cni config files are as they came:

# cat 00-meshnet.conflist 
{
	"cniVersion": "0.3.0",
	"name": "cbr0",
	"plugins": [
		{
			"delegate": {
				"hairpinMode": true,
				"isDefaultGateway": true
			},
			"type": "flannel"
		},
		{
			"capabilities": {
				"portMappings": true
			},
			"type": "portmap"
		},
		{
			"name": "meshnet",
			"type": "meshnet",
			"ipam": {},
			"dns": {}
		}
	]
}
# cat 10-flannel.conflist 
{
  "name": "cbr0",
  "cniVersion": "0.3.1",
  "plugins": [
    {
      "type": "flannel",
      "delegate": {
        "hairpinMode": true,
        "isDefaultGateway": true
      }
    },
    {
      "type": "portmap",
      "capabilities": {
        "portMappings": true
      }
    }
  ]
}
# cat meshnet-inter-node-link-type
VXLAN

That last one may be because I tried to install the vxlan version of the meshnet manifest recently...but it doesn't seem to honor that (and didn't work before I did that either).

These hosts started out fairly locked down due to corp policy (rpm-ostree, r/o root file system, firewalld, selinux...). It's plausible there is something else lingering that I havn't unblocked yet but not sure where to look next.
Appreciate any advice on what you would check.
Thanks.

[mgisch]

Does it need dns for some part of this? There might be something funky going on with that.
The meshnet containers have hostNetwork: true so inherit the base OS dns settings (corp servers) which cannot resolve things like meshnet.meshnet.svc.cluster.local (and take > 10s to decide not to find an answer).
Tried manually updating resolv.conf in the meshnet containers to make that work:

# kubectl exec -n meshnet meshnet-8s7kt -- getent hosts meshnet.meshnet.svc.cluster.local
100.73.91.37    meshnet.meshnet.svc.cluster.local
100.73.91.38    meshnet.meshnet.svc.cluster.local

but original problem remains.

[chrisy]

Does it need dns for some part of this?

# kubectl exec -n meshnet meshnet-8s7kt -- getent hosts meshnet.meshnet.svc.cluster.local
100.73.91.37    meshnet.meshnet.svc.cluster.local
100.73.91.38    meshnet.meshnet.svc.cluster.local

I do not believe so; mine seems to have only the hosts /etc files. So far as I can tell, it learns the destination IP address for tunnels (vxlan or grpc) from the HOST_IP environment variable which is specified in the daemonset configuration.

Your log files seem sparse. Mine are much chattier, I've attached one for reference -- it's from one Kube node in a 16-node cluster, when I launched a 60-node topology. Perhaps it will help to see the log file of a "working" system.

meshnet.txt

[mgisch]

Think I figured this out. My CRI-O had proxy settings configured via systemd environment and whichever process/container was trying to make the off-host gRPC connections was trying to use a corp proxy server to do so. This failed and explains why I saw no port 51111 packets in tcpdump; they were being turned into CONNECT requests via the proxy.
Fixed with suitable no_proxy entry and meshnet grpc links appear to work now.
Thank you Chris for your time to take a look.

[chrisy]

Think I figured this out. My CRI-O had proxy settings configured via systemd environment and whichever process/container was trying to make the off-host gRPC connections was trying to use a corp proxy server to do so. This failed and explains why I saw no port 51111 packets in tcpdump; they were being turned into CONNECT requests via the proxy. Fixed with suitable no_proxy entry and meshnet grpc links appear to work now. Thank you Chris for your time to take a look.

Oh, interesting. I too use an HTTP proxy for retrieving images, and have it as systemd config for containerd, and already had a no_proxy entry that included the address space used by pod networking (among others), so I never encountered this:

[Service]
{% if http_proxy is defined %}
Environment=http_proxy={{ http_proxy }}
Environment=https_proxy={{ http_proxy }}
Environment=no_proxy=.svc,.<corpdomains>,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16,100.64.0.0/10,127.0.0.0/8,<controllerhostname>
{% endif %}

I'll file this under "good to know", and likely forget it within a week! :)