los93sol

Your tutorials are great! I have this setup and working, but one thing I’m struggling with is I have an ingress (ingress-nginx) for the kube-apiserver that gets an IP from metallb, updates my internal dns, and gets a LetsEncrypt certificate for it. This all works really well except kubectl exec and kubectl port-forwards timeout after 1 minute. I’ve tried all the proxy-send-timeout and proxy-read-timeout options. I’ve even manually installed k3s 1.30 and kubectl 1.30 and enabled PortForwardWebsockets and verified it works as well, but I still get the same issue. Hoping someone else is experiencing the same and knows how to resolve it.

1 reply

los93sol

I ended up figuring this out. The problem was caused by asymmetric routing. I had k8s nodes on LAN. If I used kubectl from a VLAN it was fine, but when accessing from LAN is where I would see the issue. I found a few ways to resolve it.

Option 1 - Create a FW rule in opnsense with state tracking set to sloppy. This works but is not great because you will never be able to have a stateful rule for k8s.

Option 2 - Create a NAT rule in opnsense to force all traffic for metallb IPs through a specific gateway. This works well, but has a side effect of hiding the real IP when it comes into the cluster.

Option 3 - Move the k8s nodes to a dedicated VLAN. This is what I did and it does not seem to come with any side effects.