On the latest version of oauth2-proxy, I had to pass an additional flag under extraArgs for the issuer URL, like so:
extraArgs:
provider: oidc
oidc-issuer-url: "https://authentik.example.com/application/o/kube-apiserver/"
provider-display-name: "Authentik"
skip-provider-button: "true"
pass-authorization-header: "true"
session-store-type: cookie # Note I am not using redis
cookie-refresh: 15m