Azure CNI (dynamic pod subnets) as primary Site with manageCniPlugin=false: no return path for pod traffic in mesh, gateway hijacks remote node CIDR #921

Summary

Attaching an AKS Flex Node to an existing AKS cluster that uses Azure CNI with dynamic pod subnets (networkPlugin=azure, networkPolicy=calico, podCidr=null, one pod subnet per node pool) using unbounded-net with manageCniPlugin: false on the cluster Site. Cross-site traffic from Azure CNI pods to Flex pods does not work in either mesh or gateway mode without a manual SNAT rule. Is this topology supported, and if so, what is the intended configuration?

Environment

• unbounded v0.12.0 (operator, net-controller, net-node), AKSFlexNode v0.2.0, AKS 1.36.3 (private cluster)
• Cluster: Azure CNI, dynamic pod subnets per pool, Calico policy-only, podCidr empty, service CIDR 10.244.0.0/16
• Example ranges (anonymized): VNet 10.0.0.0/16; one AKS pool with node subnet 10.0.0.0/27 and pod subnet 10.0.2.0/23; Flex node 192.168.10.10 in 192.168.10.0/28; Flex pod CIDR 192.168.20.0/24
• Private L3 between the VNet and the Flex network (stateful firewall in the path). Node to node works both ways, including UDP 6081.
• unbounded-net-node limited via unbounded-component-overrides to the Flex node plus one AKS node (canary)

Configuration

apiVersion: unbounded-cloud.io/v1alpha3
kind: Site
metadata: {name: cluster}
spec:
  manageCniPlugin: false
  nodeCidrs: [10.0.0.0/27]
  podCidrAssignments:
    - cidrBlocks: [10.0.2.0/23]
  localCidrs: [10.0.0.0/16]
  nonMasqueradeCIDRs: [0.0.0.0/0]
---
apiVersion: unbounded-cloud.io/v1alpha3
kind: Site
metadata: {name: flex-site}
spec:
  nodeCidrs: [192.168.10.0/28]
  podCidrAssignments:
    - cidrBlocks: [192.168.20.0/24]
      nodeBlockSizes: {ipv4: 25}
  nonMasqueradeCIDRs: [192.168.20.0/24]
---
apiVersion: net.unbounded-cloud.io/v1alpha1
kind: SitePeering
metadata: {name: cluster-flex}
spec:
  sites: [cluster, flex-site]
  meshNodes: true
  tunnelProtocol: GENEVE

Observed behavior

Mesh (above):

• The AKS canary gets spec.podCIDR 10.0.12.0/24 (outside the declared cluster pool) and unbounded0 with source 10.0.12.1, and the route 192.168.20.0/25 dev unbounded0.
• Azure CNI pod 10.0.2.128 pings Flex pod 192.168.20.6. tcpdump on the AKS node shows the request entering geneve0 and being encapsulated to 192.168.10.10:6081. There is no reply, and the Flex side shows nothing for inner source 10.0.2.128. Agent health checks 10.0.12.1:9997 <-> 192.168.20.1:9997 work both ways.
• With a manual rule on the AKS node, iptables -t nat -I POSTROUTING -o unbounded0 -d 192.168.20.0/24 -j MASQUERADE (SNAT to 10.0.12.1), ping and TCP from the Azure CNI pod to the Flex pod work.
• So it looks like the Flex side only returns or accepts traffic for sources inside the peer's Unbounded pod block. Azure CNI pod IPs are never in those blocks.

Gateway (GatewayPool type Internal, routedCidrs: [10.0.2.0/23], SiteGatewayPoolAssignments for both sites, SitePeering removed):

• The gateway node gets 192.168.10.0/28 dev unbounded0, i.e. the remote Site's node CIDR.
• Its underlay route to the tunnel endpoint 192.168.10.10 then goes into unbounded0, and the tunnel breaks (node to Flex node ping fails).

Related, already fixed: with v0.8.0 the cluster Site pod pool was routed into unbounded0 on every agent node, black-holing kubelet to API on Azure CNI nodes. #740 (localCidrs) fixes that in v0.12.0.

Questions

1. Is an existing Azure CNI cluster with per-pool pod subnets a supported primary site for manageCniPlugin: false? hack/scripts/get-aks-cluster-cidrs.sh rejects clusters without networkProfile.podCidr.
2. Is there a supported way to send unmanaged-CNI pod traffic into the tunnel with a correct return path? For example, SNAT to the node's unbounded0 address, or routing the cluster Site's pod CIDR back via mesh peers.
3. In gateway mode, how should nodeCidrs and localCidrs be set so the gateway does not route the remote Site's node IPs (including the tunnel endpoint) through unbounded0?

We can provide more captures or test patches.