Shared Forwarding with Policy-Based Routing causes tcpdump to report invalid IPv4 header checksum

[VedantWani]

Important notices

• I have read the contributing guide lines at https://github.com/opnsense/src/blob/master/CONTRIBUTING.md
• I am convinced that my issue is new after having checked both open and closed issues at https://github.com/opnsense/src/issues?q=is%3Aissue

---

Describe the bug

When Shared Forwarding is enabled together with Policy-Based Routing (PBR) on a WAN gateway, tcpdump consistently reports an invalid IPv4 header checksum on packets captured on the PPPoE WAN interface.

The issue is only observed with the combination of:

• Shared Forwarding = Enabled
• Policy-Based Routing (gateway selected on firewall rule) = Enabled

The checksum warning disappears immediately when Shared Forwarding is disabled while keeping the same PBR rule.

Hardware checksum offloading, TSO and LRO are all disabled.

The packets continue to function normally (ICMP replies are received), so this may be a packet capture/BPF ordering issue rather than an actual transmission problem.

Observed on OPNsense 26.7-RC2.

---

To Reproduce

1. Install OPNsense 26.7-RC2.
2. Configure a PPPoE WAN interface.
3. Configure a LAN interface.
4. Disable:
   • Hardware Checksum Offloading
   • Hardware TCP Segmentation Offloading
   • Hardware Large Receive Offloading
5. Set Outbound NAT to Hybrid.
6. Enable Shared Forwarding under System → Settings → Networking.
7. Create a LAN firewall rule using Policy-Based Routing by selecting the WAN gateway.
8. Start a capture:

tcpdump -ni pppoe0 -vvv -xx icmp

9. Generate traffic, for example:

ping 9.9.9.11

or

traceroute 9.9.9.11

10. Observe the packet capture.
11. Disable Shared Forwarding while leaving the PBR rule unchanged.
12. Repeat the capture.

---

Expected behavior

tcpdump should report a valid IPv4 header checksum regardless of whether Shared Forwarding is enabled.

---

Actual behavior

With Shared Forwarding enabled:

IP (tos 0x0, ttl 9, id 41299, offset 0, flags [none], proto ICMP (1), length 64, bad cksum 0ed4 (->0fd4)!)
    <WAN-IP> > 9.9.9.11: ICMP echo request

Hex dump:

4500 0040 a153 0000 0901 0ed4

tcpdump recalculates the checksum as:

0x0fd4

The difference is consistently:

0x0100

which appears to correspond to the IPv4 TTL field.

The ICMP checksum is not reported as invalid.

Traffic itself appears to function normally.

---

Reproduction matrix

Shared Forwarding | Policy Routing | Result -- | -- | -- Disabled | Disabled | No checksum warning Disabled | Enabled | No checksum warning Enabled | Disabled | No checksum warning Enabled | Enabled | IPv4 header checksum reported as invalid by tcpdump

---

Describe alternatives you considered

• Disabled Hardware Checksum Offloading
• Disabled TSO
• Disabled LRO
• Verified the issue disappears when Shared Forwarding is disabled
• Verified the issue is independent of Outbound NAT mode (Hybrid used during testing)

---

Relevant log files

Example capture:

14:44:33.724226 IP (tos 0x0, ttl 9, id 41299, offset 0, flags [none], proto ICMP (1), length 64, bad cksum 0ed4 (->0fd4)!)
    <WAN-IP> > 9.9.9.11: ICMP echo request, id 35890, seq 33043, length 44
        0x0000: 0200 0000 4500 0040 a153 0000 0901 0ed4
        0x0010: 75d9 78a9 0909 090b 0800 eab9 8c32 8113

sysctl net.inet.ip.forwarding
net.inet.ip.forwarding: 1

---

Additional context

The checksum mismatch is consistently 0x0100 lower than the value calculated by tcpdump.

This suggests that tcpdump may be observing the packet after the IPv4 TTL has been modified but before the IPv4 header checksum has been updated, although I have not confirmed this in the kernel.

If this is expected due to the Shared Forwarding packet path, it would be helpful to clarify. Otherwise, this may indicate a regression in the interaction between Shared Forwarding, policy routing (route-to), and packet capture on PPPoE interfaces.

---

Environment

OPNsense 26.7-RC2 (amd64)
WAN: PPPoE

LAN: Ethernet
Outbound NAT: Hybrid
Shared Forwarding: Enabled
Hardware Checksum Offloading: Disabled

Hardware TSO: Disabled

Hardware LRO: Disabled

[fichtner]

For incoming packets? Sure. For outgoing? if checksumming deferred that's just what it is at the point of the tap.

[VedantWani]

The issue is when shared forwarding is enabled with policy based routing Gateway set in LAN rule to forward internet traffic to specific gateway, everything stops working.

For example

Doing mtr and tracert with shared forwarding enabled and PBR

                                                                                            Packets               Pings
 Host                                                                                      Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. _gateway                                                                                0.0%    15    0.7   0.7   0.4   1.2   0.2
 2. stranger.pqn.vwlabs.org                                                                 0.0%    14    1.7   2.1   1.6   3.4   0.5
 3. (waiting for reply)

tracert

Tracing route to dns.google [8.8.8.8]
over a maximum of 30 hops:

  1     1 ms     1 ms     1 ms  stranger.pqn.vwlabs.org [192.168.26.1]
  2     1 ms     1 ms     1 ms  stranger.pqn.vwlabs.org [192.168.26.1]
  3     *        *        *     Request timed out.
  4     *        *        *     Request timed out.
  5     *        *        *     Request timed out.
  6     *        *        *     Request timed out.
  7

Firewall Rule

For example

Doing mtr without shared forwarding enabled and with PBR

                                                                                             Packets               Pings
 Host                                                                                      Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. _gateway                                                                                0.0%    89    0.4   0.6   0.3   1.3   0.2
 2. stranger.pqn.vwlabs.org                                                                 0.0%    89    1.9   2.3   1.6   8.8   0.9
 3. triband-mum-59.184.208.1.mtnl.net.in                                                   94.3%    88    3.2  13.1   3.0  51.3  21.4
 4. 10.219.7.234                                                                            0.0%    88    4.0   5.7   3.1  70.8   7.6
 5. 218.248.255.22                                                                         89.7%    88   31.7  13.4   5.9  31.7   9.1
 6. (waiting for reply)
 7. 115.114.89.177.static-mumbai.vsnl.net.in                                                0.0%    88    9.1   9.7   8.1  28.6   2.9
 8. 172.28.176.221                                                                          0.0%    88    7.5   7.0   5.7  16.3   1.2
 9. 172.28.177.73                                                                          12.5%    88    8.8  11.2   7.1 188.9  21.4
10. 103.156.182.161                                                                         1.1%    88   57.8  77.7  17.6 129.0  26.0
11. dns11.quad9.net                                                                         0.0%    88   81.7  78.2  22.7 126.1  27.2

tracert

Tracing route to dns.google [8.8.8.8]
over a maximum of 30 hops:

  1     2 ms     2 ms     1 ms  stranger.pqn.vwlabs.org [192.168.26.1]
  2     *        *        *     Request timed out.
  3     3 ms     3 ms     3 ms  10.219.7.234
  4     *        *        *     Request timed out.
  5     *        *        *     Request timed out.
  6     5 ms     6 ms     5 ms  142.250.161.230
  7    10 ms     9 ms     9 ms  192.178.110.227
  8     9 ms     9 ms    10 ms  142.250.227.71
  9     8 ms     8 ms     8 ms  dns.google [8.8.8.8]

Trace complete.

Seems something is broken with shared forwarding enabled with PBR.

[fichtner]

And shared forwarding off works for the traffic? why not lead with packet flow disrupted?

[VedantWani]

Yes, turning shared forwarding off, traffic flows normally.

I m not sure what you mean by packet flow disrupted? Pretty new to debugging such issues at firewall level..

[fichtner]

bba71b3 may be relevant.. don't see the issue with shared forwarding patch at first glance 2e50d2a

[fichtner]

I don't mind reverting bba71b3 for now.. it never made its way to 26.1.x anyway

[VedantWani]

If you can give me patch I can quickly test and let you know.

[fichtner]

@VedantWani can you try this kernel?

# opnsense-update -zkr 26.7.r1-csum_ttl
(reboot)

[VedantWani]

@fichtner Didn't work. As soon as I turned on shared forwarding.. traffic flow breaks.
Double checked kernel is applied..

[VedantWani]

Whats weird is with shared forwarding enabled I see packet going twice to opnsense

Tracing route to dns.google [8.8.8.8]
over a maximum of 30 hops:

  1     1 ms     1 ms     1 ms  stranger.pqn.vwlabs.org [192.168.26.1]
  2     1 ms     1 ms     1 ms  stranger.pqn.vwlabs.org [192.168.26.1]

In earlier version, this was not the case.

Also in previous version, if shared forwarding was off with PBR, first hop is the WAN gateway and not the firewall itself.

[fichtner]

well first things first... is the TTL-changed checksum correct now? how many packets do you see in tcpdump?

[VedantWani]

checksum still shows bad.

[VedantWani]

Out of curiosity, I just tried setting WAN gateway (which has static IP instead of PPPoE) in PBR.

Turns out with shared forwarding enable no packet reaches the static IP WAN interface/vtnet2.

If I turn off shared forwarding, packet do flow normally via assigned gateway.