pfSense & OPNsense Bufferbloat Elimination: CAKE vs FQ-CoDel Queueing Disciplines on Asymmetric Gigabit WAN

1. Diagnostic Matrix & Triage Telemetry

2. Architecture & Root-Cause Mechanics

Bufferbloat is the high latency and jitter induced when network equipment buffers excessive amounts of data packets in first-in, first-out (FIFO) queues. When a network connection becomes saturated—most commonly on the upstream of asymmetric broadband connections such as DOCSIS cable (e.g., 1000 Mbps down / 35 Mbps up) or VDSL—the physical network interface on the ISP modem or local router attempts to prevent packet loss by accumulating packets in large memory buffers. While buffering prevents dropped packets, it introduces devastating propagation delays. Latency-critical UDP packets (such as game physics updates, Discord voice audio, and DNS lookups) become stuck behind massive 1500-byte TCP file transfer frames. Latency spikes from a nominal 15ms to over 600ms, creating severe packet queuing delay. To permanently eradicate bufferbloat, modern open-source firewalls like pfSense (using the `dummynet` / `ipfw` / `altq` framework) and OPNsense implement Smart Queue Management (SQM): 1. **FQ-CoDel (Fair Queueing Controlled Delay):** Divides network traffic into multiple sub-queues using a 1024-bucket hash table based on 5-tuple flow hashing (Source IP, Dest IP, Source Port, Dest Port, Protocol). FQ-CoDel tracks the 'sojourn time'—how long a packet sits in a queue. If the sojourn time exceeds a target threshold (typically 5ms) over an interval (typically 100ms), FQ-CoDel proactively drops or ECN-marks packets to signal the sender's TCP congestion control algorithm (BBR, CUBIC) to throttle back before the physical queue overflows. 2. **CAKE (Common Applications Kept Enhanced):** The cutting-edge successor to FQ-CoDel developed by the bufferbloat research team. CAKE combines fair queueing, active queue management (CoDel), and multi-tier priority queueing (DiffServ) into a single integrated algorithm. Furthermore, CAKE includes automatic framing overhead compensation (for DOCSIS, ADSL, VDSL, Ethernet VLANs), preventing overhead leakage that undermines traditional shapers. By artificially constraining your firewall's interface throughput to approximately 90-95% of your true physical bandwidth ceiling, your firewall becomes the strict bottleneck of the network. The ISP modem buffers never fill, ensuring zero packet accumulation and sub-5ms loaded latency.

3. Safety Protocols & Bench Equipment

Adhere strictly to standard lab bench precautions and ESD guidelines before opening devices:

• Never apply a traffic shaper rule without testing your base unshaped line speed via a hardwired Ethernet connection directly to the modem.
• Keep firewall CPU temperatures monitored when running CAKE on high-bandwidth links; CAKE computational overhead is roughly 2.5x higher than simple tail-drop FIFOs.
• Do not enable Hardware Offloading (TSO, LRO) on pfSense interfaces when using ALTQ or dummynet limiters, as large coalesced packets bypass queue hashing.
• Record your current firewall XML configuration backup before deploying new IPFW pipe and queue architectures.

4. Method 1: Baseline Line Capacity & Bufferbloat Benchmarking

Accurate bufferbloat mitigation begins by establishing absolute baseline metrics under sustained synthetic load: 1. **Perform the Waveform Bufferbloat Test:** - Connect a PC directly to the firewall via a Cat6 Ethernet cable. Ensure all other network activity is halted. - Navigate to `waveform.com/tools/bufferbloat`. - Run the test. Note the **Unloaded Latency**, **Download Active Latency**, and **Upload Active Latency**. - A score of C, D, or F indicates severe bufferbloat, with active latency climbing by +50ms to +500ms above baseline. 2. **Calculate Initial Shaper Targets:** - Take your consistent minimum upload and download bandwidth results. - For Fiber connections (symmetric, low baseline jitter): Multiply both download and upload by **0.95** (95% provisioned rate). - For Cable/DOCSIS connections (asymmetric, high burst jitter): Multiply download by **0.92** and upload by **0.88** to **0.90**. - Example: On a 1000/40 Mbps cable line, your target shaper rates will be **920 Mbps Download** and **36 Mbps Upload**.

5. Method 2: Configure pfSense Multi-Queue FQ-CoDel Limiters

Implement low-overhead, multi-threaded FQ-CoDel limiters in pfSense: 1. **Create the Download & Upload Limiters:** - Navigate to `Firewall` > `Traffic Shaper` > `Limiters` tab. Click `New Limiter`. - **In_Limiter (Download):** - Name: `WAN_Download_Pipe` - Bandwidth: Enter target rate (e.g., `920` `Mbit/s`). - Mask: None. - AQM: `CoDel`. - Queue Management: `FQ_CODEL`. - Click `Save`. - Under `WAN_Download_Pipe`, click `Add new Queue`. - Name: `WAN_Download_Queue` - Mask: `Destination addresses`. - AQM: `CoDel`. - Queue Management: `FQ_CODEL`. - Click `Save`. 2. **Create the Upload Limiter & Queue:** - Click `New Limiter`. - **Out_Limiter (Upload):** - Name: `WAN_Upload_Pipe` - Bandwidth: Enter target rate (e.g., `36` `Mbit/s`). - Mask: None. - AQM: `CoDel`. - Queue Management: `FQ_CODEL`. - Click `Save`. - Under `WAN_Upload_Pipe`, click `Add new Queue`. - Name: `WAN_Upload_Queue` - Mask: `Source addresses`. - AQM: `CoDel`. - Queue Management: `FQ_CODEL`. - Click `Save`. 3. **Apply Limiters to Firewall Rules:** - Navigate to `Firewall` > `Rules` > `LAN` tab. - Edit your default `LAN to Any` allow rule (IPv4). - Scroll down to `Advanced Options` > `In / Out pipe`. - In the first dropdown (In), select `WAN_Upload_Queue`. - In the second dropdown (Out), select `WAN_Download_Queue`. - Click `Save` and `Apply Changes`.

6. Method 3: Deploy CAKE SQM with Overhead Compensation in OPNsense

OPNsense features native CAKE support inside its modern Dummynet scheduler, providing exceptional framing compensation: 1. **Configure Pipes in OPNsense:** - Navigate to `Firewall` > `Traffic Shaper` > `Pipes`. - Click `+` to add a new pipe for Download: - Bandwidth: `920` `Mbit/s` - Scheduler Type: `FlowQueue-CoDel` or `Cake` (if using FreeBSD 14 / OPNsense 24.x+ kernel). - Under CAKE options: - Framing: Select `docsis` for cable, `ethernet` for fiber, or `ptm` for VDSL. - Overhead: Enter `18` (for standard DOCSIS Ethernet encapsulation). - Tin Mode: Select `diffserv4` (automatically creates 4 priority buckets based on DSCP tags: Bulk, Best Effort, Video, Voice). - Click `Save`. Repeat for the Upload pipe (e.g., `36` `Mbit/s`, framing `docsis`). 2. **Create Queues and Rules:** - In `Queues`, create child queues mapping destination IP hashing for download and source IP hashing for upload. - In `Rules`, create matching rules assigning LAN inbound traffic to the Upload pipe, and WAN inbound traffic to the Download pipe. - Apply changes and restart the shaping engine via `Apply`.

7. Method 4: Disable Hardware Offloading & Verify Latency Under Saturation

Hardware Large Receive Offload (LRO) and TCP Segmentation Offload (TSO) aggregate multiple network frames into single giant 64KB buffers before passing them to the CPU. This breaks packet-level queue scheduling: 1. **Disable Offloading:** - In pfSense / OPNsense, go to `System` > `Advanced` > `Networking`. - Check the following boxes to **disable** offloading: - `Disable Hardware Checksum Offload` - `Disable Hardware TCP Segmentation Offload (TSO)` - `Disable Hardware Large Receive Offload (LRO)` - Click `Save` and reboot the firewall appliance. 2. **Verification & Fine Tuning:** - Open a persistent terminal ping to Cloudflare DNS: ```cmd ping 1.1.1.1 -t ``` - Launch a saturated multi-stream upload test (e.g., upload a large 10GB file to Google Drive or run a Speedtest upload). - Observe your continuous ping: - If ping remains within +1ms to +3ms of idle ping (e.g., rising from 12ms to only 14ms), your shaper is perfectly calibrated (Grade A+ Bufferbloat). - If ping jumps by +30ms or more, lower your upload limiter rate by another 3% until the spike vanishes.

8. Real-World Field Case Studies

9. Prevention & Long-Term Reliability

Follow these preventative bench routines to maintain hardware longevity and prevent recurrent failures:

• Re-run bufferbloat benchmarks every 6 months to detect ISP profile changes or line reprovisioning.
• Ensure firewall storage is healthy; excessive firewall logging under heavy load can starve CPU cycles needed for shaper pacing.
• Audit connected devices quarterly to ensure unmanaged switches do not drop 802.1p CoS priority tags.
• If your ISP upgrades your bandwidth plan, immediately update the shaper pipe capacities in firewall settings.

10. Frequently Asked Bench Questions (FAQ)