“We swapped the CPE router. Swapped the fiber patch leads. Cleaned the SFP connectors. I was about to blame solar flares when I decided to look at the error counters.”

In May 2011, while working as a Field Engineer at Spectranet, a premium enterprise customer raised a critical escalation regarding packet loss.

The Mystery

I initiated a continuous ping test to their customer premises equipment (CPE). The result was a steady 2% packet loss.

CPU utilization on the router was sitting at 3%. Total bandwidth throughput was well below the link capacity.

From a high-level view, the interface reported Up/Up, hiding the underlying physical failure.

[!NOTE] An interface showing FastEthernet0/1 is up, line protocol is up only proves electrical/optical connectivity. It tells you nothing about framing errors or duplex negotiation.

The Discovery: Deep Error Inspection

I logged into our Cisco CPE router and executed a detailed interface diagnostic.

# show interface FastEthernet0/1
FastEthernet0/1 is up, line protocol is up
  Hardware is Fast Ethernet, address is 001a.30b2.88c0
  Full-duplex, 100Mb/s, 100BaseTX/FX
  129384 input errors, 9832 CRC, 0 frame
  0 output errors, 4832 collisions, 2394 late collision

9,832 CRC Errors. 2,394 Late Collisions.

The Root Cause: Duplex Mismatch

A Late Collision is the definitive indicator of a Duplex Mismatch.

Customer Switch Port: Manually hardcoded to 100 Mbps / Full Duplex.
Provider Router Port: Set to Auto-Negotiation (Auto/Auto).

When one side of an Ethernet link is hardcoded to Full Duplex and the other side is set to Auto-Negotiation, the IEEE 802.3u standard requires the Auto side to fallback to Half Duplex.

[!IMPORTANT] The Full Duplex side transmits data whenever it wants. The Half Duplex side listens before transmitting. When the Full side transmits while the Half side is sending, a Late Collision occurs, resulting in corrupted frames and dropped packets.

The Fix

I hardcoded both speed and duplex settings on our interface to match the customer switch:

configure terminal
 interface FastEthernet0/1
  description ## P2P Link to Customer Core ##
  speed 100
  duplex full
 end
write memory

The error counters stopped incrementing immediately. Ping packet loss dropped to 0.0%.

The Verdict

Key Takeaway

Never Trust Auto-Negotiation on Legacy Hardware.

Always inspect physical interface error counters when troubleshooting performance degradation. Don’t rely on status indicators—look directly for CRC Errors and Late Collisions.