The obscuring of Palo Alto firewall interface error counters
On some older Palo Alto firewalls such as a PA-3250, the command show counter interface ethernet1/13 will show over 100 "Detailed physical port counters read from MAC". I have firsthand experience that some of these counters will increment for cable issues while nothing shows up in the summary RX or TX error values that are more commonly monitored.
show counter interface ethernet1/13
Interface: ethernet1/13
--------------------------------------------------------------------------------
--------------------------------------------------------------------------------
Physical port counters read from MAC:
--------------------------------------------------------------------------------
rx-broadcast 4095113
<snip>
--------------------------------------------------------------------------------
--------------------------------------------------------------------------------
Detailed physical port counters read from MAC:
--------------------------------------------------------------------------------
snmpBcmCustomReceive0 0
<snip>
snmpBcmReceivedPkts1024to1518Octets 4132098419
snmpBcmReceivedPkts128to255Octets 10411057465
snmpBcmReceivedPkts1519to2047Octets 0
snmpBcmReceivedPkts2048to4095Octets 0
snmpBcmReceivedPkts256to511Octets 5239213697
snmpBcmReceivedPkts4095to9216Octets 0
snmpBcmReceivedPkts512to1023Octets 264711813
snmpBcmReceivedPkts64Octets 1667989221
snmpBcmReceivedPkts65to127Octets 799363660
snmpBcmReceivedPkts9217to16383Octets 0
<snip>
snmpBcmRxFecCorrectable 268435456
snmpBcmRxFecUncorrectable 3050
snmpBcmRxLlfcPrimary 0
snmpBcmRxLlfcSecondary 0
snmpBcmRxPFCControlFrame 0
<snip>
snmpDot3StatsFCSErrors 304106033
snmpDot3StatsFrameTooLongs 0
snmpDot3StatsInRangeLengthError 0
snmpDot3StatsInternalMacReceiveErrors 26440618
snmpDot3StatsInternalMacTransmitErrors 0
snmpDot3StatsLateCollisions 0
snmpDot3StatsMultipleCollisionFrames 0
snmpDot3StatsSQETTestErrors 0
snmpDot3StatsSingleCollisionFrames 0
snmpDot3StatsSymbolErrors 25382788
snmpEtherRxOversizePkts 0
snmpEtherStatsBroadcastPkts 2767864081
snmpEtherStatsCRCAlignErrors 304106033 The same values can generally be seen in "show system state filter-pretty sys.s1.p13.detail". Values from a different firewall, still a PA-3250:
show system state filter-pretty sys.s1.p13.detail
sys.s1.p13.detail: {
snmpBcmCustomReceive0: 0x0,
<snip>
snmpBcmReceivedPkts1024to1518Octets: 0xf64ad173,
snmpBcmReceivedPkts128to255Octets: 0x26c8c2728,
snmpBcmReceivedPkts1519to2047Octets: 0x0,
snmpBcmReceivedPkts2048to4095Octets: 0x0,
snmpBcmReceivedPkts256to511Octets: 0x138480e9b,
snmpBcmReceivedPkts4095to9216Octets: 0x0,
snmpBcmReceivedPkts512to1023Octets: 0xfc72e85,
snmpBcmReceivedPkts64Octets: 0x636b8b91,
snmpBcmReceivedPkts65to127Octets: 0x2fa5524c,
snmpBcmReceivedPkts9217to16383Octets: 0x0,
<snip>
snmpBcmRxFecCorrectable: 0x10000000,
snmpBcmRxFecUncorrectable: 0xbea,
snmpBcmRxLlfcPrimary: 0x0,
snmpBcmRxLlfcSecondary: 0x0,
snmpBcmRxPFCControlFrame: 0x0,
<snip>
snmpDot3StatsFCSErrors: 0x0,
snmpDot3StatsFrameTooLongs: 0x0,
snmpDot3StatsInRangeLengthError: 0x0,
snmpDot3StatsInternalMacReceiveErrors: 0x0,
snmpDot3StatsInternalMacTransmitErrors: 0x0,
snmpDot3StatsLateCollisions: 0x0,
snmpDot3StatsMultipleCollisionFrames: 0x0,
snmpDot3StatsSQETTestErrors: 0x0,
snmpDot3StatsSingleCollisionFrames: 0x0,
snmpDot3StatsSymbolErrors: 0x0,
snmpEtherRxOversizePkts: 0x0,
snmpEtherStatsBroadcastPkts: 0x4185d8,
snmpEtherStatsCRCAlignErrors: 0x0,
<snip>
}This has been reduced in the output of both commands on the newer firewalls such as the PA-1420. Everything that has a value of zero seems to be hidden. There is a KB article that acknowledges this hiding.
So the guidance is to monitor to see if anything has been revealed. It is difficult to do that if you don't know what you are looking for! I will document a few that I have found.
From an actual firewall I manage:
show counter interface ethernet1/19
--------------------------------------------------------------------------------
Detailed physical port counters read from MAC:
--------------------------------------------------------------------------------
rx packets 64 bytes 1798167
rx packets 65 to 127 bytes 43692423
rx packets 128 to 255 bytes 3654577895
rx packets 256 to 511 bytes 365404112
rx packets 512 to 1023 bytes 185088270
rx packets 1024+ bytes 1167646688
packets with bad crc 5
mac receive error 2
bad_octets_rcv 1320
--------------------------------------------------------------------------------
show system state filter-pretty sys.s1.p19.detail
sys.s1.p19.detail: {
bad_crc: 0x5,
bad_octets_rcv: 0x528,
mac_rcv_error: 0x2,
pkts1024tomax_octets: 0x45b0fbe0,
pkts128to255_octets: 0xda152b2a,
pkts256to511_octets: 0x15cdc3b2,
pkts512to1023_octets: 0xb0b043d,
pkts64_octets: 0x1b74e8,
pkts65to127_octets: 0x29b21f5,
}From this LIVEcommunity post:
flow control transmitted 10954
receive fifo overrun 29127
packets with bad crc 1
bad_octets_rcv 1522Another LIVEcommunity post:
sys.s1.p5.detail: { 'bad_crc': 0x157352, 'fragments_pkts': 0x1247, 'mac_rcv_error': 0x228b09, From a KB article:
sys.s1.p2.detail: { 'oversize_pkts': 0x1208cc42,So we could compile a table like this:
| show system state filter-pretty | show counter interface | SNMP |
|---|---|---|
| bad_crc | packets with bad crc | |
| bad_octets_rcv | bad_octets_rcv | |
| mac_rcv_error | mac receive error | |
| fragments_pkts | ||
| flow control transmitted | ||
| rcv_fifo_overrun | receive fifo overrun | |
| oversize_pkts | .1.3.6.1.2.1.2.2.1.13.4 |
I have not yet found a source of all possible values that one might want to watch for.
This article claims that the counter (Hardware interface counter > receive incoming errors) shows the sum of mac_rcv_error and rcv_fifo_overrun.