We have an SNMPv3 device that is continuously losing contact with Spectrum.  We can reach this device every time with an SNMPwalk even when Spectrum cannot communicate.

How can we troubleshoot this further?

Release : 20.2, 21.2, 22.2

Component : SPCCSS - SpectroSERVER Core

In SNMPv3 all Engine ID´s must be unique, from the Engine ID Spectrum will store the Engine Time which cannot exceed 150 seconds and the Engine Boots which can never decrease in value.

When another device on the network has the same Engine ID the Engine Time and Engine Boots will not match and this causes the problem.

1. Go to $SPECROOT/vnmsh from bash shell and type ./connect to enter CLI

2. Type ./update action=0x10331 mh=<vnm_mh>  to dump the SNMPv3 cache  (output to VNM.OUT in $SPECROOT/SS)

3. Start Wireshark trace (Windows) or TCPDump (Linux) and filtered on the problematic device

4. Type ./update action=0x10333 mh=<vnm_mh>  to clear the SNMPv3 cache

5. If its a discovery problem try to discover the device again, if its a lost contact wait for a few poll cycles to see why the device is not responding.

6. Once the device has regained contact examine the sniffer trace (tcpdump) and compare it to the previous values from the dump of SNMPv3 cache

     The Engine Boots cannot be a lower number and the Engine Time has to be less then 150 seconds.  If either of these are present, then most likely there are devices with duplicate SNMPv3 EngineID´s

If you have any further doubts or questions, please provide sniffer trace (tcpdump) and VNM.OUT with SNMPv3 cache to Spectrum Support for further analysis.  We will also need the SNMPv3 community string in order to decode the tcpdump / wireshark trace.

When looking at the tcpdump / wireshark  if we see a SNMPv3 Report being sent from the device  1.3.6.1.6.3.15.1.1.5.0 (wrong Digest) this would indicate that the credentials are incorrect.  If after verifying the credentials are correct please make sure that the default SNMPV3 settings match what is being sent from the device in the $SPECROOT/SS/.vnmrc file as

snmpv3_default_auth_protocol=md5
snmpv3_default_priv_protocol=des

For example if these settings are set but the device is using SHA and 3DES then you have to prefix the community string between password as follows

#v3/P:SHA^authpassword:3DES^privpassword/userid

Linux command for TCPdump as root user

tcpdump -w troubleshoot.pcap -vv -A -T snmp -s 0 "(dst port 162) or (src port 161) or (dst port 161) and (host <SpectroSERVER IP>)"

Troubleshooting Devices configure SNMPv3 SHA/AES

Unsupported characters for SNMPv3 community strings

SNMPv3 traps not being processed by CA Spectrum's Trap Director (10.3.1 and earlier)

Further SNMPv3 Troubleshooting in Docs