Discussion:
[Xymon] Devmon tests clear but snmpwalk works
Colin Coe
2016-05-26 12:20:53 UTC
Permalink
Hi all

I know this is the Xymon list not Devmon but I think I'll have most luck on
this list so apologies in advance.

I have a few of Cisco switches that I've just started monitoring at a
couple of remote sites. snmpwalk works from the Xymon server but the Devmon
tests are all clear (Devmon runs on the Xymon server).

When I look at the devmon.log I just see lots of
---
[16-05-***@20:19:09] No SNMP data found for ifName on swi02
[16-05-***@20:19:09] No SNMP data found for ifOutPktsSec on swi02
[16-05-***@20:19:09] No SNMP data found for ifOutCollisions on swi02
---

Any ideas on tracking this down?

We have many if these switches at other sites and they display correctly in
Xymon.

Thanks

CC
Jeremy Laidman
2016-05-26 12:42:50 UTC
Permalink
Colin

Might be the 32bit ID limit. See here:

http://lists.xymon.com/archive/2014-November/040624.html

J

On Thu, 26 May 2016, 22:21 Colin Coe <***@gmail.com> wrote:

> Hi all
>
> I know this is the Xymon list not Devmon but I think I'll have most luck
> on this list so apologies in advance.
>
> I have a few of Cisco switches that I've just started monitoring at a
> couple of remote sites. snmpwalk works from the Xymon server but the Devmon
> tests are all clear (Devmon runs on the Xymon server).
>
> When I look at the devmon.log I just see lots of
> ---
> [16-05-***@20:19:09] No SNMP data found for ifName on swi02
> [16-05-***@20:19:09] No SNMP data found for ifOutPktsSec on swi02
> [16-05-***@20:19:09] No SNMP data found for ifOutCollisions on swi02
> ---
>
> Any ideas on tracking this down?
>
> We have many if these switches at other sites and they display correctly
> in Xymon.
>
> Thanks
>
> CC
> _______________________________________________
> Xymon mailing list
> ***@xymon.com
> http://lists.xymon.com/mailman/listinfo/xymon
>
Nikolai Lifanov
2016-05-26 12:49:12 UTC
Permalink
I have patches to "fix" message size limit.
This updates SNMP_Session and other external modules to
the latest "vanilla" upstream versions and changes dm_snmp
to use the API properly (and bump max_pdu_size to 64k).

On 05/26/2016 08:42, Jeremy Laidman wrote:
> Colin
>
> Might be the 32bit ID limit. See here:
>
> http://lists.xymon.com/archive/2014-November/040624.html
>
> J
>
>
> On Thu, 26 May 2016, 22:21 Colin Coe <***@gmail.com
> <mailto:***@gmail.com>> wrote:
>
> Hi all
>
> I know this is the Xymon list not Devmon but I think I'll have most
> luck on this list so apologies in advance.
>
> I have a few of Cisco switches that I've just started monitoring at
> a couple of remote sites. snmpwalk works from the Xymon server but
> the Devmon tests are all clear (Devmon runs on the Xymon server).
>
> When I look at the devmon.log I just see lots of
> ---
> [16-05-***@20:19:09] No SNMP data found for ifName on swi02
> [16-05-***@20:19:09] No SNMP data found for ifOutPktsSec on swi02
> [16-05-***@20:19:09] No SNMP data found for ifOutCollisions on swi02
> ---
>
> Any ideas on tracking this down?
>
> We have many if these switches at other sites and they display
> correctly in Xymon.
>
> Thanks
>
> CC
> _______________________________________________
> Xymon mailing list
> ***@xymon.com <mailto:***@xymon.com>
> http://lists.xymon.com/mailman/listinfo/xymon
>
>
>
> _______________________________________________
> Xymon mailing list
> ***@xymon.com
> http://lists.xymon.com/mailman/listinfo/xymon
>
Nikolai Lifanov
2016-05-26 12:57:07 UTC
Permalink
On 05/26/2016 08:53, Nikolai Lifanov wrote:
> On 05/26/2016 08:49, Nikolai Lifanov wrote:
>> I have patches to "fix" message size limit.
>> This updates SNMP_Session and other external modules to
>> the latest "vanilla" upstream versions and changes dm_snmp
>> to use the API properly (and bump max_pdu_size to 64k).
>>
>
> See attached. You can tweak max_pdu_size in dm_snmp.pm
>

Sorry, this should read "max_pdu_len".

>> On 05/26/2016 08:42, Jeremy Laidman wrote:
>>> Colin
>>>
>>> Might be the 32bit ID limit. See here:
>>>
>>> http://lists.xymon.com/archive/2014-November/040624.html
>>>
>>> J
>>>
>>>
>>> On Thu, 26 May 2016, 22:21 Colin Coe <***@gmail.com
>>> <mailto:***@gmail.com>> wrote:
>>>
>>> Hi all
>>>
>>> I know this is the Xymon list not Devmon but I think I'll have most
>>> luck on this list so apologies in advance.
>>>
>>> I have a few of Cisco switches that I've just started monitoring at
>>> a couple of remote sites. snmpwalk works from the Xymon server but
>>> the Devmon tests are all clear (Devmon runs on the Xymon server).
>>>
>>> When I look at the devmon.log I just see lots of
>>> ---
>>> [16-05-***@20:19:09] No SNMP data found for ifName on swi02
>>> [16-05-***@20:19:09] No SNMP data found for ifOutPktsSec on swi02
>>> [16-05-***@20:19:09] No SNMP data found for ifOutCollisions on swi02
>>> ---
>>>
>>> Any ideas on tracking this down?
>>>
>>> We have many if these switches at other sites and they display
>>> correctly in Xymon.
>>>
>>> Thanks
>>>
>>> CC
>>> _______________________________________________
>>> Xymon mailing list
>>> ***@xymon.com <mailto:***@xymon.com>
>>> http://lists.xymon.com/mailman/listinfo/xymon
>>>
>>>
>>>
>>> _______________________________________________
>>> Xymon mailing list
>>> ***@xymon.com
>>> http://lists.xymon.com/mailman/listinfo/xymon
>>>
>>
>
Nikolai Lifanov
2016-05-26 13:01:53 UTC
Permalink
On 05/26/2016 08:57, Nikolai Lifanov wrote:
> On 05/26/2016 08:53, Nikolai Lifanov wrote:
>> On 05/26/2016 08:49, Nikolai Lifanov wrote:
>>> I have patches to "fix" message size limit.
>>> This updates SNMP_Session and other external modules to
>>> the latest "vanilla" upstream versions and changes dm_snmp
>>> to use the API properly (and bump max_pdu_size to 64k).
>>>
>>
>> See attached. You can tweak max_pdu_size in dm_snmp.pm
>>
>
> Sorry, this should read "max_pdu_len".
>

List ate my attachments. Here:
https://lifanov.com/files/patches/devmon-use-snmp-session-api.diff
https://lifanov.com/files/patches/devmon-external-module-update.diff

>>> On 05/26/2016 08:42, Jeremy Laidman wrote:
>>>> Colin
>>>>
>>>> Might be the 32bit ID limit. See here:
>>>>
>>>> http://lists.xymon.com/archive/2014-November/040624.html
>>>>
>>>> J
>>>>
>>>>
>>>> On Thu, 26 May 2016, 22:21 Colin Coe <***@gmail.com
>>>> <mailto:***@gmail.com>> wrote:
>>>>
>>>> Hi all
>>>>
>>>> I know this is the Xymon list not Devmon but I think I'll have most
>>>> luck on this list so apologies in advance.
>>>>
>>>> I have a few of Cisco switches that I've just started monitoring at
>>>> a couple of remote sites. snmpwalk works from the Xymon server but
>>>> the Devmon tests are all clear (Devmon runs on the Xymon server).
>>>>
>>>> When I look at the devmon.log I just see lots of
>>>> ---
>>>> [16-05-***@20:19:09] No SNMP data found for ifName on swi02
>>>> [16-05-***@20:19:09] No SNMP data found for ifOutPktsSec on swi02
>>>> [16-05-***@20:19:09] No SNMP data found for ifOutCollisions on swi02
>>>> ---
>>>>
>>>> Any ideas on tracking this down?
>>>>
>>>> We have many if these switches at other sites and they display
>>>> correctly in Xymon.
>>>>
>>>> Thanks
>>>>
>>>> CC
>>>> _______________________________________________
>>>> Xymon mailing list
>>>> ***@xymon.com <mailto:***@xymon.com>
>>>> http://lists.xymon.com/mailman/listinfo/xymon
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Xymon mailing list
>>>> ***@xymon.com
>>>> http://lists.xymon.com/mailman/listinfo/xymon
>>>>
>>>
>>
>
> _______________________________________________
> Xymon mailing list
> ***@xymon.com
> http://lists.xymon.com/mailman/listinfo/xymon
>
Root, Paul T
2016-05-26 13:38:40 UTC
Permalink
Look in your hosts.db file, and see if the switches are identified as the proper template and that the community string is correct. The newer NX type switches do not work with the old template.

Use snmpwalk/snmpget to pull some of the OIDs of the template.

From: Xymon [mailto:xymon-***@xymon.com] On Behalf Of Colin Coe
Sent: Thursday, May 26, 2016 7:21 AM
To: ***@xymon.com
Subject: [Xymon] Devmon tests clear but snmpwalk works

Hi all

I know this is the Xymon list not Devmon but I think I'll have most luck on this list so apologies in advance.

I have a few of Cisco switches that I've just started monitoring at a couple of remote sites. snmpwalk works from the Xymon server but the Devmon tests are all clear (Devmon runs on the Xymon server).

When I look at the devmon.log I just see lots of
---
[16-05-***@20:19:09] No SNMP data found for ifName on swi02
[16-05-***@20:19:09] No SNMP data found for ifOutPktsSec on swi02
[16-05-***@20:19:09] No SNMP data found for ifOutCollisions on swi02
---

Any ideas on tracking this down?

We have many if these switches at other sites and they display correctly in Xymon.

Thanks

CC
This communication is the property of CenturyLink and may contain confidential or privileged information. Unauthorized use of this communication is strictly prohibited and may be unlawful. If you have received this communication in error, please immediately notify the sender by reply e-mail and destroy all copies of the communication and any attachments.
Colin Coe
2016-05-26 22:46:29 UTC
Permalink
Hi all and many thanks for all the replies.

Jeremy: I did the tcpdump as per the thread and found that the R= value was
significantly less than the maximum 32 bit integer on some of the switches
but negative on others. Censored, truncated output follows. How do we
deal with the negative numbers?
---
06:35:54.584298 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto UDP
(17), length 72)
172.22.106.11.60150 > 192.168.78.254.snmp: { SNMPv2c C=public {
GetRequest(28) R=-2032073048 .1.3.6.1.2.1.1.1.0 } }
06:35:55.153728 IP (tos 0x0, ttl 254, id 47570, offset 0, flags [none],
proto UDP (17), length 335)
192.168.78.254.snmp > 172.22.106.11.60150: { SNMPv2c C=public {
GetResponse(287) R=-2032073048 .1.3.6.1.2.1.1.1.0="Cisco IOS Software,
CGS2520 Software (CGS2520-IPSERVICESK9-M), Version censored, RELEASE
SOFTWARE (fc1)^M^JTechnical Support:
http://www.cisco.com/techsupport^M^JCopyright
(c) 1986-2011 by Cisco Systems, Inc.^M^JCompiled sometime by prod_rel_team"
} }
06:35:55.157010 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto UDP
(17), length 78)
172.22.106.11.60150 > 192.168.78.254.snmp: { SNMPv2c C=public {
GetRequest(34) R=-2032073047 .1.3.6.1.4.1.9.9.109.1.1.1.1.5.1 } }
06:35:55.731062 IP (tos 0x0, ttl 254, id 47571, offset 0, flags [none],
proto UDP (17), length 79)
192.168.78.254.snmp > 172.22.106.11.60150: { SNMPv2c C=public {
GetResponse(35) R=-2032073047 .1.3.6.1.4.1.9.9.109.1.1.1.1.5.1=5 } }
06:35:55.732084 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto UDP
(17), length 74)
172.22.106.11.60150 > 192.168.78.254.snmp: { SNMPv2c C=public {
GetRequest(30) R=-2032073046 .1.3.6.1.4.1.9.2.1.2.0 } }
06:35:56.316063 IP (tos 0x0, ttl 254, id 47572, offset 0, flags [none],
proto UDP (17), length 82)
192.168.78.254.snmp > 172.22.106.11.60150: { SNMPv2c C=public {
GetResponse(38) R=-2032073046 .1.3.6.1.4.1.9.2.1.2.0="power-on" } }
---

Nikolai: I'm a bit hesitant to apply patches like this so I'll exhaust all
other avenues before looking at this.

Paul: Double checked Devmon's hosts.db and I can confirm the details are
correct.

Thanks


On Thu, May 26, 2016 at 9:38 PM, Root, Paul T <***@centurylink.com>
wrote:

> Look in your hosts.db file, and see if the switches are identified as the
> proper template and that the community string is correct. The newer NX type
> switches do not work with the old template.
>
>
>
> Use snmpwalk/snmpget to pull some of the OIDs of the template.
>
>
>
> *From:* Xymon [mailto:xymon-***@xymon.com] *On Behalf Of *Colin Coe
> *Sent:* Thursday, May 26, 2016 7:21 AM
> *To:* ***@xymon.com
> *Subject:* [Xymon] Devmon tests clear but snmpwalk works
>
>
>
> Hi all
>
>
>
> I know this is the Xymon list not Devmon but I think I'll have most luck
> on this list so apologies in advance.
>
>
>
> I have a few of Cisco switches that I've just started monitoring at a
> couple of remote sites. snmpwalk works from the Xymon server but the Devmon
> tests are all clear (Devmon runs on the Xymon server).
>
>
>
> When I look at the devmon.log I just see lots of
>
> ---
>
> [16-05-***@20:19:09] No SNMP data found for ifName on swi02
>
> [16-05-***@20:19:09] No SNMP data found for ifOutPktsSec on swi02
>
> [16-05-***@20:19:09] No SNMP data found for ifOutCollisions on swi02
>
> ---
>
>
>
> Any ideas on tracking this down?
>
>
>
> We have many if these switches at other sites and they display correctly
> in Xymon.
>
>
>
> Thanks
>
>
>
> CC
> This communication is the property of CenturyLink and may contain
> confidential or privileged information. Unauthorized use of this
> communication is strictly prohibited and may be unlawful. If you have
> received this communication in error, please immediately notify the sender
> by reply e-mail and destroy all copies of the communication and any
> attachments.
>
Jeremy Laidman
2016-05-26 23:49:25 UTC
Permalink
On Fri, 27 May 2016, 08:46 Colin Coe <***@gmail.com> wrote:

> Hi all and many thanks for all the replies.
>
> Jeremy: I did the tcpdump as per the thread and found that the R= value
> was significantly less than the maximum 32 bit integer on some of the
> switches but negative on others. Censored, truncated output follows. How
> do we deal with the negative numbers?
>


Can you confirm if positive IDs are accepted by devmon? If not, this might
be a red herring.

OK, so I'm no SNMP expert, but this is what I think is happening. When a
large 64bit integer (greater than 2^32-1) that is cast to a 32bit integer,
the highest bit is interpreted as a sign bit of 1, meaning negative. The
request ID can be a 64bit integer, but not greater than 2^32-1. But some
switches convert into an incompatible type that causes problems with 32bit
libraries. Some builds of SNMP libraries have an API option to avoid
sending requests with negative IDs.

In my case, I think I had a 32bit MRTG package installed on a 64bit OS, and
I had to uninstall it and reinstall the 64bit version of package. I'm
guessing that the 32bit version sent out negative IDs in queries but
couldn't cope with the 64bit values in the responses. It's also possible
that I remember things wrongly (it was a loooong time ago), so don't trust
my conjecture about the cause but of this.

I recall some people recompiled their Perl module to enable the "avoid
using negative IDs" option in the API.

More here:

https://lists.oetiker.ch/pipermail/mrtg-developers/2002-September/000103.html

This link mentions Dell switches but my problem was with Cisco switches.

J

---
> 06:35:54.584298 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto UDP
> (17), length 72)
> 172.22.106.11.60150 > 192.168.78.254.snmp: { SNMPv2c C=public {
> GetRequest(28) R=-2032073048 .1.3.6.1.2.1.1.1.0 } }
> 06:35:55.153728 IP (tos 0x0, ttl 254, id 47570, offset 0, flags [none],
> proto UDP (17), length 335)
> 192.168.78.254.snmp > 172.22.106.11.60150: { SNMPv2c C=public {
> GetResponse(287) R=-2032073048 .1.3.6.1.2.1.1.1.0="Cisco IOS Software,
> CGS2520 Software (CGS2520-IPSERVICESK9-M), Version censored, RELEASE
> SOFTWARE (fc1)^M^JTechnical Support: http://www.cisco.com/techsupport^M^JCopyright
> (c) 1986-2011 by Cisco Systems, Inc.^M^JCompiled sometime by prod_rel_team"
> } }
> 06:35:55.157010 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto UDP
> (17), length 78)
> 172.22.106.11.60150 > 192.168.78.254.snmp: { SNMPv2c C=public {
> GetRequest(34) R=-2032073047 .1.3.6.1.4.1.9.9.109.1.1.1.1.5.1 } }
> 06:35:55.731062 IP (tos 0x0, ttl 254, id 47571, offset 0, flags [none],
> proto UDP (17), length 79)
> 192.168.78.254.snmp > 172.22.106.11.60150: { SNMPv2c C=public {
> GetResponse(35) R=-2032073047 .1.3.6.1.4.1.9.9.109.1.1.1.1.5.1=5 } }
> 06:35:55.732084 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto UDP
> (17), length 74)
> 172.22.106.11.60150 > 192.168.78.254.snmp: { SNMPv2c C=public {
> GetRequest(30) R=-2032073046 .1.3.6.1.4.1.9.2.1.2.0 } }
> 06:35:56.316063 IP (tos 0x0, ttl 254, id 47572, offset 0, flags [none],
> proto UDP (17), length 82)
> 192.168.78.254.snmp > 172.22.106.11.60150: { SNMPv2c C=public {
> GetResponse(38) R=-2032073046 .1.3.6.1.4.1.9.2.1.2.0="power-on" } }
> ---
>
> Nikolai: I'm a bit hesitant to apply patches like this so I'll exhaust all
> other avenues before looking at this.
>
> Paul: Double checked Devmon's hosts.db and I can confirm the details are
> correct.
>
> Thanks
>
>
> On Thu, May 26, 2016 at 9:38 PM, Root, Paul T <***@centurylink.com>
> wrote:
>
>> Look in your hosts.db file, and see if the switches are identified as the
>> proper template and that the community string is correct. The newer NX type
>> switches do not work with the old template.
>>
>>
>>
>> Use snmpwalk/snmpget to pull some of the OIDs of the template.
>>
>>
>>
>> *From:* Xymon [mailto:xymon-***@xymon.com] *On Behalf Of *Colin Coe
>> *Sent:* Thursday, May 26, 2016 7:21 AM
>> *To:* ***@xymon.com
>> *Subject:* [Xymon] Devmon tests clear but snmpwalk works
>>
>>
>>
>> Hi all
>>
>>
>>
>> I know this is the Xymon list not Devmon but I think I'll have most luck
>> on this list so apologies in advance.
>>
>>
>>
>> I have a few of Cisco switches that I've just started monitoring at a
>> couple of remote sites. snmpwalk works from the Xymon server but the Devmon
>> tests are all clear (Devmon runs on the Xymon server).
>>
>>
>>
>> When I look at the devmon.log I just see lots of
>>
>> ---
>>
>> [16-05-***@20:19:09] No SNMP data found for ifName on swi02
>>
>> [16-05-***@20:19:09] No SNMP data found for ifOutPktsSec on swi02
>>
>> [16-05-***@20:19:09] No SNMP data found for ifOutCollisions on swi02
>>
>> ---
>>
>>
>>
>> Any ideas on tracking this down?
>>
>>
>>
>> We have many if these switches at other sites and they display correctly
>> in Xymon.
>>
>>
>>
>> Thanks
>>
>>
>>
>> CC
>> This communication is the property of CenturyLink and may contain
>> confidential or privileged information. Unauthorized use of this
>> communication is strictly prohibited and may be unlawful. If you have
>> received this communication in error, please immediately notify the sender
>> by reply e-mail and destroy all copies of the communication and any
>> attachments.
>>
>
> _______________________________________________
> Xymon mailing list
> ***@xymon.com
> http://lists.xymon.com/mailman/listinfo/xymon
>
Phil Crooker
2016-05-30 03:22:01 UTC
Permalink
How do the IOS versions compare between the switches that work and those that don't?

?

________________________________
From: Xymon <xymon-***@xymon.com> on behalf of Colin Coe <***@gmail.com>
Sent: Monday, 30 May 2016 12:26 PM
To: Jeremy Laidman
Cc: ***@xymon.com
Subject: Re: [Xymon] Devmon tests clear but snmpwalk works

Hi Jeremy

Yep, did this after reading through your other thread again.

Still no change in behavior. Tis very strange as we have many of this model switch responding correctly and the cisco router (and another type of Cisco switch) installed on those sites are working as expected.

Still looking...

Thanks

On Mon, May 30, 2016 at 10:28 AM, Jeremy Laidman <***@rebel-it.com.au<mailto:***@rebel-it.com.au>> wrote:
On Fri, May 27, 2016 at 9:49 AM Jeremy Laidman <***@rebel-it.com.au<mailto:***@rebel-it.com.au>> wrote:
Can you confirm if positive IDs are accepted by devmon? If not, this might be a red herring.

Try applying this adjustment to devmon and see if it helps. This forces the SNMP_Session module to avoid negative request IDs:

*** devmon.orig 2011-05-06 15:32:03.000000000 +1000
--- devmon 2016-05-30 12:26:14.000000000 +1000
***************
*** 30,35 ****
--- 30,37 ----
# Do start-up housekeeping
initialize($VERSION);

+ $SNMP_Session::default_avoid_negative_request_ids = 1;
+
# Enter main loop
do_log("Entering poll loop", 0);
while (1) {
Jeremy Laidman
2016-05-30 05:21:20 UTC
Permalink
On Mon, May 30, 2016 at 12:56 PM Colin Coe <***@gmail.com> wrote:

> Hi Jeremy
>
> Yep, did this after reading through your other thread again.
>
> Still no change in behavior. Tis very strange as we have many of this
> model switch responding correctly and the cisco router (and another type of
> Cisco switch) installed on those sites are working as expected.
>

Ah, was worth a shot.

Another possibility I seem to recall people having problems with is the use
of bulk responses. In SNMPv2c, you can send a query for a tree walk and
get one single response (depending on size) with lots of OIDs in it, and
some devices might have problems with this. If you switch the devmon
config to v1, you disable bulk responses, and it might be a bit more
reliable for some devices, although it'll generate a bit more outbound
traffic in some cases.

J
Loading...