Progress report

Progress has been slow and painful mostly because of my own stupidity. I have been going nutz with authentication error while parsing INFORM packets from SNMPv3 agents and couldn’t find the fault only to find out that I was passing receive buffer length to the decode function and not received length resulting in a longer buffer padded with zeros and messing up with the authentication hash generation. Such is life…

That is fixed now and SNMPv3 INFORM and TRAP handling is moving into alpha (from pre-release) with a couple of hours of testing planned for next week against actual agents in the real world. I expect to have this nailed down soon and ready for release.

One thing I have discovered from e-mails I’ve been receiving is that high level of control while coding SNMP is not always desirable. To make it easier for people who just want to get to the data, I have created a SimpleSnmp class that will allow SNMPv1 and SNMPv2 request generation and reply processing with no or very little knowledge of how SNMP works.

I have replaced a lot of code based on other libraries implementations from the encode/decode methods in base type classes. This has taken some doing and a lot of debugging. To make sure this all works, I have written a testing server using a proven and stable C++ SNMP library and then written a whole lot of c# apps to send encoded data to it for verification. Just so you know, this is a bad idea and just the testing of Integer32 class has taken a better part of 3 days on a quad-core system I use for development.

Besides the “big” changes, I have been working on introducing higher level of consistency to class methods and properties. During development you tend to focus so much on delivering functionality that you forget that classes performing similar functions should have similar methods and properties. That work continues and I don’t think it will be fully done in the next release but at least I don’t plan to remove anything but just keep adding what’s missing.

Exception generation is another area that I have tried to put a little time into making it more sensible. In the earlier releases I have been creating exception classes as needed and ended up with a lot of them being used only once or twice. Again, work is far from over but reduction in the number of exception classes thrown and better notification of what went wrong is the goal. Introduction of error codes in each exception class so that caller can determine if it is fatal or recoverable (as most SNMP exceptions on a network manager are) should help.

At this point, I expect I’ll take TripleDES, AES-192 and AES-256 privacy classes completely out of the next release and target to re-introduce them later down the road when I have the time to finish implementing them. I work on the library away from agents that support these protocols so I am finding it very hard to find the time to spend in a lab testing code with all the other obligations I have. If you have a project that absolutely depends on these privacy protocols being implemented, let me know and I’ll see what I can do to speed things out but for now this functionality is out.

Check back often and look out for library release 0.5 no later then a month from now. This release will be the first “real” beta with all functionality included and tested as much as possible in the lab environment. This is where I’ll start doing the real world testing with the applications that are in the pipeline.

SNMP Version 3 TRAPs and INFORMs

I finally managed to figure out proper handling of TRAP and INFORM packets with SNMP version 3.

Turns out the SnmpV3Packet.decode function will not be sufficient because it attempts to authorize and un-encrypt the packet. With notification packet types we need to look-ahead into the packet to determine the values required to perform these operations. Values needed are part of the USM header, EngineId, SecurityName (or user name), EngineBoots and EngineTime.

To be able to do a lookup of the required values prior to full packet decode, authentication and privacy processing, I have added SnmpV3Packet.GetUSM method that performs a partial packet decode and returns UserSecurityModel class containing values in the header that we need to proceed. As part of the GetUSM parsing, required values are set in the SnmpV3Packet class so the only additional step needed to finalize the packet decoding is lookup of the security parameters for the EngineId/SecurityName combination (that should be stored by the application) and setting the appropriate AuthenticationProtocol, PrivacyProtocol and secrets for both so that packet can be decoded.

In other words, once you receive a packet you will call SnmpV3Packet.GetUSM and retrieve UserSecurityModel class. Using EngineId and SecurityName and match them with the authentication and privacy settings you have stored for each SNMP agent, set the appropriate Authentication, AuthenticationSecret, Privacy and PrivacySecret values and then call SnmpV3Packet.decode to process the packet.

Back to the beginning of SNMP encoding

I am going to back up a little to the very beginning of how to encode SNMP packets.

All SNMP values are encoded using TLV format. T standards for type, a one byte value that represents the type of value that is encoded, L is length of the encoded value excluding the T and L part, and V is the value itself.

The smart thing about this type of data representation is that you can nest types. For example, a sequence is a collection of types. T value in a sequence tells you that value part of it contains other TLV encoded values. Parsing of a sequence is bound by the L which represents the aggregate length of all “child” TLV encoded values.

In SNMP, entire packet starts with a SMI_SEQUENCE encoded TLV representing all components of the packet. Initial sequence contains values (such as protocol version) and other sequences (like the Pdu).

BER Encoding and Decoding Integer32 values

This is the encoding I had most trouble with. 2’s complement just made no sense for a long time. Probably because I didn’t really read the documentation I managed to find but just skimmed through it hoping to understand it quickly and easily. This didn’t happen.

Based on the info I found, you have to use 2 different procedures to encode positive and negative numbers. For a positive number, encoding is relatively simple.

List<byte> encoded = new List<byte>
// Value to encode
int valueToEncode = 1234567;
byte[] b = BitConverter.GetBytes(valueToEncode);
for(int i=3; i>=0; i++) {
  if( b[i] == 0 &amp;&amp; encoded.Count == 0 ) {
    // skip 0 values from the beginning of the value
    continue;
  } else {
    encoded.Add(b[i]);
  }
}
// Encoding rule states we can't have 9 x 1s at the beginning of the encoded value
if( encoded[0] == 0xff &amp;&amp; (encoded[1] &amp; 0x80) != 0 ) {
  // we have 9 1's at the beginning of the encoded value. Insert a 0
  encoded.Insert(0,0x00);
}
// Most significant bit of the most significant byte can't bet 1 or this is a negative number
if( (encoded[0] &amp; 0x80) != 0 ) {
  encoded.Insert(0,0x00);
}
return encoded.ToArray(); // This is the encoded value as byte array

If you are asking, what just happened here, here is what… Positive value is encoded using exactly the same format as data is internally stored on your computer. One difference on little endian systems (like your Windows PC) is that bytes need to be reversed. In the interest of space, only significant bytes are encoded. That means that we can discard all the most significant bytes that are value 0 (zero) since they don’t define the value we are trying to transmit. The rest of the bytes are copied to the encoded data buffer.

Before being ready for transmission, we have to make sure that we don’t have 9 consecutive bits at the beginning of the encoded value set to 1 (one). If we do, we just add a single byte with the value of 0x00 to the beginning of the encoded value and presto…all done.

Negative value is where I got into trouble. I just assumed that setting the first bit of the most significant byte to 1 will act as a signed flag (0 for positive, 1 for negative) and I can move on. It’s not that easy.

Here is how you do it:

int intValue = -123456;
// First make sure we have a negative number
if( intValue < 0 ) {
  // Convert number to its positive representation
  int posVal = -(intValue);
  // Convert positive number to 2s complement format
  // (this means convert all 1s to 0s and 0s to 1s)
  int twosComplementVal = ~posVal;
  // Add 1 to finalize 2s complement conversion
  twosComplementVal += 1;
  // Get the value bytes
  byte[] b = BitConverter.GetBytes(twosComplementVal);
  List<byte> encoded = new List<byte> // somewhere to store the encoded value
  for (int i = 3; i >= 0; i--)
  {
    if (encoded.Count > 0 || b[i] != 0xff)
    {
      // only add values that are significant
      encoded.Add(b[i]);
    }
  }
  // if we don't have room to set the signed bit, add a byte for it
  if ((encoded[0] &amp; 0x80) == 0)
    encoded.Insert(0,0xff);
}

I am going to leave decoding for another time. You will be able to see how this code works in soon to be released version 0.5 of the SnmpSharpNet library.

Additional note: April/3, 2009

This was a giant waste of time. After a few tests it was apparent that internally c# stores Int32 values in 2s complement format. All that you need to do is call BitConverter.GetBytes(Int32Value) and presto, everything is encoded in the right format. All you have to do is trim the unnecessary bytes. New encoding and decoding process using internal c# 2s complement integer representation has been (or will be if you are reading this before release date) included in library version 0.5.2

BER Encoding and Decoding OID values

I have decided that next version will be 0.5 and a formal beta release. One of the goals I have set for myself is that 0.5 beta will contain only my own code. To accomplish this, I have to re-code the BER encoding/decode functionality of Oid, Integer32, UInteger32 and Counter64 classes.

I started the code rewrite with the Oid class.

Before being able to do any work I had to do a little research and figure out how it’s supposed to be done. This is what I have found out.

First, what is OID? OID is an array of integers representing an OID tree branch. OIDs are represented in dotted decimal notation. For example, .1.3.6.1.2.1.1.1.0.

Before talking about encoding, there is a special way of encoding the first 2 integers in the OID. First two values are encoded in a single value generated using * 40 +

In the interest of space, individual OID instances are not encoded with the standard TLV (type, length, value) notation but as individual values using 7 bits per byte to encode the value and using the 8th bit as “more” label. Individual bytes are encoded in most significant byte first (opposite of the way numbers are encoded on Intel PCs).

How this works is that if you need to encode a 3 byte value to represent a sub-oid, first 2 bytes will have bit 8 (bit 0x80) set to 1 and 3rd byte bit 8 will be set to 0 meaning “no more data to follow”. This is what is known as long form encoding.

When you only need a single byte to encode a value, remembering that first bit always means “more data follows” thus highest value that can be encoded is 127, then only one byte is added with the highest or most significant bit being set to 0.

What does this mean? Well, to encode a single number, you’ll need to determine how many bytes you’ll need to encode it and then encode it with 7 bits per byte.

Here is an example in C#:

int value = 1024; // value to encode
List<byte> buffer = new List<byte>();
while( value != 0 )
{
  byte[] b = BitConverter.GetBytes(value);
  // set high bit on each byte we are processing
  if( (b[0] & 0x80) == 0 ) b[0] |= 0x80;
  buffer.Insert(0, b[0]);
  value >>= 7; // shift value by 7 bits to the right
}
// Almost done. Clear high bit in the last byte
buffer[buffer.Count-1] = (byte)(buffer[buffer.Count-1] & ~0x80);
byte[] result = buffer.ToArray();

Above procedure should be repeated for each value in the OID. Remember, you should not use the above procedure for encoding the first two integers in the OID.

Now to decoding. To decode a received OID, first  you have to talk care of the first two sub-OIDs. To reverse the encoding of the first two sub-OID values into a single value, do the following:

// Take first byte of the encoded OID
byte byteValue = inpacket[0];
// Convert it to Int32 value
Int32 intValue = (Int32)byteValue;
// Get first sub-OID
int oid1 = intValue / 40;
// Get second sub-OID
int oid2 = intValue % 40;

Now proceed to decode individual sub-OIDs following the initial value:

// This is the length of the encoded OID from the TLV header
int headerLength = 40;
// Encoding position starts at 1 - first byte was processed
// as first 2 encoded sub-OIDs
int position = 1;
List<int> oid = new List<int>;
while( headerLength > 0 ) {
  // First check if sub-OID is encoded in the short form
  if( (inpacket[position] & 0x80) == 0 ) {
    // First bit = 0, short form encoding
    oid.Add((int)inpacket[position];
    position += 1;
    headerLength -= 1;
  } else {
    // First bit = 1, long form encoding
    int result = 0; // This is where we store the sub-oid value
    bool done = false; // Individual value completion flag
    do {
      result <<= 7; // shift the value
      byte[] b = BitConverter.GetBytes(result);
      // Process first byte only
      if( (b[0] & 0x80) == 0 )
        done = true;
      else
        b[0] = (byte)(b[0] & ~0x80); // clear the &quot;more&quot; bit
      result = result | b[0]; // add the value to the result
      position += 1;
      headerLength -= 1;
    while( headerLength > 0 && ! done );
    oid.Add(result);
  }
}

Introduction

I have registered a new SNMP project on SourceForge in December and since then there have been a lot of questions about the project activity.

Since I’m the only project developer, project is moving at the pace that depends on my ability to set aside time to work on it. Sometimes, this can lead to a slow release cycle and questions if project is still active.

This blog is an attempt to provide some background on current work on the project and direction it is talking. I am also hoping that it will provide a platform for comments and requests that I can use in setting the direction for future development.

Current development priorities are:

  • Provide utility class for simplified access to “base” SNMP commands in protocol versions 1 and 2c. These will include Get, Get-Next, Get-Bulk and Set.
  • Complete testing of V2TRAP and INFORM packets in protocol ver. 2c
  • Update the component to provide classes and methods to support SNMP version 3 handling of INFORM and TRAP notifications
  • Cleanup exception throwing and document it

I expect most of the priority tasks to be completed for the next beta release 0.4.4 that I am hoping to have ready in the first or second weeks of March.

Other then coding work that needs to be done, documentation is not keeping up with the development. Following needs to be done urgently:

  • Restructure in-code comments to provide more information on how to use individual classes
  • Build class by class documentation demonstrating required behavior as it relates to SNMP and additional functionality built in for post retrieval processing convenience
  • More explained examples. This will be very important for SNMP v3 TRAP and INFORM handling because it requires “agent like” functionality in the Manager application.

Since project has just come out of Alpha stage (and still has Alpha and pre-release level code in it), documentation was not my priority but that has to change and will change.

Let me know if you like to component or what you would like to see changed. Stay tuned for updates…