Packet Handler

Packet Structure

I have used a formal packet structure in LoRa transmissions to help provide a level of data integrity. Apart from incorporating specific sender/receiver MAC addresses, the packet header includes a checksum that is calculated over the packet payload. While it is a very simplistic approach, this seems to adequately deal with transmission collisions—a collision invariably results in a badly formed packet, which is subsequently rejected by the gateway either because the MAC addresses have been corrupted and are thus not recognised or because the payload checksum is invalid. To date, there has been enough variation in the timings of the individual processors for me not to have had to worry about any formal 'back-off' process to avoid repeated collisions—my current network comprises 8 Nodes, each transmitting every 60 seconds, and I rarely see a bad packet.

I realise that the Semtech LoRa chips include a checksumming capability in hardware and that this may be far more efficient than the software checksumming process I am currently using but, at the time I was initially developing a packet structure, I didn't really understand how this worked. I am yet to revisit this subject to assess whether or not the present packet checksum adds anything to the overall process so, for the time being, it remains.

I also recognise that much of this, as well as 'reliable' packet delivery, is handled by the WAN part of LoRaWAN and it has always been my intention to look at migrating my system to a LoRaWAN configuration at some point, although I do not ever want my network to become part of any wider network—I am happy for it to be an 'isolated' Intranet of Things. However, since the present arrangement provides a consistently reliable level of connectivity, my current efforts are focused on refining other elements of my set-up.

The basic packet [header] structure that I have used is illustrated below and described in the following file.

Packet Structure

Langlo Packet Structure.docx

[92 KB]

The latest implementation of this packet structure includes additional fields, REL and ACK flags to support the implementation of a basic form of reliable packet delivery (See below).

Data Elements

LoRa messages are ultimately transmitted as a stream of bytes. In the present case, union datatypes have been used to superimpose a formal structure on this byte stream. This greatly simplifies the task of assembling and reading byte streams comprising more complex data structures.

The basic element of the union that defines the packet in the present context is a simple array of bytes (specified as uint8_t). This is the data element that is referenced when sending or receiving a packet. The packet structure, the struct packetContent element, is then superimposed on top of this byte stream to simplify the storage and retrieval of the packet contents.

A second union, genericPayload, superimposes a variable structure on the packet payload, allowing the payload to be populated according to the requirements of individual sensors or applications.

A third union, involving the individual data elements associated with the payloads of individual packet types (e.g. powerPayload, voltagePayload etc.), provides access to the specific data elements of a given packet payload.

The structure of the individual packet data elements was originally described in the Langlo Packets.h file illustrated below and included in the Langlo.zip archive referenced elsewhere.

More recently, I have moved all of these details into a library containing everything to do with the management of packet assembly and transmission (see below). Please note that the organisation of packet structures—the range and identification of individual payload types—has expanded since the original implementation, with some associated changes to data values and/or elements, but the principle and actual structures remain the same.

I am also aware that the IoT community encourages brevity in communications and that I am not adhering to that principle very well at all with the current packet structure—my minimum information unit is generally one byte, or 8 bits, when a single bit is all that is required in some situations. I have also seen JSON-based data structures that are probably more elegant and/or flexible in general IoT applications. Once again I will simply say that this is a pilot system, established to verify the viability of what I am trying to do. Optimisation of the communications protocol, which may involve migrating to LoRaWAN and/or the use of a more economical data format, is something I'll get on to when all the more critical issues are under control.

Packet Buffer

While my initial applications, essentially just monitoring the status of various sensors, had no real need of a reliable packet delivery service, the control of a pump was more critical. The commercial system that I was using had failed on a couple of occasions, with the result that a pump kept running when it shouldn't have and emptied one of my tanks. This was not a major problem at the time, but it could have been if we were in drought.

In implementing a basic reliable packet delivery mechanism then, my primary goal was to be able to reliably identify when a pump was or was not running, regardless of whether or not it was meant to be. More generally, however, I was faced with two primary issues in relation to the reliable delivery of packets.

The first was that I would need to save any packet that required acknowledgement, in case it needed to be resent, until acknowledgement had been received.

The second was the question of how to manage multiple reliable streams in parallel. If multiple streams were to be managed, there was the potential need to retain copies of an arbitrary number of packets until their receipt had been acknowledged. I would then also need some practical means of checking off acknowledged packets so that I could delete the copy that had been retained.

These are not really issues for an end Node, which will most likely only ever be managing a single 'conversation' [with a Gateway Node], but a Gateway Node may be managing 'conversations' with several end Nodes and may thus well need to be concurrently keeping track of several exchanges.

In the event, I have implemented a buffer arrangement using a linked list managed through Ivan Seidel's LinkedList library.

Packets that require reliable delivery are stored in the sender's buffer list until receipt has been acknowledged, or until the packet resend limit has been exceeded. Up to three attempts are currently made, at 0.5 second intervals, to send a packet that requires acknowledgement (both the maximum number of resend attempts and the resend interval are configurable). On receipt of an acknowledgement, or on exceeding the resend limit, the sender locates the relevant packet in the buffer list and deletes it.

If the resend limit has been exceeded with no acknowledgement of packet receipt, the sending Node must take appropriate action—for the Gateway, this will usually be to either notify the MQTT broker, if the original packet was generated in response to an MQTT message, or perform some default action. Similarly, there would need to be some fallback option if the packet originated at a Node that was seeking further instruction in response to some change in local conditions.

My original packet structure has been extended to support a very simple implementation of reliable delivery by using the top bit (REL) of the packet Type field to indicate whether or not the sender requires acknowledgement of receipt, and the top bit (ACK) of the Length to indicate whether or not the packet is an acknowledgement of receipt. There is no need to include the payload of the original packet in an acknowledgement (so the Length field will actually be unused) as the sender will retain a copy of the entire packet until receipt is acknowledged and each packet can be uniquely identified by the sender's MAC address and the packet Sequence Number. In this way, any packet can be subject to reliable delivery at the request of the sending Node.

Packet Structure + ACK

Langlo Packet Structure + ACK.docx

[82 KB]

I am yet to decide whether it might be logical, convenient or otherwise to include functions to support reliable packet delivery, beyond those required to store and retrieve the relevant packet elements, in the library package discussed below. I will need to work through a couple of applications first, to see how much common code there might be that would lend itself to inclusion in a library.

In the mean time, I've started making notes on the reliable delivery protocol on a separate page and my first implementation is also described, with code examples, elsewhere on this site.

Further details pending...

The PacketHandler Library

The PacketHandler library contains everything needed to manage the encoding, transmission, subsequent receipt and decoding of data in my network. The PacketHandler currently uses the packet structure described above, but could be used equally well to manage any packet or data structure or, indeed, no structure at all—just an arbitrary byte stream.

Much of the motivation for building this library was simply to remove any need to deal with packet structure detail when developing application sketches—the application can simply pass data to the PacketHandler and the PacketHandler will take care of packaging that data for delivery to the Gateway. The Gateway then uses the same library to interpret received data.

The library also includes the necessary conditional compiler instructions to enable it to be used with any of the hardware platforms described on this site.

Some PacketHandler methods also draw on the NodeHandler library, which includes methods used in the management of individual Nodes. The NodeHandler is not really necessary when dealing with the software for individual Nodes, but it does simplify the management of Gateway Nodes, which need to know about all of the Nodes with which they communicate. My ultimate goal is for this information to be gathered by a Gateway Node dynamically, when Sensor Nodes first communicate with a Gateway. For the time being however, the Gateway's 'Node Table' is configured statically, through the NodeHandler.

Early versions of the PacketHandler package also included the EepromHandler library. The EepromHandler library really has very little to do with packets or Nodes as such but was included, at the time, for pure convenience (I only had to manage a single library package). In practice, this library is entirely independent of the PacketHandler and NodeHandler libraries and more recently has been packaged independently.

PacketHandler

The PacketHandler class defines the structure and manages the content of data packets.

The normal flow then, within an application sketch, would be to provide the common header information that would identify a Node and its intended Gateway, identify the general nature of the information to be sent to the Gateway, assign values to associated variables, then send the data to the Gateway.

Predefined Data Types

The PacketHandler library includes several predefined, public data types that are used at various points in the process of packet assembly. The most fundamental type is that which effectively defines the individual packet structures.

The order in the above list simply reflects the order in which the different types have been progressively defined. At some point I may come back and organise the list in alphabetical order, as I have done below when describing the different access methods, but while the structure is under development, it helps backward compatibility to just add new types to the end of the list.

Data Packets and Methods

A packet instance is created, and optionally initialised, using the begin method. Several versions of the method are available, with varying degrees of packet header initialisation (refer above for the structure/content of the packet header):

The relevant header fields can, nonetheless, be set at any point within a sketch:

A corresponding set of methods is provided to retrieve header information:

Checksumming is discussed in more detail below.

When sending or receiving packets, the structure of the packet data element is established or determined by setting or reading the packet type with the respective methods:

To simplify the processing of received packets, the structure of which is defined by the Type field in the packet header, the PacketHandler library includes methods that recognise packet types and output the packet contents in a predefined format, to the Serial Monitor, local OLED display or MQTT Broker:

The currently available packet types are listed in the enumerated type PH_packetType (see above), which is, in turn, defined within the PacketHandler library. The specific methods used with each of these packet types are listed below (click to expand a type):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62

#include <SPI.h>// [SX1267] LoRa radio interface
#include <LoRa.h>// LoRa protocol [SX1267]
#include <LoRaWan_APP.h>// LoRa protocol [SX1262]
#include <PacketHandler.h>// Packet Handler
[Other #include directives as required for the application]

const uint32_t gatewayMAC = [gateway MAC address];
const uint32_t myMAC = [Node MAC address];
uint16_t messageCounter = 0;
uint16_t batteryVoltage;

PacketHandler packet;

void setup() {

  [ Radio initialisation ]

// The packet header will also usually be initialised just
// once, during sketch setup as follows

  packet.begin(gatewayMAC,myMAC);// Initialise the Packet Handler
}

// The body of the sketch will then include a cycle,
// assembling and transmitting packets

void loop() {

  [Increment the Sequence Number]
  
  packet.setSequenceNumber(messageCounter);// Set the Sequence #

// The following could be any supported packet type and
// associated variables

  packet.setPacketType(VOLTAGE);// Identify the packet content

  [Read battery voltage]// Or whatever data/sensor(s) is/are to be read

  packet.setVoltage(batteryVoltage);// Set the relevant data variables

// Packet content can be output locally as required

  packet.hexDump();// Output a hex dump of the packet to the Serial Monitor
  packet.serialOut();// Output the packet elements to the Serial Monitor
  packet.displayOut(displayPtr);// Display the packet elements on the Node display (if present)

// Transmit the packet
// For platforms using the Sandeep Mistry LoRa library (SX1276 LoRa node chip)
  
  LoRa.beginPacket();// Initialise transmission
  LoRa.write(packet.byteStream(), packet.packetByteCount());// Load the output buffer
  LoRa.endPacket();// Send the buffer contents

// For platforms using the Heltec LoRaWan_APP library (SX1262 LoRa node chip)

  Radio.Send(packet.byteStream(), packet.packetByteCount());

// Repeat the above for any other packets that need to be sent in a cycle

  [Sleep for a while]
}

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121

#include <SSD1306.h>// OLED display
#include <SPI.h>// LoRa radio interface
#include <LoRa.h>// LoRa protocol
#include <PacketHandler.h>// Packet Handler
[ Other #include directives as required for the application ]

const uint32_t myMAC = [Node MAC address];
[ Other definitions as required ]

SSD1306 display(0x3c, SDA_OLED, SCL_OLED), *displayPtr;

PacketHandler packet;

void setup() {
  
  [ Initialise Serial Monitor ]
  [ Initialise OLED Display ]
  [ Initialise LoRa ]
  
  pinMode(LED_BUILTIN,OUTPUT);

// Initialise the packet objects
  
  packet.begin();

// Set up the pointer to the OLED display
// This is passed to the Packet object when required for output

  displayPtr = &display;
}

// The body of the sketch will then include a cycle,
// receiving and processing packets

void loop() {
 
//  Just loop around checking (promiscuosly) to see if there's anything to process

  if ( receivePacket() ) {
  
    digitalWrite(LED_BUILTIN,HIGH);
         
// Send relevant information to the Serial Monitor
    
    packet.hexDump();
    packet.serialOut();
    
// print RSSI & SNR of packet
    
    Serial.print("[loop] RSSI ");
    Serial.print(LoRa.packetRssi());
    Serial.print("   SNR ");
    Serial.println(LoRa.packetSnr());
    Serial.println();

// Send relevant information to the OLED display
    
    packet.displayOut(displayPtr);   
      
// Print RSSI and SNR of packet
     
    display.drawHorizontalLine(0, 52, 128);
    display.setTextAlignment(TEXT_ALIGN_LEFT);
    display.drawString(0, 54, "RSSI " + (String)LoRa.packetRssi() + "dB");
    display.setTextAlignment(TEXT_ALIGN_RIGHT);
    display.drawString(128, 54, "SNR " + (String)LoRa.packetSnr() +"dB");
    display.display();
  }

  [ Do anything else that needs to be done with a received packet ]
  
  digitalWrite(LED_BUILTIN,LOW);
}

bool receivePacket() {
  
// Only if there's something there

  if ( LoRa.parsePacket() ) {
    while ( LoRa.available() ) {
    
//    Read the packet header
//    We're just reading promiscuously at this point, so we'll actually take anything

      int headerLimit = packet.headerSize();
      
      for (int i = 0; i < headerLimit; i++) {
        packet.setByte(i,LoRa.read());
      }
      
//    If transmissions are directed, we would check that the Destination MAC address was
//    ours before continuing. For the present, we'll take anything.

//    Read in the rest of the packet

      uint8_t byteCount = packet.packetByteCount();
      for (int i = headerLimit; i < byteCount; i++) {
        packet.setByte(i,LoRa.read());
      }
     
//  Verify the checksum before we go any furhter
   
      if (packet.verifyPayloadChecksum()) {

// All good
        
        return true;
      } else {
        Serial.println("[receivePacket] Invalid checksum, packet discarded");
        Serial.println();
        packet.erasePacketHeader();
        return false;
      }
    }
  } else {
    
//  Nothing there

    return false;     
  }
}

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157

#include <SSD1306.h>// OLED display
#include <SPI.h>// LoRa radio interface
#include <LoRa.h>// LoRa protocol
#include <WiFi.h>// WiFi client
#include <PubSubClient.h>// MQTT client
#include <PacketHandler.h>// Packet Handler
#include <Ticker.h>// Asynchronous task management
[ Other #include directives as required ]

const uint32_t myMAC = [Node MAC address];
const int statusCheckInterval = 600;// 600 seconds (10 min)
[ Other definitions as required ]

// Set up supporting clients

Ticker heartbeatTicker, healthCheckTicker, statusTicker;
WiFiClient localClient;
PubSubClient mqttClient(localClient), *mqttClientPtr;
SSD1306 display(0x3c, SDA_OLED, SCL_OLED), *displayPtr;

// Instantiate the Packet Handler
// In addition to generating MQTT messages for received
// packets (inPacket), the Gateway will issue its own
// MQTT status messages (outPacket)

PacketHandler inPacket, outPacket;

void setup() {

  [ Initialise Serial Monitor ]
  [ Initialise OLED Display ]
  [ Initialise LoRa ]
  [ Connect to WiFi ]
  [ Connect to the MQTT Server ]

// Initialise the packet objects

  inPacket.begin();
  outPacket.begin(myMAC,myMAC,0);

// Set up the pointers to the OLED display and MQTT client
// These are passed to the Packet object when required for output

  displayPtr = &display;
  mqttClientPtr = &mqttClient;

// Let the broker know there's been a reset

  outPacket.setPacketType(RESET);
  outPacket.setResetCode(0);
  outPacket.mqttOut(mqttClientPtr);

// And finally, set up the timers

  heartbeatTicker.attach( inPacket.getHeartbeatInterval(), heartbeat );
  healthCheckTicker.attach( inPacket.getHealthCheckInterval(), healthCheck );
  statusTicker.attach( statusCheckInterval, statusCheck );
}

// The body of the Gateway sketch just receives packets
// and sends associated MQTT messages

void loop() {

  if ( receivePacket() ) {
    inPacket.mqttOut(mqttClientPtr);

// We can also output the packet contents, display them
// on the OLED display or flash a LED on receipt

    inPacket.hexDump();// Output a hex dump of the packet to the Serial Monitor
    inPacket.serialOut();// Output the packet elements to the Serial Monitor
    inPacket.displayOut(displayPtr);// Display the packet elements on the Node display

// The tickers will also chime in when their respective timers
// fire and execute the Health Check and Gateway status functions

  }
}

bool receivePacket() {
  if ( LoRa.parsePacket() ) {
    while ( LoRa.available() ) {
      LoRa.beginPacket();
    
// Read the packet header and check the destination MAC

      int headerLimit = inPacket.headerSize();
      uint8_t nextByte;
      
      for (int i = 0; i < headerLimit; i++) {
        nextByte = LoRa.read();
        inPacket.setByte(i,nextByte);
        Serial.printf( "%02X", nextByte);
        if ((i % 4) == 3) {
          Serial.println("    ");
        } else {
          Serial.print("  ");
        }
      }
      
      if (inPacket.destinationMAC() == myMAC) {

// Read in the rest of the packet

        uint8_t byteCount = inPacket.packetByteCount();
        for (int i = headerLimit; i < byteCount; i++) {
          inPacket.setByte(i,LoRa.read());
        }
       
// Verify the checksum
     
        if (inPacket.verifyPayloadChecksum()) {
          return true;
        } else {
          Serial.println("[receivePacket] Invalid checksum, packet discarded");
          Serial.println();
          inPacket.erasePacketHeader();
          return false;
        }
      } else {
        Serial.print("[receivePacket] This appears to be for someone else--To: 0x");
        Serial.print(inPacket.destinationMAC(),HEX);
        Serial.print(" From: 0x");
        Serial.println(inPacket.sourceMAC(),HEX);
        inPacket.erasePacketHeader();
        return false; 
      }
    }
  } else {
    return false;     
  }
}

void heartbeat() {
  inPacket.incrementNodeTimers();
}

void healthCheck() {
  inPacket.nodeHealthCheck(mqttClientPtr);
}

void statusCheck() {
       
// readBatteryVoltage() and readDS18B20Sensor() are examples
// of local functions that return Gateway status information
     
  outPacket.setPacketType(VOLTAGE);
  outPacket.setVoltage(readBatteryVoltage());
  outPacket.serialOut();
  outPacket.mqttOut(mqttClientPtr);

  outPacket.setPacketType(TEMPERATURE);
  outPacket.setTemperature(readDS18B20Sensor());
  outPacket.serialOut();
  outPacket.mqttOut(mqttClientPtr);
}