An ESP32 can send temperature readings to a server, but things become more complicated when several applications need the same information. For example, an environmental monitoring device might need to send readings to a dashboard, a data storage system, and an alert service. The device should not need separate communication logic for every application.

MQTT solves this problem through a publish/subscribe communication model. Instead of sending data directly to each application, the ESP32 publishes messages to a central broker. The broker then distributes those messages to applications that have subscribed to the relevant topics.

MQTT is an application-layer messaging protocol that normally operates over a reliable transport such as TCP. Standard MQTT connections commonly use port 1883, while connections protected by TLS commonly use port 8883. Although MQTT messages are relatively small, building a dependable system still requires careful decisions about message delivery, reconnection, topic organization, authentication, and data handling.

This article explains how MQTT works, how its main features can be used in ESP32 projects, and what should be considered when developing a reliable communication system with ESP-IDF.

Understanding the MQTT Publish/Subscribe Model

A typical MQTT system has three main components.

The publisher does not need to know the identity or address of the subscribers. This separation makes it easier to add new applications without changing the firmware of the device that produces the data.

Consider an industrial monitoring system with an ESP32 identified as esp32-017. The device could publish its readings to plant/line2/esp32-017/telemetry.

An operations dashboard could subscribe to plant/line2/+/telemetry to receive readings from devices on line 2. A historical data collection service could subscribe to plant/line2/# to receive messages from the entire line. A separate control application could publish commands to plant/line2/esp32-017/cmd/setpoint.

MQTT supports two wildcard characters in subscription filters:

Wildcards are allowed in subscription filters, not in published topic names. Topic names are case-sensitive, so uppercase and lowercase letters must match exactly.

Wildcards also do not match system topics beginning with $, such as $SYS, when the subscription filter begins with a wildcard.

The slash-separated topic structure is an organizational convention. The broker routes messages according to topic names and does not interpret the information contained in the message payload.

Establishing an MQTT Connection

An MQTT connection begins when the client establishes a transport connection and sends a CONNECT packet. The broker responds with CONNACK, which indicates whether the connection has been accepted.

Opening a TCP connection alone does not confirm that the MQTT session has been established successfully.

Clients use SUBSCRIBE and SUBACK packets to establish subscriptions and confirm them. PUBLISH packets carry application messages between clients and the broker. PINGREQ and PINGRESP packets help maintain an otherwise idle connection.

The Keep Alive setting determines the expected interval for communication between the client and broker. If the broker receives no control packet from the client within approximately 1.5 times the negotiated Keep Alive interval, it can treat the connection as lost.

Understanding these connection stages is important when diagnosing communication failures in an ESP32-based application.

Choosing MQTT Instead of HTTP

MQTT and HTTP are both useful communication protocols, but they suit different application requirements.

HTTP is commonly used for configuration endpoints, firmware downloads, and requests in which a client needs a response from a server. MQTT is particularly useful for frequent sensor updates, asynchronous events, and situations where multiple applications need the same information.

With MQTT, an ESP32 can maintain a connection to the broker and publish messages as new readings become available. The broker distributes those messages to the appropriate subscribers without requiring the sensor device to send an individual request to every application.

HTTP can also reuse TCP connections, so MQTT does not automatically guarantee lower overhead or power consumption in every situation. Actual performance depends on the size of the messages, communication frequency, network conditions, radio behavior, and other implementation choices.

MQTT also introduces additional responsibilities. The system needs a broker, and the device must manage connection state, retries, and communication failures.

The choice between MQTT and HTTP should therefore depend on the application's communication pattern, resource requirements, and reliability needs.

Selecting the Appropriate MQTT Quality of Service

MQTT provides three Quality of Service levels that control message delivery between communicating clients.

QoS 0: At Most Once

At QoS 0, the sender publishes the message without waiting for an MQTT delivery acknowledgment. The message is delivered at most once by that protocol exchange, but it can be lost.

This level is suitable for frequent sensor readings when an occasional missing measurement is acceptable and a newer reading will soon replace it.

QoS 1: At Least Once

QoS 1 uses a PUBACK acknowledgment. The sender can retransmit a message if the acknowledgment is not received, which means the receiver may receive the same message more than once.

Applications using QoS 1 should be able to recognize and safely handle duplicate messages when necessary.

For example, a temperature dashboard can generally tolerate a repeated reading. A command that activates a motor or changes a machine setting requires more careful handling.

QoS 2: Exactly Once

QoS 2 provides exactly-once delivery at the MQTT protocol level through a four-step exchange involving PUBLISH, PUBREC, PUBREL, and PUBCOMP.

This process requires additional communication and state compared with QoS 0 and QoS 1.

QoS applies separately to the publisher-to-broker and broker-to-subscriber stages. The broker delivers a message at a QoS no higher than both the publication's QoS and the subscription's permitted QoS.

Even QoS 2 does not guarantee that a physical operation will happen exactly once at the application level. A device may restart or process a command again after a failure.

For important operations, applications should use unique command identifiers, application-level acknowledgments, duplicate detection, and checks of the resulting device state.

Retained Messages, Persistent Sessions, and Last Will

MQTT provides several features that help devices and applications handle changing connection states. Retained messages, persistent sessions, and Last Will and Testament serve different purposes.

Retained Messages

When a message is published with the retain flag enabled, the broker stores it as the last retained value for that topic. A new subscriber can receive this value immediately after subscribing instead of waiting for the next publication.

This feature is useful for information such as device configuration, operating mode, and online or offline status.

However, a retained message may no longer represent the current state of a device. Including a timestamp can help subscribers determine how recent the information is.

To remove a retained value, a client can publish a zero-length retained message to the same topic.

Persistent Sessions

Persistent sessions help preserve subscriptions and certain in-flight QoS 1 and QoS 2 messages when a client disconnects.

In MQTT 3.1.1, the Clean Session flag determines whether the broker maintains the client's session state across connections. MQTT 5 separates this behavior into Clean Start and Session Expiry Interval.

Retained messages are different from session state. They belong to topics and are not tied to an individual client's session.

ESP-MQTT normally uses MQTT 3.1.1 unless MQTT 5 support is enabled and selected in the configuration.

When developing a device, it is important to understand whether the broker is expected to preserve subscriptions after disconnection. The application should not assume that subscriptions remain active if the configured session behavior does not support that expectation.

Last Will and Testament

The Last Will and Testament, commonly called LWT, allows a client to register a message that the broker can publish if the connection ends unexpectedly.

For example, an ESP32 can register an offline status message during connection setup. After connecting successfully, it can publish an online status message to the same status topic.

If the broker detects an unexpected disconnection, it publishes the registered offline message according to the configured LWT settings.

A retained offline message can help newly connected subscribers learn that the device was last reported offline.

However, the broker needs time to detect the failure. Detection often depends on the Keep Alive timeout, so the offline message may arrive well after the device actually stops operating.

LWT is useful for status reporting, but it should not replace a safety-critical heartbeat or a dedicated mechanism for detecting dangerous equipment failures.

Implementing MQTT on ESP32 with ESP-IDF

ESP-IDF provides the development framework and networking components needed to build MQTT applications on ESP32 devices. ESP-MQTT provides the client functionality for connecting to a broker, publishing messages, subscribing to topics, and responding to connection events.

Before setting up the client, the ESP32 should already have a working Wi-Fi connection, a valid IP address, and access to the configured broker.

The configuration structure depends on the ESP-IDF version. ESP-IDF 4.x used a different, flat configuration structure. ESP-IDF 5.x uses the newer configuration structure and includes ESP-MQTT as a component. Starting with ESP-IDF 6.0, ESP-MQTT is distributed separately and must be added as a project dependency through the component manager.

For ESP-IDF 5.x projects, the MQTT component dependency must be declared in the project's component configuration where required.

Configuring the MQTT Client

The client configuration includes the broker address, a unique client identifier, and any required authentication or session settings.

A development setup might use a broker on a private network with an address such as mqtt://192.168.1.10:1883. This is suitable only for an appropriately isolated test environment.

Real credentials should never be transmitted through unencrypted MQTT over an untrusted network. Production systems should use MQTT over TLS, verify the broker's certificate, and authenticate devices using individual credentials or client certificates.

Once the configuration is ready, the application initializes the MQTT client, registers an event handler, and starts the client. The client then establishes and manages the connection in the background.

Handling MQTT Events

The event handler responds to important changes in the client's state.

When the broker connection is established, the ESP32 can publish its online status and subscribe to the command topics it needs to receive.

When data arrives, the handler can inspect the received topic and payload. When the connection is lost, the application can record the event and allow the client to reconnect.

ESP-MQTT provides explicit lengths for received topics and payloads. These data buffers should not automatically be treated as null-terminated C strings.

Large messages may also arrive in fragments. The current data offset and total message length help the application identify the fragments and reconstruct the complete message.

The application should use a suitably bounded buffer, reassemble fragmented messages when necessary, and validate the complete payload before acting on it.

Publishing Temperature Readings

A sensor task can periodically obtain a temperature measurement and prepare a structured message for publication.

For example, the payload could contain a JSON field named temperature_c with a temperature value expressed in degrees Celsius.

Before publishing, the application should check that the message was formatted successfully and was not truncated by the available buffer size.

The reading can then be published to the telemetry topic using the selected QoS level and retain setting.

The return value from an ESP-MQTT publication call indicates whether the client accepted the publication for processing or sending. It does not prove that a subscriber received the message or acted on it.

For QoS 1, the MQTT_EVENT_PUBLISHED event indicates that the broker acknowledged the publication. If the application needs confirmation that a command was executed, it must implement a separate application-level acknowledgment.

The publication method also matters for real-time behavior. A call that performs network transmission can block while waiting for network activity, so it should not be placed in a time-critical execution path without considering its timing effects.

ESP-MQTT provides an enqueue mechanism that allows messages to be handed to the MQTT task for transmission. Another option is to use a FreeRTOS queue between the sensor-sampling task and the communications task. This separation helps keep sensor timing predictable.

Designing MQTT Topics and Message Payloads

Topic names should be organized around how applications consume the information rather than only around the internal structure of the firmware.

A hierarchy such as site, area, device, and message type can make a system easier to expand.

For example, an industrial monitoring system might use the following topics:

Separating telemetry, reported status, commands, and acknowledgments also makes it easier to apply different access permissions to different message types.

A telemetry payload might contain a timestamp, a temperature reading, and a sequence number. For example, the temperature could be represented as 27.45 degrees Celsius, accompanied by a sequence number of 1042.

The sequence number can help a receiver identify missing or repeated readings. Timestamps require a valid clock, which is commonly obtained through the Simple Network Time Protocol (SNTP).

The message format should define units, field names, and a schema version so that different applications interpret the data consistently.

MQTT transports payloads as bytes and does not interpret their contents. JSON is convenient during development because people can inspect it easily. Other formats, such as CBOR and Protocol Buffers, can reduce message size when bandwidth or device resources are limited.

Managing Reconnection and Offline Messages

Wireless connections can fail because of interference, weak signals, router restarts, or broker outages. A reliable design needs to decide what happens to readings generated while the device is disconnected.

If only the latest reading matters, the application may discard older measurements and publish the newest value after reconnecting.

If every sample must be preserved, the device may need a bounded queue or persistent storage. The application should define what happens when the queue becomes full.

The ESP-MQTT outbox uses RAM and does not preserve pending readings through power loss. Therefore, it should not be treated as durable storage.

Every device should also use a unique MQTT client ID. If another client connects using the same ID, the broker can disconnect the existing client. Repeated conflicts can cause both devices to experience continuing connection problems.

After reconnecting, the application should determine whether the broker restored the previous session instead of assuming that every earlier subscription is still active.

Large deployments need additional care. If hundreds or thousands of devices reconnect at the same fixed interval after a broker outage, they can overload the broker with simultaneous connection attempts.

Bounded exponential backoff with random jitter can help spread out those attempts when the library's existing retry behavior does not already provide this protection.

It is also useful to monitor connection failures, dropped messages, outbox occupancy, and publication delays. Network event handlers should remain short, with heavier processing delegated to application tasks.

Securing MQTT Before Deployment

Security needs to be part of the design before an IoT system is deployed outside a controlled development environment.

Use TLS and verify the broker certificate. Encryption without certificate verification does not adequately protect a device against broker impersonation. ESP-IDF provides certificate configuration options, including a directly supplied certificate and certificate-bundle attachment.

Use individual device credentials. Per-device credentials or client certificates make it easier to manage access and revoke a compromised device without changing the credentials of an entire fleet.

Restrict topic permissions. Broker access-control lists should limit each device to the topics it needs. A sensor should not automatically receive broad access to every device's command topics.

Protect stored credentials. Sensitive information should be stored securely, and the system should have a plan for credential rotation and revocation.

Restrict broker access. Network access should be limited to the intended clients, administrative changes should be logged, and abusive clients should be rate-limited where appropriate.

TLS consumes memory and adds connection-handshake time. Both should be measured on the target hardware using the intended certificate chain and configuration.

Certificate validity checks also depend on having a reliable clock. MQTT itself does not provide authorization rules; those rules must be enforced by the broker.

Testing MQTT Beyond a Successful Publication

A message appearing on a dashboard is only the beginning of the testing process. A dependable system must also behave correctly during failures and unexpected conditions.

The following checks are useful when evaluating the implementation:

  1. Confirm that subscribers receive readings on the intended topics.
  2. Verify that repeated QoS 1 commands do not cause a physical operation to be performed more than intended.
  3. Interrupt Wi-Fi and check that the device reconnects, restores the required subscriptions, and handles pending readings according to the design.
  4. Connect a new subscriber after a retained status message has been published and confirm that the expected retained value is received.
  5. Disconnect the device unexpectedly and check that the broker publishes the registered Last Will message within the expected Keep Alive detection window.
  6. Send malformed or oversized command payloads and confirm that they are rejected safely.
  7. Connect multiple devices using different client IDs and verify that they can remain connected simultaneously.
  8. Attempt to publish to or subscribe to topics outside the permitted access-control rules and confirm that unauthorized access is denied.
  9. Configure the device to connect to a broker with an untrusted certificate and verify that the connection is rejected.
  10. Run the system for an extended period and monitor memory usage, error counters, connection stability, and message delivery delays.

These tests help reveal problems that are not visible during normal operation. They should be repeated under conditions representative of the intended application before the system is deployed.

Conclusion

MQTT separates the device producing information from the applications that consume it. This makes it easier to distribute sensor readings, support additional subscribers, and organize communication across connected devices.

However, reliable MQTT communication involves more than publishing messages successfully. Topic organization, delivery guarantees, retained data, session behavior, duplicate handling, authentication, and recovery from network interruptions all influence the performance of the complete system.

A practical starting point is to define the message structure, configure a broker, establish a connection from the ESP32, and test normal message delivery. Once that works, test the failure cases and security settings before expanding the system to additional devices.