Introduction
It’s been two years since I’ve written an article. It’s never too late to start again, especially for a subject that covers the whole chain from physical measurement to its analysis in a simple way. We will therefore talk about a subject that seems particularly important to me: how to record the measurements of sensors that send their data asynchronously, visualize their live status or analyze their past measurements. To do this, we will talk about humidity, temperature and atmospheric pressure sensors in the field of home automation. The objective will not be to reinvent Home Automation (HA), but to explore Node-Red and the excelent holoviz Holoviews & Panel libraries using sensors costing a few euros. The same principle could be applied to any field. In a few sections:- Hardware: from Zigbee to MQTT
- Dashboard live data using Holoviews stream
- Record data using Node-Red and PostgreSQL
- Dashboard historical data
Hardware
- a computer that can run Zigbee2MQTT[EN], here a RaspberryPi 3B+.
- A zigbee sniffer. I opted for one advised on a guide by projetsdiy.fr: le CC2531 de Texas Instrument[FR] (links are certainly sponsored), but others are compatible [EN].
- A set of sensors using the Zigbee protocol and preferably compatible with the Zigbee2MQTT library [EN]. Here, a set of temperature, humidity, pressure sensors by Xiaomi are used.
Setting up
We won’t be talking details here. Each module is thoroughly documented by each project website. I’ll be giving only an overview with specifics for this project.Zigbee sniffer and MQTT broker
The documented is very good on the main website: https://www.zigbee2mqtt.io. A small tip: note the time and the order for which you reset your sensors. You can then find their id back on the journal usingjournalctl -u zigbee2mqtt.service
. Then you can give them friendly names in the configuration file /opt/zigbee2mqtt/data/configuration.yaml
.
You must also add the retain
option for each sensor. We need your broker to record the last sent data by your sensor. Otherwise, if nobody is listening, the information is lost for ever. In the case of Zigbee sensor, new data may take up to one hour to be sent (remember, it’s low consumption). When rebooting your application, you may still want to know what was the last data and when it was.
Note: Xiaomi sensors have a small button to send current measurements right away.
Extract from the configuration file:devices:
'0x00158d00054692fd':
friendly_name: TH_Chambre
retain: true
TH for Temperature, Humidity.
Database
I have opted for PostgreSQL that you can simply install withsudo apt install postgresql postgresql-contrib
I advise you to make a user, let’s call it pi, with remote access to a database called sensors. I have added a schematics th where one table per sensor will be automatically made. We’ll come back to in in the section about Node-Red.
Node-Red
The documentation is here as well thorough: https://nodered.org/docs/getting-started/raspberrypiPython 3.7 and its libraries
Luckily, the last Raspbian version is compatible with Python 3.7. Simply callingsudo apt install python3
will suffice (python2 stays the default engine).
Then:
pip3 install paho-mqtt
for MQTT.pip3 install holoviews panel hvplot
for the dashboardpip3 install sqlalchemy psycopg2
to access the database.
Visualize live data
A little bit of python…
You can reproduce herebelow notebook or check it on github here. To executed from a computer without jupyter (like a RPi), you can convert it to a python file withjupyter nbconvert yournotebook.ipynb --to python
A full fledge Dashboard
To make it a Panel application, you must call theservable
method from a Panel layout as the last line of your jupyter notebook. Using above example:
layout=pn.panel(room_plots)
layout.servable()
Then from your RPi terminal (or any other server ), call:
panel serve yournotebook.ipynb --port 8080 --address youraddress
If you want to make it accessible from your local network like a normal http website (on port 80), you must bind the python3.7 execute as follows:
sudo setcap 'cap_net_bind_service=+ep' /usr/bin/python3.7
Note that all python3.7 application will have access to ports under 1024, which can be considered as a security issue.
You can find my whole application on GitHub following this link. You’ll need to adapt it to your configuration of course!Make it a service
You now have a new app. But you want it to start automatically without having to prompt passwords and such. Then make it a service! It is very simple. You first need to create a file in the right folder, as follows:>> sudo nano /etc/systemd/system/home_on_bokeh.service
Description=Serve panel to access zigbee sensor data
After=multi-user.target
[Service]
Type=simple
ExecStart=/home/pi/.local/bin/panel serve /home/pi/home_on_bokeh/plotting_live_data.py --port 80 --address yourrpiaddress
Restart=on-abort
User=pi
[Install]
WantedBy=multi-user.target
change yourrpiaddress
and, if necessary, --port 80
for an accessible port. You now need to define the broker address and authentification as well as Bokeh paramters as environmental variables:
>> sudo systemctl edit home_on_bokeh.service
[Service]
Environment="MQTT_ADDRESS=localhost"
Environment="MQTT_USER=pi"
Environment="MQTT_PASSWORD=verycomplexpassphrase"
Environment="BOKEH_ALLOW_WS_ORIGIN=raspberrypi.local:80,192.168.10.10:80"
This file is accessible under /etc/systemd/system/home_on_bokeh.service.d
.
Note the BOKEH_ALLOW_WS_ORIGIN
variable: without it, your thing isn’t accessible! Change the address here by your real address.
>> sudo systemctl start home_on_bokeh
enable it for automated start:
>> sudo systemctl enable home_on_bokeh
check that everything goes well (no sudo
is necessary)
>> systemctl status home_on_bokeh
And check the logs:
>> journalctl -u home_on_bokeh -r
You can now access the dashboard from any of your devices present on your local network!
Next time, I’ll discuss how Node-Red can be used to record the data then make a new dashboard to look at them (hint: it’s already on GitHub).