IntroductionIt’s been two years since I’ve written an article. It’s never too late to start again, especially for a subject that covers the whole chain from physical measurement to its analysis in a simple way. We will therefore talk about a subject that seems particularly important to me: how to record the measurements of sensors that send their data asynchronously, visualize their live status or analyze their past measurements. To do this, we will talk about humidity, temperature and atmospheric pressure sensors in the field of home automation. The objective will not be to reinvent Home Automation (HA), but to explore Node-Red and the excelent holoviz Holoviews & Panel libraries using sensors costing a few euros. The same principle could be applied to any field. In a few sections:
- Hardware: from Zigbee to MQTT
- Dashboard live data using Holoviews stream
- Record data using Node-Red and PostgreSQL
- Dashboard historical data
- a computer that can run Zigbee2MQTT[EN], here a RaspberryPi 3B+.
- A zigbee sniffer. I opted for one advised on a guide by projetsdiy.fr: le CC2531 de Texas Instrument[FR] (links are certainly sponsored), but others are compatible [EN].
- A set of sensors using the Zigbee protocol and preferably compatible with the Zigbee2MQTT library [EN]. Here, a set of temperature, humidity, pressure sensors by Xiaomi are used.
Setting upWe won’t be talking details here. Each module is thoroughly documented by each project website. I’ll be giving only an overview with specifics for this project.
Zigbee sniffer and MQTT brokerThe documented is very good on the main website: https://www.zigbee2mqtt.io. A small tip: note the time and the order for which you reset your sensors. You can then find their id back on the journal using
journalctl -u zigbee2mqtt.service. Then you can give them friendly names in the configuration file
/opt/zigbee2mqtt/data/configuration.yaml. You must also add the
retainoption for each sensor. We need your broker to record the last sent data by your sensor. Otherwise, if nobody is listening, the information is lost for ever. In the case of Zigbee sensor, new data may take up to one hour to be sent (remember, it’s low consumption). When rebooting your application, you may still want to know what was the last data and when it was.
Note: Xiaomi sensors have a small button to send current measurements right away.Extract from the configuration file:
TH for Temperature, Humidity.
devices: '0x00158d00054692fd': friendly_name: TH_Chambre retain: true
DatabaseI have opted for PostgreSQL that you can simply install with
sudo apt install postgresql postgresql-contribI advise you to make a user, let’s call it pi, with remote access to a database called sensors. I have added a schematics th where one table per sensor will be automatically made. We’ll come back to in in the section about Node-Red.
Node-RedThe documentation is here as well thorough: https://nodered.org/docs/getting-started/raspberrypi
Python 3.7 and its librariesLuckily, the last Raspbian version is compatible with Python 3.7. Simply calling
sudo apt install python3will suffice (python2 stays the default engine). Then:
pip3 install paho-mqttfor MQTT.
pip3 install holoviews panel hvplotfor the dashboard
pip3 install sqlalchemy psycopg2to access the database.
Visualize live data
A little bit of python…You can reproduce herebelow notebook or check it on github here. To executed from a computer without jupyter (like a RPi), you can convert it to a python file with
jupyter nbconvert yournotebook.ipynb --to python
A full fledge DashboardTo make it a Panel application, you must call the
servablemethod from a Panel layout as the last line of your jupyter notebook. Using above example:
Then from your RPi terminal (or any other server ), call:
If you want to make it accessible from your local network like a normal http website (on port 80), you must bind the python3.7 execute as follows:
panel serve yournotebook.ipynb --port 8080 --address youraddress
sudo setcap 'cap_net_bind_service=+ep' /usr/bin/python3.7
Note that all python3.7 application will have access to ports under 1024, which can be considered as a security issue.You can find my whole application on GitHub following this link. You’ll need to adapt it to your configuration of course!
Make it a serviceYou now have a new app. But you want it to start automatically without having to prompt passwords and such. Then make it a service! It is very simple. You first need to create a file in the right folder, as follows:
>> sudo nano /etc/systemd/system/home_on_bokeh.service
Description=Serve panel to access zigbee sensor data After=multi-user.target [Service] Type=simple ExecStart=/home/pi/.local/bin/panel serve /home/pi/home_on_bokeh/plotting_live_data.py --port 80 --address yourrpiaddress Restart=on-abort User=pi [Install] WantedBy=multi-user.target
yourrpiaddressand, if necessary,
--port 80for an accessible port. You now need to define the broker address and authentification as well as Bokeh paramters as environmental variables:
>> sudo systemctl edit home_on_bokeh.service
This file is accessible under
[Service] Environment="MQTT_ADDRESS=localhost" Environment="MQTT_USER=pi" Environment="MQTT_PASSWORD=verycomplexpassphrase" Environment="BOKEH_ALLOW_WS_ORIGIN=raspberrypi.local:80,192.168.10.10:80"
BOKEH_ALLOW_WS_ORIGIN variable: without it, your thing isn’t accessible! Change the address here by your real address.
You can now start the service:
sudo chmod 600 /etc/systemd/system/home_on_bokeh.service.d/override.conf
enable it for automated start:
>> sudo systemctl start home_on_bokeh
check that everything goes well (no
>> sudo systemctl enable home_on_bokeh
And check the logs:
>> systemctl status home_on_bokeh
You can now access the dashboard from any of your devices present on your local network! Next time, I’ll discuss how Node-Red can be used to record the data then make a new dashboard to look at them (hint: it’s already on GitHub).
>> journalctl -u home_on_bokeh -r