Introducing Sky-Weather: An At-Home Real-Time Streaming Weather Station!
Sky-Weather is an at-home weather station that enables real-time monitoring of temperature and humidity levels across designated areas. This project harnesses the power of Python, Raspberry Pis, Google Cloud, and more. It seamlessly integrates these tools to deliver accurate and up-to-date weather data. In this blog post, we will explore the project, providing a comprehensive technical description.
Let’s Dive In!
*The code can be found in the GitHub Repository here.
Architecture
The Sky-Weather station utilizes Raspberry Pi Pico Ws as the gatherers. It then utilizes Google Cloud Pub/Sub to receive weather data, which is then streamed through Apache Beam (Google Cloud Dataflow runner). This streamed data ends up landing in Google Cloud BigQuery.
For CI/CD, this project utilized Terraform and GitHub Actions.
See the architectural diagram below:
Parts and Software utilized in the Project
The following parts were utilized in the project
Hardware
- 2 Raspberry Pi Pico W
- 2 Si7021 Temperature, and Humidity Sensors (Adafruit)
- 2 Micro USB Power Cords for the Pis
- 2 QT Wires (for connecting Pis to sensors)
Software for Pis
- Thonny IDE (for flashing the Raspberry Pi Pico W’s)
- MicroPython
- URequests
- MicroPython_Base64 Library
- MicroPython_Si7021 Library
Software for Streamer
- Google Cloud SDK
- Apache Beam SDK
- Python 3.10
- Google Cloud Pub/Sub
- Google Cloud Dataflow
- Google Cloud BigQuery
- Google Cloud Storage
CI/CD/DevOps Infrastructure
- GitHub Actions
- Terraform
- Docker/Docker-Compose
Environment Tools
- flake8
- black
- isort
- PyCharm
Assembling the Raspberry Pi Pico Ws
This step is simple once you gather the Si7021 weather sensors, the Pis, and the QT wires. You will need to attach the QT Wires into the appropriate GPIO Pins (specifics can be found in weather station’s documentation)
The Weather Recording and Message Publishing Python Code
In this section, I will give a high-level overview of the Python code utilized to record temperatures and publish the data to Google Cloud Pub/Sub.
We will not cover the Dummy Data Generator.
This code base for the “Publisher” is ran with Micro-Python within the Raspberry Pi Pico Ws, so it will be missing access to some libraries, such as datetime and requests.
def stream(self):
# ~~~Be a meterologist forever~~~
# ,--.::::::::::::::::::::::::::::::::::::....:::::::
# )::::::::::::::::::::::::::::::::.. ..::::
# _'-. _:::::::::::::::::::::::::::.. ,--. ..::
# ( ) ),--.::::::::::::::::::::::. ( ) .::
# )-._::::::::::::::::::.. `--' ..::
# _________________):::::::::::::::::::.. ..::::
# ::::::::::::::::::::::::::::::::::::::::....:::::::
# ::::::::: :: A drip of water... :::::::::::::::::::
# :::::::: c> :::::::::::::::::::::::::::::::::::::::
# !:!:!:!:<(; !:!:!:!:!:!:!:!:!:!:!:!:!:!:!:!:!:!:!:!
# !!!!!!!! /\ !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
# |!|!|!|!|!|!|!|!|!|!|!|!|!|!|!|!|!|!|!|!|!|!|!|!|!|
# |||||||||||||||||||||||||||||||||||||||||||||||||||
# I|I|I|I|I|I|I|I|I|I|I|I|I|I|I|I|I|I|I|I|I|I|I|I|I|I
# IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII
#
led = machine.Pin("LED", machine.Pin.OUT)
led.on()
try:
# Connect to WiFi
self.connect_to_wifi()
# Declare machines
i2c = machine.I2C(0)
si7021 = SI7021(i2c)
rtc = machine.RTC()
gcp_access_token = self._refresh_gcp_access_token()
while True:
refresh_access_token_bool = self.collect_weather_and_post_to_pubsub(
si7021=si7021,
rtc=rtc,
access_token=gcp_access_token,
)
if refresh_access_token_bool:
gcp_access_token = self._refresh_gcp_access_token()
else:
led.off()
sleep(30)
led.on()
finally:
led.off()This code starts by turning on the Pi’s LED light to indicate that it is running. Next it connects to the configuration-specified WiFi network. From there it generates an instance of a Si7021 wrapper class (located here). After this, it generates the first access token for Google Cloud Pub/Sub publishing, and starts to collect and gather weather data every 30 seconds.
The Streaming of Data to BigQuery
The streamer code utilizes Apache Beam SDK and utilizes Google Cloud Dataflow as a runner.
It runs with the following Apache Beam pipeline code:
with beam.Pipeline(options=pipeline_options) as pipeline:
# noinspection PyTypeChecker
(
pipeline
| "Read from PubSub"
>> beam.io.ReadFromPubSub(subscription=input_subscription)
| "Decode" >> beam.Map(lambda x: x.decode("utf-8"))
| "Prepare for BigQuery" >> beam.ParDo(PrepareDataForBigQuery())
| "Write to BigQuery"
>> beam.io.WriteToBigQuery(
table_ref,
schema=bq_schema_str,
create_disposition=beam.io.BigQueryDisposition.CREATE_IF_NEEDED,
write_disposition=beam.io.BigQueryDisposition.WRITE_APPEND,
)
)In this code the pipeline does the following:
- reads from Pub/Sub subscription
- Decodes to UTF-8
- Prepares the data to be streamed into BigQuery (does this with a parallel operation, “ParDo”)
- Writes the prepared data to BigQuery
CI/CD
The CI/CD code utilizes GitHub Actions and Terraform.
The GitHub Actions code simply does linting checks (black, flake8, isort) and Terraform linting/validation checks. For deployment, GitHub Actions applies the Terraform deployment.
The Terraform deployment creates a GCS Bucket, Pub/Sub topic & subscription, BigQuery dataset & table, and Dataflow streaming job.
Conclusion
This blog post was a quick description of the Sky-Weather project. It discussed the project’s architecture, technologies used, and a high-level overview of some of the code. I hope it was fun to read and encourages you to play around with IoT/Embedded Systems projects.
Thank you, and Happy Coding!
