How the Hardware Sentry Dashboards for Grafana Came True

Marc Douguedroit is a senior developer and co-founder of Sentry Software. His deep expertise in hardware monitoring and observability platforms, and his ability to develop solutions with the customer in mind made him the perfect candidate to create Hardware dashboards for Grafana. He agreed to share his experience in this post.

So, Marc, tell us how the project started...

The project started right after we redeveloped Hardware Sentry as an OpenTelemetry Collector and managed to push the collected hardware metrics and sustainability data to Prometheus. It was a good start, but we needed to give more to our final users. Getting raw data is a good thing, but being able to leverage and visualize this data is even better. We decided to create dashboards to showcase the power of Hardware Sentry. Our choice naturally went to Grafana, THE observability platform for Prometheus.

After developing solutions for BMC Software for many years, implementing dashboards for a new platform must have been quite a challenge...

You're right. It was the biggest challenge I faced. I had to consider the project from a totally new monitoring perspective. BMC TrueSight, with its PATROL Agent, and Prometheus and Grafana are entirely different technologies. I had to change my mindset while adapting an existing product for a different audience.

How are BMC and Prometheus/Grafana users so different?

In BMC TrueSight, a metric with no alert serves no purpose. Users do not usually look at the collected data, they are only interested in the warnings and alerts generated by the product. In addition, they prefer a product that needs little customization and works by itself.

This is the opposite in the Prometheus world. Users want data, as much as possible, and they want to decide what to do with it. They want to build their own PromQL queries, set their own alerts and build or customize dashboards tailored to their environment.

Thus, our dashboards had to be an example of data visualization that would inspire the final users. The dashboards are not necessarily the final product as Grafana users would have the final say in how the data collected by Hardware Sentry is used.

You mentioned PromQL queries. can you tell us more about it?

PromQL is the biggest benefit of Prometheus. It's easy to learn once you've understood its concepts, and it opens up a lot of possibilities. One metric can be used in many ways, and when combined with the metric labels, the queries can be quite powerful. The more I learn about it, the more I like it!

Adapting Hardware Sentry to a new audience was your biggest challenge. did you face others?

The other main challenge was to select which data to highlight in the dashboards, and which data to discard. Hardware Sentry collects lots of hardware information about a wide variety of server brands, disk arrays, switches, etc. The collected data is never the same from one model to another. You can imagine that consolidating this data can be very tricky.

I can imagine, yes. So how did you proceed?

Let's say that it has been an iterative process, and it took me several months - with interruptions obviously - to come up with a version I was proud enough to share on Grafana Labs. My first version was a full interface. You could drill from the Sites overview down to the status of a physical disk. It had 24 dashboards, which was way too many since this made the installation complex and future upgrades even more complicated. I wanted to provide as much information as possible, but this was not in the spirit of Grafana dashboards. Grafana users want a clear overview of their infrastructure in a single dashboard; they do not want a PATROL Console (laugh).

I finally lowered the number of dashboards to 3:

• The Hardware Sentry - Main dashboard shows the overall status of the sites
• The Hardware Sentry - Site dashboard provides an overview of the site and its hosts
• The Hardware Sentry - Host dashboard shows a clear picture of a host and its hardware components

I only kept the most valuable and commonly collected data among the hundreds of server models supported by Hardware Sentry.

What has been the most difficult information to display?

Displaying the sustainability data, history of a site's energy consumption, energy costs, and CO₂ emissions, for example, was by far the most complicated thing to do from a technical point of view. Prometheus and Grafana are built to show a snapshot of a metric, or to compare metrics with their past values. It's not really designed to show daily historical data. Building the appropriate PromQL queries to calculate daily, monthly, and annual values was quite challenging, and displaying them in Grafana maybe even more! I had to use undocumented features to be able to show aggregated charts. I don't think I have seen historical data shown this way in any other dashboard.

Does the outcome meet your expectations?

We have come to a good compromise between showing as much data as possible while keeping the dashboards visually appealing and easy to skim through. The critical metrics are straightforward and browsing through the panels is simple.

There's still room for improvement, and I hope final users will send us their suggestions so I can make them even better.

What did you like the most while creating grafana dashboards?

I had to rewire my brain. Seeing things from a Prometheus/Grafana user point of view is very different than what I've been used to. I had to think differently.

What did you like the least while creating Grafana dashboards?

Excluding labels from table panels almost drove me... insane (especially when we had a lot of tables)! For every query, I had to exclude the labels I did not want to be displayed one by one. All our metrics have a lot of labels, sometimes more than a dozen. Let me tell you... this takes a lot of time and a lot of mouse clicks!

According to you, what are the strengths and weaknesses of grafana?

The Grafana team is very responsive, and Grafana is evolving quickly. New versions come out very often, showing the team is active and believes in their product.

Its weakness is that it can be a little complicated to learn. Keeping up with the new versions and the new panels can also be a little challenging.

What advice would you give to someone who wants to build their first Grafana dashboard?

I may be stating the obvious, but I recommend starting by reading the documentation and looking at the most popular dashboards to figure out the capabilities of Grafana and users' expectations. There are also a couple of videos to get started and make the right choices.

Thanks to Marc for sharing his experience in developing dashboards with Grafana. Hardware Sentry Dashboards can be downloaded from Grafana Labs. Visit Sentry Software Web site for more information about our product.

Watch it Live Talk to an expert