Grafana: new query interface

The first change is that you don’t have to choose between query by tag and query by metric id anymore, you can do both at the same time. Querying by tag will refine the available list of metric names (much like a filter) and can result in multiple metrics from a single query. By selecting a metric name, you restrict the query to only display that one. This filtering is really nice when there’s tons of metrics available, like in the case of hundreds of OpenShift pods being monitored with the same tenant.

The simple key/value pairs interface is now replaced with a more elaborated query builder, following the pattern: 'tag-key' 'operator' 'tag-value(s)' ['AND/OR' etc.]

The following images show a walk-through:

The text fields include dynamic suggestions, you can use Grafana template variables within tag values, or enter free text. Once you’ve set up a tag query expression, the relevant metrics immediately show up on the chart and the list of available metrics in the dropdown list in updated.

This query builder lets you build almost any tag query that the Hawkular server understands. There are however some corner cases. For now this builder doesn’t allow you to prioritize expressions with parentheses. For instance, you cannot build c1 = 'foo' OR (b1 != 'B' AND a1 = 'abcd'). As a workaround you can turn off the query builder and directly type in your query expression.

It will be sent as is to the Hawkular Metrics server. This will also be useful to fill the gap if the language evolves server-side and this plugin isn’t updated immediately.

The other important feature is the ability to run stats queries against Hawkular Metrics, instead of raw queries. There are several reasons to do this:

To enable it, just clear the raw checkbox.

When you clear the raw checkbox, you can configure Multiple series aggregation to None, Sum or Average and can configure Stat type as avg, min, max, median, sum and different percentiles. You can display several different Stat types within the same query.

If the query returns multiple series, use Multiple series aggregation to define if and how to aggregate them. None will show them individually on the chart. But consider for instance the case of an OpenShift pod with many replicas, and you’re tracking their memory usage. It may be more relevant here to aggregate all of them, both sum and average are meaningful here.

The Stat type option refers to an aggregation at a different level: not between multiple metrics, but within a single metric, all raw datapoints are aggregated within time buckets.

These two improvements aim a common goal, that is facilitating querying over large amounts of data. This is becoming crucial especially in the context of microservices and applications running on container platforms, as the number of metrics explodes. Proper metrics tagging is the corner stone to make sense of this data.