We will be using the same example as used in the previous
article.
However since writing that article, the configuration of the tracer and Prometheus metrics support has been
simplified. There is now no explicit configuration of either, with only some auto configuration of MetricLabel beans
to identify some custom labels to be added to the Prometheus metrics, e.g.
Metrics configuration used in both services:
@Configuration
public class MetricsConfiguration {
@Bean
public MetricLabel transactionLabel() {
return new BaggageMetricLabel("transaction", "n/a"); (1)
}
@Bean
public MetricLabel versionLabel() {
return new ConstMetricLabel("version", System.getenv("VERSION")); (2)
}
}
| 1 |
This metric label identifies the business transaction associated with the metrics, which can be used to isolate the
specific number of requests, duration and errors that occurred when the service was used within the particular business
transaction |
| 2 |
This metric label identifies the service version, which is especially useful in the Canary deployment use case being
discussed in this article |
The first step is to following the instructions in the example
for deploying and using the services within OpenShift.
Once the ./genorders.sh script has been running for a while, to generate plenty of metrics for version 0.0.1 of the
services, then deploy the new version of the services. This is achieved by:
-
updating the versions in the pom.xml files, within the simple/accountmgr and simple/ordermgr folders
from 0.0.1 to 0.0.2
-
re-run the mvn clean install docker:build command from the simple folder
-
deploy the canary versions of the services using the command oc create -f services-canary-kubernetes.yml
As our services accountmgr and ordermgr determine the backing deployment based on the respective labels
app: accountmgr and app: ordermgr, simply having a second deployment with these labels will make them serve requests
in a round-robin manner.
This deployment script has been pre-configured with the 0.0.2 version, and to only start a single instance of the
new version of the services. This may be desirable if you want to monitor the behaviour of the new service versions over
a reasonable time period, but as we want to see results faster we will scale them up to see more activity. You can do this
by expanding the deployment area for each service in the OpenShift web console and selecting the up arrow to scale
up each service:
Figure 2: Scaling up canary deployment
Now we can monitor the Prometheus dashboard, using the following query, to see the error ratio per service and version:
sum(increase(span_count{error="true",span_kind="server"}[1m])) without (pod,instance,job,namespace,endpoint,transaction,error,operation,span_kind) / sum(increase(span_count{span_kind="server"}[1m])) without (pod,instance,job,namespace,endpoint,transaction,error,operation,span_kind)
The result of this query can be seen in Figure 1 at the top of the article.
This chart shows that version 0.0.2 of the accountmgr service has not generated any errors, while the 0.0.2
of the ordermgr appears to be less error prone than version 0.0.1.
Based on these metrics, we could decide that the new versions of these services are better than the previous, and
therefore update the main service deployments to use the new versions.
In the OpenShift web console you can do this by clicking the three vertical dots in the upper right hand side of the
deployment region and selecting Edit YAML from the menu.
This will display an editor window where you can change the version from 0.0.1 to 0.0.2 in the YAML file.
Figure 3: Update the service version
After you save the YAML configuration file, in the web console you can see the service going through a "rolling update" as OpenShift incrementally changes each service instance over to the new version.
After the rolling update has completed for both the ordermgr and accountmgr services, then you can scale down or
completely remove the canary version of each deployment.
An alternative to performing the rolling update would simply be to name the canary version something else (i.e. specific
to the version being tested), and when it comes time to switch over, simply scale down the previous deployment version. This
would be more straightforward, but wouldn’t show off the cool rolling update approach in the OpenShift web console :-)
Figure 5: Scaling down canary deployment
|
|
Although we have updated both services at the same time, this is not necessary. Generally microservices would be managed
by separate teams and subject to their own deployment lifecycles.
|