≡

Using confd and consul for config management

| Created | Modified

This tutorial demonstrates how to use confd and consul to manage application configurations. consul acts as a key/value store for your configuration data, while confd uses templates to generate configuration files based on the values in consul. This approach helps to decouple configuration from your application code.

Consul is used in clouds as a key/value store to hold configs, just like etcd, Zookeeper, AWS SSM, etc. Additionally, consul can provide a DNS server for service discovery.

In this tutorial, we want to test out how we can create configs for our applications in a development stack by running confd locally. This will take configuration templates and build configs from the information provided by consul.

The idea is that we just provide templates to our applications, and from these templates, generate our configurations from key/values stored on consul.

Advantages

Example

Install confd

We want to use a local confd – one file – thanks to Go and an easy install:

sudo wget -O /usr/local/bin/confd https://github.com/kelseyhightower/confd/releases/download/v0.16.0/confd-0.16.0-linux-amd64
sudo chmod a+x /usr/local/bin/confd

Run dockerized consul

Open up a new terminal and run consul, exposing the KV store to localhost on port 8500 (in memory only!):

docker run --rm --name=dev-consul -p 8500:8500 -e CONSUL_BIND_INTERFACE=eth0 consul

Store some values:

curl -X PUT -d 'db.example.com' http://localhost:8500/v1/kv/myapp/database/url
curl -X PUT -d 'Charles Brown' http://localhost:8500/v1/kv/myapp/database/user

This is how you get values:

curl -f http://localhost:8500/v1/kv/myapp/database/url?raw
curl -f http://localhost:8500/v1/kv/myapp/database/user?raw
curl -f http://localhost:8500/v1/kv/myapp/database/non-existent?raw

Add some config files to our working directory.

conf.d/myconfig.toml

[template]
src = "myconfig.conf.tmpl"
dest = "myconfig.conf"
keys = [
"/myapp/database/url",
"/myapp/database/user",
]

templates/myconfig.conf.tmpl

[myconfig]
database_url = {{getv "/myapp/database/url"}}
database_user = {{getv "/myapp/database/user"}}

We now have the following directory structure:

Directory Structure

├── conf.d
│ └── myconfig.toml
└── templates
└── myconfig.conf.tmpl

Create our configs by templates

$ confd -onetime -backend consul -node 127.0.0.1:8500 -confdir $(pwd)/
2018-06-07T13:42:41+02:00 refpad-16 confd[7689]: INFO Backend set to consul
2018-06-07T13:42:41+02:00 refpad-16 confd[7689]: INFO Starting confd
2018-06-07T13:42:41+02:00 refpad-16 confd[7689]: INFO Backend source(s) set to 127.0.0.1:8500
2018-06-07T13:42:41+02:00 refpad-16 confd[7689]: INFO Target config myconfig.conf out of sync
2018-06-07T13:42:41+02:00 refpad-16 confd[7689]: INFO Target config myconfig.conf has been updated

$ cat myconfig.conf
[myconfig]
database_url = db.example.com
database_user = Charles Brown

$ touch xxx >> myconfig.conf

$ confd -onetime -backend consul -node 127.0.0.1:8500 -confdir $(pwd)/
2018-06-07T13:44:49+02:00 refpad-16 confd[8151]: INFO Backend set to consul
2018-06-07T13:44:49+02:00 refpad-16 confd[8151]: INFO Starting confd
2018-06-07T13:44:49+02:00 refpad-16 confd[8151]: INFO Backend source(s) set to 127.0.0.1:8500
2018-06-07T13:44:49+02:00 refpad-16 confd[8151]: INFO myconfig.conf has md5sum 7a30123886573e65b3c9d31a4e1c1abf should be e919f09ba963caad1051c212c5ca9453
2018-06-07T13:44:49+02:00 refpad-16 confd[8151]: INFO Target config myconfig.conf out of sync
2018-06-07T13:44:49+02:00 refpad-16 confd[8151]: INFO Target config myconfig.conf has been updated

Summary

There are some more advanced examples on the consul GitHub. We aim for an infrastructure that is able to reconfigure while running, with notifications, monitoring, and service discovery. These topics are not covered here.

Also, there are better tools like remco or consul-template that support multiple backend sources, secret stores, and better authentication. Just dig into it.

Further readings: