Using confd and consul for config management

Consul is used in clouds as key/value store to holds configs just like etcd, Zookeeper, AWS SSM etc. Additionaly consul can provide a DNS server (service discovery).

In this tutorials we wanna test out, how we can create configs for our applications in development stack by local confd run. This will take configuration templates build configs by information provided by consul.

The idea is that we just provide templates to our applications and by these templates generate our configurations from key/values stored on consul.

Advantages

Applications repositories defined the config-format
No more versionless configs (yet we need to get enviornments straight!)

And hence itโ€™s consul weโ€™re one step closer to cloud-readiness

Example

Install confd

We wanna use local confd โ€“ one file โ€“ thanks to golang and easy install:

sudo wget -O /usr/local/bin/confd https://github.com/kelseyhightower/confd/releases/download/v0.16.0/confd-0.16.0-linux-amd64
sudo chmod a+x /usr/local/bin/confd

Run dockerized consul

Open up a new terminal and run consul kv exposed to localhost by port 8550 (in memory only!):

docker run --rm --name=dev-consul -p 8500:8500 -e CONSUL_BIND_INTERFACE=eth0 consul
Store some values
curl -X PUT -d 'db.example.com' http://localhost:8500/v1/kv/myapp/database/url
curl -X PUT -d 'Charles Brown' http://localhost:8500/v1/kv/myapp/database/user

# This is how you get values
curl -f http://localhost:8500/v1/kv/myapp/database/url?raw
curl -f http://localhost:8500/v1/kv/myapp/database/user?raw
curl -f http://localhost:8500/v1/kv/myapp/database/non-existent?raw
Add some config files to our working directory

conf.d/myconfig.toml

[template]
src = "myconfig.conf.tmpl"
dest = "myconfig.conf"
keys = [
"/myapp/database/url",
"/myapp/database/user",
]

templates/myconfig.conf.tmpl

[myconfig]
database_url = {{getv "/myapp/database/url"}}
database_user = {{getv "/myapp/database/user"}}
We now have folling directory structure:

Directory Structure

โ”œโ”€โ”€ conf.d
โ”‚ โ””โ”€โ”€ myconfig.toml
โ””โ”€โ”€ templates
โ””โ”€โ”€ myconfig.conf.tmpl

Create our configs by templates

$ confd -onetime -backend consul -node 127.0.0.1:8500 -confdir $(pwd)/ 
2018-06-07T13:42:41+02:00 refpad-16 confd[7689]: INFO Backend set to consul
2018-06-07T13:42:41+02:00 refpad-16 confd[7689]: INFO Starting confd
2018-06-07T13:42:41+02:00 refpad-16 confd[7689]: INFO Backend source(s) set to 127.0.0.1:8500
2018-06-07T13:42:41+02:00 refpad-16 confd[7689]: INFO Target config myconfig.conf out of sync
2018-06-07T13:42:41+02:00 refpad-16 confd[7689]: INFO Target config myconfig.conf has been updated
$ cat myconfig.conf 
[myconfig]
database_url = db.example.com
database_user = Charles Brown
$ touch xxx >> myconfig.conf
$ confd -onetime -backend consul -node 127.0.0.1:8500 -confdir $(pwd)/ 
2018-06-07T13:44:49+02:00 refpad-16 confd[8151]: INFO Backend set to consul
2018-06-07T13:44:49+02:00 refpad-16 confd[8151]: INFO Starting confd
2018-06-07T13:44:49+02:00 refpad-16 confd[8151]: INFO Backend source(s) set to 127.0.0.1:8500
2018-06-07T13:44:49+02:00 refpad-16 confd[8151]: INFO myconfig.conf has md5sum 7a30123886573e65b3c9d31a4e1c1abf should be e919f09ba963caad1051c212c5ca9453
2018-06-07T13:44:49+02:00 refpad-16 confd[8151]: INFO Target config myconfig.conf out of sync
2018-06-07T13:44:49+02:00 refpad-16 confd[8151]: INFO Target config myconfig.conf has been updated

Summary

There are some more advances examples on consul GitHub. We aim for an infrastructure that is able to reconfigure while running. Notifications. Monitoring. Service-Descovery. These topics are not covered here.

Also there are better tools like remco or consul template that support multiple backend sources and do secret stores and better auth.Just dig into it.

Further readings
– How Should I Get Application Configuration into my Docker Containers?
– DigitalOcean: How To Use Confd and Etcd to Dynamically Reconfigure Services
– Consul security model and Hashicorpโ€™s Vault