Prometheus

Prometheus is monitoring system and time series database.

Note: In the following we will assume that your instance is called hello (default) as well as that the hostname is hello as well.

Installing the Prometheus Node Exporter

If all you want to do is monitor your gokrazy installation, just add the Prometheus node exporter:

gok add github.com/prometheus/node_exporter

Installing the Prometheus Time Series Database

To install the prometheus monitoring system, add the prometheus program to your gokrazy instance:

gok add github.com/prometheus/prometheus/cmd/prometheus

We need a valid prometheus config for prometheus to start successfully. Start with the default prometheus.yml from the prometheus repository. Save it to your gokrazy instance directory, e.g. ~/gokrazy/hello/prometheus.yml.

Open your instance’s config.json in your editor:

gok edit

To use the prometheus config file, we need to do two things in the gokrazy config:

  1. Copy over the prometheus.yml with Package config: Extra files
  2. Add Package config: Command-line flags
  3. We also add WaitForClock to get accurate timestamps for our timeseries data

Your config should look something like this:

{
    "Hostname": "hello",
    "Packages": [
        "github.com/gokrazy/fbstatus",
        "github.com/gokrazy/hello",
        "github.com/gokrazy/serial-busybox",
        "github.com/gokrazy/breakglass",
        "github.com/prometheus/prometheus"
    ],
    "PackageConfig": {
        "github.com/prometheus/prometheus": {
            "CommandLineFlags": [
                "--config.file=/etc/prometheus/prometheus.yml"
            ],
            "WaitForClock": true,
            "ExtraFilePaths": {
                "/etc/prometheus/prometheus.yml": "prometheus.yml"
            }
        }
    }
}

Then, deploy as usual:

gok update

In theory, this is enough to deploy prometheus, but unfortunately we are missing the web assets for the web UI which are not included in our build of prometheus.

You can use prometheus as is, if you never want to go to the web UI on hello:9090. If you visit the web UI anyway, you will get this (or a similar) error:

Error opening React index.html: open web/ui/static/react/index.html: no such file or directory

To get the web UI to work, we first need the missing files. You have two options:

Option A: Build Web Assets

Follow Prometheus → Building from source, meaning:

git clone https://github.com/prometheus/prometheus
cd prometheus
make assets-compress

Instruct gokrazy to use this build of Prometheus:

gok -i hello add .

Configure gokrazy to build Prometheus with the builtinassets Go build tag:

{
    "Hostname": "hello",
    "Packages": [
        "github.com/gokrazy/fbstatus",
        "github.com/gokrazy/hello",
        "github.com/gokrazy/serial-busybox",
        "github.com/gokrazy/breakglass",
        "github.com/prometheus/prometheus"
    ],
    "PackageConfig": {
        "github.com/prometheus/prometheus": {
            "GoBuildTags": [
                "builtinassets"
            ],
            "CommandLineFlags": [
                "--config.file=/etc/prometheus/prometheus.yml"
            ],
            "WaitForClock": true,
            "ExtraFilePaths": {
                "/etc/prometheus/prometheus.yml": "prometheus.yml"
            }
        }
    }
}

To update Prometheus, you’ll need to update your Prometheus dir instead of using gok get:

git pull && make assets-compress

Option B: Download Web Assets

The web assets are released with every prometheus release and you need to update them by hand with every new version.

  1. Go to https://github.com/prometheus/prometheus/releases
  2. Expand the section “Assets” for the lastest release
  3. Download prometheus-web-ui-VERSION.tar.gz with VERSION being something like 2.46.0 (example url for version 2.46.0: https://github.com/prometheus/prometheus/releases/download/v2.46.0/prometheus-web-ui-2.46.0.tar.gz)
  4. Extract the archive file: tar xf prometheus-web-ui-2.46.0.tar.gz
  5. Enable breakglass via gokrazy web
  6. Create the right folder on your device: ssh hello mkdir -p /perm/home/prometheus/web/ui
  7. Copy over the static/ directory you exctracted from the tar file: scp -r static/ hello:/perm/home/prometheus/web/ui
  8. Go to hello:9090 to check if the web UI is working