Zharp Agent Setup

One command installs the agent, finds what runs on the server and starts collecting. There is no config file to write.

The agent is an OpenTelemetry Collector built for Zharp. It sends host metrics, logs, container and database metrics and application traces, and lets you monitor databases that aren't reachable from the internet.

# PostgreSQL — as a superuser in psql
CREATE USER zharp_monitor WITH PASSWORD 'choose-a-password';
GRANT pg_monitor TO zharp_monitor;

# then on the server
sudo zharp-collector add postgres --password 'choose-a-password'

Application logs

Besides log files, the agent collects what your applications print, each under its own service name in Logs:

• Docker containers — read through the Docker API, from the moment the agent starts (history isn't replayed); containers started later are picked up from their first line. The service is the zharp.service label, else the container's ZHARP_SERVICE variable, else the Compose service name or the container name; well-known images (nginx, postgres, redis, python …) also set the source. To leave a container out, give it the label zharp.logs=false. The agent's own container is never read.
• systemd services (Linux) — read from the journal, named after the unit without .service, with the level taken from the entry's priority. Operating-system units are left out: systemd and systemd-*, dbus, cron, ssh, polkit, NetworkManager, snapd, udisks2, accounts-daemon, rsyslog, getty, user@ and session-* scopes, sudo, su, login, the kernel and the agent itself. While the journal is read, /var/log/syslog and /var/log/messages aren't tailed as well — they hold the same lines — and System logs keeps only auth.log and kern.log.
• PM2 (Linux) — the logs in ~/.pm2/logs of root and every user, named after the app (cluster instances are merged). The agent runs as its own user, so the install gives it read access to exactly those folders with ACLs — read on the logs folder, pass-through on the folders above it; home folders are never chmod-ed. Where that isn't possible (no setfacl), the host shows PM2 as Needs setup with the commands to run.

Stack traces are sent as one record, and card numbers, bearer tokens, JWTs, AWS keys and private keys are masked before anything leaves the server. Add your own exclude and mask rules with sudo zharp-collector add rule, or per container with labels — see Log Configuration.

A restart of the agent carries on where it stopped, without sending lines twice. To install without application logs, add --no-app-logs (or set ZHARP_NO_APP_LOGS=1):

# Linux and macOS: install without application logs
curl -fsSL https://api.zharp.io/api/v1/install/agent.sh | sh -s -- --token YOUR_TOKEN --no-app-logs

# Windows: set this before the install command
$env:ZHARP_NO_APP_LOGS="1"

# add them later, one source at a time
sudo zharp-collector add docker-logs
sudo zharp-collector add journal
sudo zharp-collector add pm2

More than one database of an engine

One agent can report on several PostgreSQL, MySQL, Redis or MongoDB servers. The first of each engine needs no name. Add every further one with --name (letters, digits, - or _); it is listed on the host as, for example, PostgreSQL (orders) and can be picked for its own private monitor. Running add again without a name only updates the first database's password. Agents set up before this keep working as they are.

# a second (third, ...) database of the same engine gets a name of its own
sudo zharp-collector add postgres --name orders --host 10.0.0.12 --password 'choose-a-password' --db orders

Service, environment and version

Everything the agent sends — logs, metrics and traces — carries the same tags: service.name, deployment.environment.name, service.version, and for logs zharp.source, the technology that wrote the line. Set them with ZHARP_SERVICE / ZHARP_ENV / ZHARP_VERSION, the Docker labels zharp.service, zharp.env, zharp.version, zharp.source, the Kubernetes annotations zharp.io/service, zharp.io/env, zharp.io/version, zharp.io/source, or OpenTelemetry resource attributes. Unified service tagging has the details and examples.

Useful commands

Private database monitors

Once a database is collecting, it can be monitored without opening it to the internet. Create a monitor, choose Database monitor, set Connection to Private and pick it under Agent connection by server — for example PostgreSQL on db-01, or PostgreSQL "orders" on db-01 for a named one. The monitor goes down when the agent can't connect to the database, when the database is close to its connection limit, or when the agent stops reporting. Private Databases has the details.