Prometheus — What It Is? Exposing and Scraping Go Service Metrics

Jul 22, 2026 posted by Ilman Iqbal

Prometheus is a time-series database that collects metrics from your apps. It does not run inside your Go service. Your service only exposes numbers; Prometheus pulls them on a schedule.

Your Go service                    Prometheus
┌─────────────────┐               ┌─────────────────┐
│  Gin / gRPC     │               │  Scrapes every  │
│  /metrics       │ ◄── pull ──── │  10 seconds     │
│  (counters,     │               │  Stores history │
│   histograms)   │               │  Evaluates rules│
└─────────────────┘               └─────────────────┘

Without Prometheus	With Prometheus
Your app knows "I got 42 requests" right now	Prometheus stores that number every 10 seconds
You can only see current state via `curl /metrics`	You can see trends: "requests went up at 3pm"
No alerts	"CPU > 80% for 10 minutes → alert"

Step 1 — Configuring the application to expose metrics

Your Go app uses libraries that register metrics and expose them at GET /metrics on the HTTP port.

In Gin, Prometheus middleware only applies to routes registered after it. Therefore, ensure that the Prometheus middleware is registered before registering any routes to the Gin engine.

What	Library	Example metric
HTTP (Gin)	go-gin-prometheus	`gin_requests_total`, `gin_request_duration_seconds_*`
gRPC server	go-grpc-prometheus	`grpc_server_handled_total`
gRPC client (gateway)	go-grpc-prometheus	`grpc_client_handled_total`
Logs	shared logger	`log_messages_total{log.level="info"}`
Go runtime	built-in	`process_cpu_seconds_total`, `process_`, `go_memstats_`

Registering `gin_*` metrics

What it tracks: Every HTTP request and response that passes through the Gin engine — including /healthz, /v1/* (REST via gRPC-Gateway), and 404/405 responses.

import ginprometheus "github.com/zsais/go-gin-prometheus"

func NewHandler(logger logging.Logger, cfg *config.Config, redisClient *redis.Client, lis net.Listener) http.Handler {
    engine := gin.New()

    once.Do(func() {
        prom = ginprometheus.NewPrometheus("gin")
    })
    if prom != nil {
        prom.Use(engine)
    }

    // Routes must be registered AFTER prom.Use() so middleware applies to them.
    endpointsHealth(engine, redisClient)
    endpointsGRPCGateway(engine, lis)
}

PS: prom.Use(engine) does two things:

Adds middleware that counts every HTTP request (gin_requests_total, gin_request_duration_seconds_*, etc.)
Registers GET /metrics endpoint on the Gin engine so it is accessible over the HTTP port

Metric	Meaning
`gin_requests_total`	HTTP request count, partitioned by status code, method, handler, host, and URL
`gin_request_duration_seconds_*`	HTTP request latency (histogram)
`gin_request_size_bytes`	Approximate request body size
`gin_response_size_bytes`	Response body size

Example /metrics response for Gin metrics:

gin_requests_total{code="200",method="GET",url="/healthz",host="localhost:8080"} 10
gin_requests_total{code="404",method="GET",url="/favicon.ico",host="localhost:8080"} 1

Registering `grpc_server_*` metrics

What it tracks: RPCs handled by the gRPC server on port 50051 (e.g. RPC methods called directly over gRPC, or via the gRPC-Gateway). Metrics are recorded after the RPC completes, so they include information on the response as well.

grpcServer = grpc.NewServer(
    grpc.ChainUnaryInterceptor(
        grpcprometheus.UnaryServerInterceptor,  // record each RPC (timing, status, method)
        authInterceptor.UnaryInterceptor(),
    ),
    grpc.ChainStreamInterceptor(
        grpcprometheus.StreamServerInterceptor,
    ),
)
v1.RegisterLookupServiceServer(grpcServer, lookupServer)
grpcprometheus.Register(grpcServer)  // registers collectors and enables handling-time histograms

Metric	Meaning
`grpc_server_handled_total`	RPC count by service, method, and gRPC status (OK, Unauthenticated, …)
`grpc_server_handling_seconds_*`	RPC latency histogram
`grpc_server_msg_received_total`	Protobuf messages received per RPC
`grpc_server_msg_sent_total`	Protobuf messages sent per RPC

Use case: Monitor RPC calls, latency, auth failures, and errors on the gRPC server.

Registering `grpc_client_*` metrics

What it tracks: gRPC calls made by gRPC-Gateway when REST clients hit the HTTP server. The gateway translates HTTP into an internal gRPC call to send to the local gRPC server. Metrics are recorded after the RPC completes, so they include information on the response as well.

opts := []grpc.DialOption{
    grpc.WithTransportCredentials(insecure.NewCredentials()),
    grpc.WithChainUnaryInterceptor(grpcprometheus.UnaryClientInterceptor),
    grpc.WithChainStreamInterceptor(grpcprometheus.StreamClientInterceptor),
}
// gateway dials local gRPC server using these opts
gw.RegisterLookupServiceHandlerFromEndpoint(ctx, gwmux, lis.Addr().String(), opts)

Metric	Meaning
`grpc_client_started_total`	Outbound RPC attempts started
`grpc_client_handled_total`	Outbound RPC count by service, method, and status
`grpc_client_handling_seconds_*`	Latency of the internal gateway→gRPC hop
`grpc_client_msg_received_total`	Messages received on the client side
`grpc_client_msg_sent_total`	Messages sent on the client side

Use case: Detect slowness or failures in the HTTP→gRPC translation layer.

Registering `log_messages_total` metrics

What it tracks: Total log lines emitted by the app, partitioned by log level.

import "github.com/prometheus/client_golang/prometheus"

var (
    once     sync.Once
    instance logging.Logger
)
func Singleton(cfg Config, writers ...io.Writer) logging.Logger {
    once.Do(func() {
        instance = New(cfg, writers...)
        prometheus.MustRegister(logCounter)
    })
    return instance
}

Metric	Meaning
`log_messages_total{log.level="info"}`	Info-level log lines
`log_messages_total{log.level="error"}`	Error-level log lines
`log_messages_total{log.level="debug"}`	Debug-level log lines

Use case: Alert on error log spikes (e.g. log_messages_total{log.level="error"} > 100).

Go runtime metrics (built-in)

What it tracks: Process health — automatically registered by the Prometheus Go client library.

Metric	Meaning
`process_cpu_seconds_total`	Total CPU time consumed by the process
`process_open_fds`	Number of open file descriptors
`process_resident_memory_bytes`	Resident memory size
`go_memstats_*`	Go heap, GC, and allocation stats
`go_gc_duration_seconds_*`	Garbage collection pause times

Use case: Alert on CPU/memory spikes.

Step 2 — Configuring Prometheus to scrape and store metrics

Once Step 1 is done and GET /metrics is serving data, Prometheus needs to pull those values on a schedule and store them as time series. How you set that up depends on where Prometheus runs:

Locally — you run Prometheus in Docker Compose and point it at your service with prometheus.yml.
In cluster — the shared cluster Prometheus discovers your pod via annotations; no per-service prometheus.yml in the deployment.

There are commonly three types of metrics Prometheus stores:

Counter — only increases (gin_requests_total)
Histogram — latency distribution (gin_request_duration_seconds_bucket)
Gauge — goes up and down (process_open_fds, memory)

Use rate(...[1m]) for "per second" on counters, e.g. rate(gin_requests_total[1m]).

To run Prometheus locally

1. Start your service on a port (e.g. 8082).

2. Create telemetry/prometheus/iam-config-management/prometheus.yml with the following content:

scrape_configs:
  - job_name: 'iam-config-management'       # job name in Prometheus
    scrape_interval: 10s                    # scrape interval defined here
    static_configs:
      - targets: ['host.docker.internal:8082']  # lets the Prometheus container reach a process on your host (WSL/Linux)

rule_files:
  - alerts-cpu.yaml
  - alerts-fds.yaml
  - alerts-logger.yaml
  - alerts-memory.yaml
  - alerts-status.yaml

3. Create telemetry/prometheus/docker-compose.yml as shown below:

services:
  prometheus:
    image: prom/prometheus:v3.0.1
    container_name: prometheus
    ports:
      - "9090:9090"
    volumes:
      - ./iam-config-management:/etc/prometheus   # must contain prometheus.yml from step 2
    extra_hosts:
      # Prometheus runs inside Docker. From inside the container, localhost means the container itself — not your WSL/host where go run is listening.
      # host-gateway is a Docker Compose special value: it automatically uses the host's gateway IP (the IP the container uses to reach the host).
      # This adds an entry to the container's /etc/hosts file that allows the container to resolve host.docker.internal to the host's gateway IP.
      # prometheus.yml says: targets: ['host.docker.internal:8082']. Prometheus resolves host.docker.internal to the host machine.
      - "host.docker.internal:host-gateway"
    command:
      - "--log.level=debug"
      - "--config.file=/etc/prometheus/prometheus.yml"

4. Start Prometheus

cd telemetry/prometheus
docker compose up

5. Open the UI at http://localhost:9090.

6. Verify scraping by navigating to Status → Target health. Find the job called iam-config-management. State should be UP. If DOWN:

Service not running on 8082
Wrong target in prometheus.yml (try 172.17.0.1:8082 or your WSL IP)
Firewall blocking the port

7. Run queries in Prometheus. Go to Graph (or Query in newer UI) and try:

# Total HTTP requests
gin_requests_total

# HTTP requests per second
rate(gin_requests_total[1m])

# gRPC calls handled
grpc_server_handled_total

# Log messages by level
log_messages_total

# CPU usage
rate(process_cpu_seconds_total[1m])

# Memory (MB)
go_memstats_sys_bytes / 1024 / 1024

8. Generate traffic and watch values change. Hammer the /healthz endpoint:

for i in $(seq 1 20); do curl -s http://localhost:8082/healthz > /dev/null; done

In Prometheus, re-run gin_requests_total or rate(gin_requests_total[1m]) — numbers should move.

9. Stop everything

cd telemetry/prometheus
docker compose down

You have also defined alerts in prometheus.yml. In the Prometheus UI, you'll see Alert rules and whether they're firing. Locally they usually stay green unless you stress the service.

To run Prometheus in cluster

In the cluster, the shared cluster Prometheus discovers targets automatically from pod annotations — there is no per-service prometheus.yml in the deployment.

Helm stamps the following annotations on each pod. They are required for in-cluster Prometheus to find and scrape your service:

metadata:
  annotations:
    prometheus.io/scrape: "true"
    prometheus.io/path: /metrics
    prometheus.io/port: "8082"

Annotation	Meaning
`prometheus.io/scrape`	Tells the cluster Prometheus to scrape this pod
`prometheus.io/path`	Metrics endpoint path exposed by the Gin HTTP server
`prometheus.io/port`	HTTP port where `/metrics` is served (not the gRPC port)

The cluster Prometheus operator reads these annotations, builds scrape targets, and pulls metrics on its own schedule — the same counters and histograms you verified locally, now stored with labels like app and cluster.

Note on Grafana when running locally

The cluster Prometheus will scrape metrics together with labels like app and cluster, which enables the Grafana dashboard to display panels based on these labels. As local Prometheus will scrape without those labels, panels in the local Grafana dashboard may look empty. Therefore, use Prometheus Graph for local debugging; use Grafana mainly against cluster Prometheus.