Training Example: Prometheus – Review the Data, Give Your Score & Compare to the Real AI Evaluation

[H1] Open source metrics and monitoring for your systems and services
Monitor your applications, systems, and services with the leading open source monitoring solution. Instrument, collect, store, and query your metrics for alerting, dashboarding, and other use cases.Get startedDownloadGet startedDownloadDimensional data modelPrometheus models time series in a flexible dimensional data model. Time series are identified by a metric name and a set of key-value pairs.Powerful queriesThe PromQL query language allows you to query, correlate, and transform your time series data in powerful ways for visualizations, alerts, and more.Precise alertingAlerting rules are based on PromQL and make full use of the dimensional data model. A separate Alertmanager component handles notifications and silencing.Simple operationPrometheus servers operate independently and only rely on local storage. Developed in Go, the statically linked binaries are easy to deploy across various environments.Instrumentation librariesPrometheus provides a large number of official and community-contributed metrics instrumentation libraries that cover most major languages.Ubiquitous integrationsPrometheus comes with hundreds of official and community-contributed integrations that allow you to easily extract metrics from existing systems.Modern monitoring
[H2] Monitoring for the cloud native world
Designed for the cloud native world, Prometheus integrates with Kubernetes and other cloud and container managers to continuously discover and monitor your services. It is the second project to graduate from the CNCF after Kubernetes.Even though Borgmon remains internal to Google, the idea of treating time-series data as a data source for generating alerts is now accessible to everyone through those open source tools like Prometheus [...]Site Reliability Engineering: How Google Runs Production Systems (O'Reilly Media)
[H2]
[IMG: GitHub logo]
Open Source
Prometheus is 100% open source and community-driven. All components are available under the Apache 2 License on GitHub.Star us on GitHubLoading stars...
[H2] Open Governance
Prometheus is a Cloud Native Computing Foundation graduated project.
[IMG: CNCF logo]
[IMG: CNCF logo]

[H1] Download
We provide precompiled binaries and Docker images for most officially maintained Prometheus components. If a component is not listed here, check the respective repository on Github for further instructions.There is also a constantly growing number of independently maintained exporters listed at Exporters and integrations.On this pageOperating SystemArchitecture
[H2] prometheus
The Prometheus monitoring system and time series database. prometheus/prometheus3.12.0-rc.0 / 2026-05-19Pre-releaseRelease notesFile nameOSArchSizeSHA256 Checksumprometheus-3.12.0-rc.0.darwin-amd64.tar.gzdarwinamd64159.97 MiB4c15776a9d2570d1c5867cc574fecbfbb57e1a0812e9077ec2c189ee00d9c20cprometheus-3.12.0-rc.0.darwin-arm64.tar.gzdarwinarm64151.62 MiB2cb359c95d39781f48b4db64517ccc3d8d619d44e29db1d074a28ae6ef9e9ee2prometheus-3.12.0-rc.0.linux-amd64.tar.gzlinuxamd64145.37 MiB01738b37cba198aea5033a6ee01ba46b6c6ad90e79b5f34613af059eb2d27f15prometheus-3.12.0-rc.0.windows-amd64.zipwindowsamd64149.67 MiBe23a3a8fabe4dee4ebe1bfa10f44b9ee57ba19873753ed9bff2e183e0b4917de3.11.3 / 2026-04-27LatestRelease notesFile nameOSArchSizeSHA256 Checksumprometheus-3.11.3.darwin-amd64.tar.gzdarwinamd64157.53 MiBebd86b50e816e3752edff25202593fc31e08771f14c5f6d4a24737f0304385beprometheus-3.11.3.darwin-arm64.tar.gzdarwinarm64149.54 MiB742773c5b3958eec5e6b58802f25cf77b47a319219ce0d508ed2f657c61d8859prometheus-3.11.3.linux-amd64.tar.gzlinuxamd64143.21 MiB9479af67673316278958cda1f39b88a09f8921084e039c65acca060d0447bb38prometheus-3.11.3.windows-amd64.zipwindowsamd64147.61 MiB3bb74babbb3d13af6a19c83fa2ad1bc92d1a34d6975ad15b2d79fbd1706c350f3.5.3 / 2026-04-27LTSRelease notesFile nameOSArchSizeSHA256 Checksumprometheus-3.5.3.darwin-amd64.tar.gzdarwinamd64117.68 MiB408eec9f1138ad5d30509038b2e8ae798ed2910e7faaa0e7f61ca22db222aaf5prometheus-3.5.3.darwin-arm64.tar.gzdarwinarm64112.18 MiB1883df59fbea254b2e3f112feb6406533be7c062aef20f5d0b40b9e9acdb77e2prometheus-3.5.3.linux-amd64.tar.gzlinuxamd64110.14 MiB8c30b9d99664e39b0363c0ba54fab30a7958e9d3de27246bf26ed85e6cfb8946prometheus-3.5.3.windows-amd64.zipwindowsamd64113.75 MiB6ebe148731493138e9a020766487c5e16f0591e35006aeeb561197078948b32d
[H2] alertmanager
Prometheus Alertmanager prometheus/alertmanager0.32.1 / 2026-04-29LatestRelease notesFile nameOSArchSizeSHA256 Checksumalertmanager-0.32.1.darwin-amd64.tar.gzdarwinamd6434.16 MiB4ace51175fda599e1d51862051f517ff70884a0537b36e79524ed46a194c3c06alertmanager-0.32.1.darwin-arm64.tar.gzdarwinarm6432.17 MiB4fd3d3d6156cbe76016583b852dc80beda7ee693d28ac27477a9b3da06d87a93alertmanager-0.32.1.linux-amd64.tar.gzlinuxamd6433.12 MiBe3ba4a543111dd4bbf436838385cbf88108f0b128a723e61fe97d3569294ad4dalertmanager-0.32.1.windows-amd64.zipwindowsamd6433.86 MiB04191f62a87a944371abda919eedf2e36066ad544aa5d09c22620343e3046873
[H2] blackbox_exporter
Blackbox prober exporter prometheus/blackbox_exporter0.28.0 / 2025-12-04LatestRelease notesFile nameOSArchSizeSHA256 Checksumblackbox_exporter-0.28.0.darwin-amd64.tar.gzdarwinamd6415.91 MiB12d7a3010235862d073bbb111b997870a50070bcda3b912bca8f0095cfda23c6blackbox_exporter-0.28.0.darwin-arm64.tar.gzdarwinarm6414.98 MiBec6c70ccca92e209dd22be76a4fa244f4bd31afdae3ddb2bb082144100ec52bbblackbox_exporter-0.28.0.linux-amd64.tar.gzlinuxamd6415.40 MiBcaf5d242fb1cf6d5cb678f3f799f22703d4fafea26b03dcbbd7e1f1825e06329blackbox_exporter-0.28.0.windows-amd64.zipwindowsamd6415.62 MiBf79af4599679d05a4976e78596c8bfed6a088eeef3baced100128921fdee3ac5
[H2] consul_exporter
Exporter for Consul metrics prometheus/consul_exporter0.13.0 / 2024-11-06LatestRelease notesFile nameOSArchSizeSHA256 Checksumconsul_exporter-0.13.0.darwin-amd64.tar.gzdarwinamd649.44 MiBc60739251dc50cbc9bf3fdeeb9e91a46abd50653d7a5df9091836ce02d7f91e0consul_exporter-0.13.0.darwin-arm64.tar.gzdarwinarm648.99 MiB7b6d68a2a2222489416b3f1c9c2219956bdded7bdc456808c33c6997854b6920consul_exporter-0.13.0.linux-amd64.tar.gzlinuxamd649.59 MiB2a8da4147330c6e19c9665deca1c419d507e100de6c8b7c58c0715ff25453773consul_exporter-0.13.0.windows-amd64.zipwindowsamd649.76 MiB4c378b2827ba2631b4be6551784deef089527f6d6882dd1bc2976d5f58a3d614
[H2] graphite_exporter
Server that accepts metrics via the Graphite protocol and exports them as Prometheus metrics prometheus/graphite_exporter0.16.0 / 2024-10-29LatestRelease notesFile nameOSArchSizeSHA256 Checksumgraphite_exporter-0.16.0.darwin-amd64.tar.gzdarwinamd6420.93 MiBb9c1eff6bd5dcb61d4106ab61fd29518561a7b63d5b8c71000f8ad09baaf6e28graphite_exporter-0.16.0.darwin-arm64.tar.gzdarwinarm6419.92 MiBacafc449dcdccc795fc3dff8d8832fa90bd6930f5c6a4a2305f5e8ee419f7ffagraphite_exporter-0.16.0.linux-amd64.tar.gzlinuxamd6421.21 MiB129acf14bb62dc32596ff8aed40a526a66260b2736801bc546be407436dd32d8graphite_exporter-0.16.0.windows-amd64.zipwindowsamd6412.09 MiB9a762d1a7a19ed8e369cb590c7faf7555e7290925a919b0b056ef272c23cd254
[H2] memcached_exporter
Exports metrics from memcached servers for consumption by Prometheus. prometheus/memcached_exporter0.16.0 / 2026-04-08LatestRelease notesFile nameOSArchSizeSHA256 Checksummemcached_exporter-0.16.0.darwin-amd64.tar.gzdarwinamd648.93 MiBa3adb18e9fb492b6018d7f150d55ed2673b82aa7a4b2f57365a54e69fd953bc2memcached_exporter-0.16.0.darwin-arm64.tar.gzdarwinarm648.37 MiB7a46ec5dab6b3db95f3e952fa3bd8a8d786065f69b2a06782191b23f410c278fmemcached_exporter-0.16.0.linux-amd64.tar.gzlinuxamd648.81 MiBec669cdce5258e48e0b3747719bb60c0f91d74a21fd4033ec259e80db6c0b0edmemcached_exporter-0.16.0.windows-amd64.zipwindowsamd648.93 MiB6fba79c3996e03896e08101aca4cacacc41aeb0df054d66b43ceccc38ece8ea2
[H2] mysqld_exporter
Exporter for MySQL server metrics prometheus/mysqld_exporter0.19.0 / 2026-03-18LatestRelease notesFile nameOSArchSizeSHA256 Checksummysqld_exporter-0.19.0.darwin-amd64.tar.gzdarwinamd649.57 MiB423bc6774589e172fd16bb0a0424652a57245adeddf47a72fe3cd847b5a02617mysqld_exporter-0.19.0.darwin-arm64.tar.gzdarwinarm648.96 MiB67932adc812b0dd811b01c6fe450b36b29f2c5e91497b6aa577008a9ff1c857amysqld_exporter-0.19.0.linux-amd64.tar.gzlinuxamd649.42 MiB97238be558bd1a6aa6b9a927fa21d91dc5cabe6b9e00678b5cafa2bbb3899e72mysqld_exporter-0.19.0.windows-amd64.zipwindowsamd649.58 MiBa90b724476c6f1abd7f4f1bd904f54242ea7d650c1e95acbab820046963c9785
[H2] node_exporter
Exporter for machine metrics prometheus/node_exporter1.11.1 / 2026-04-07LatestRelease notesFile nameOSArchSizeSHA256 Checksumnode_exporter-1.11.1.darwin-amd64.tar.gzdarwinamd645.34 MiB782318ceb48cb5501271a666d1b015a9406c02cd45dbc9513deca005b91e03a5node_exporter-1.11.1.darwin-arm64.tar.gzdarwinarm644.94 MiBe987428618362c2d2540a68b722bd982ef1486c9961631298f20ea8fd57d3be4node_exporter-1.11.1.linux-amd64.tar.gzlinuxamd6411.52 MiB9f5ea48e5bc7b656f8a91a32e7d7deb89f70f73dabd0d974418aca15f37d6810
[H2] promlens
PromLens – The query builder, analyzer, and explainer for PromQL prometheus/promlens0.3.0 / 2022-12-05LatestRelease notesFile nameOSArchSizeSHA256 Checksumpromlens-0.3.0.darwin-amd64.tar.gzdarwinamd6416.90 MiBa9dfe174d57e6ecfb456db2a7161de3d49f72ce8cdc312855497c89f442d02cbpromlens-0.3.0.darwin-arm64.tar.gzdarwinarm6416.25 MiBbff5696e41d11dab37edacb941c70cc879a9e22a484403685bf1c53e2108d29apromlens-0.3.0.linux-amd64.tar.gzlinuxamd6417.15 MiB8fdcc621cf559b7e55c0e3cf334b8662ae8f53cf999cdf5d7d303d2841f62ef0promlens-0.3.0.windows-amd64.zipwindowsamd6417.69 MiBdaf50de8f0660324dc208105e5b89a2974fdabb037987ed95d963fea94501933
[H2] pushgateway
Push acceptor for ephemeral and batch jobs. prometheus/pushgateway1.11.2 / 2025-10-30LatestRelease notesFile nameOSArchSizeSHA256 Checksumpushgateway-1.11.2.darwin-amd64.tar.gzdarwinamd6411.19 MiB87b91fc5cb53c8afdae7eb58d9931784c93b9a225522cbdedff38b59e01d5b31pushgateway-1.11.2.darwin-arm64.tar.gzdarwinarm6410.61 MiB0387c770a4b33ae1cc189ac34489b2105d378099ae65f9872df0ea49829c172fpushgateway-1.11.2.linux-amd64.tar.gzlinuxamd6411.04 MiB2ec72315e150dda071fdeef09360780a386a67e5207ebaa53bb18f2f1a3b89cfpushgateway-1.11.2.windows-amd64.zipwindowsamd6411.18 MiBbf62b67e13d7257d75d2e2eb9090a92562296f6631ba0284d82161d1659a9248
[H2] statsd_exporter
StatsD to Prometheus metrics exporter prometheus/statsd_exporter0.29.0 / 2025-03-03LatestRelease notesFile nameOSArchSizeSHA256 Checksumstatsd_exporter-0.29.0.darwin-amd64.tar.gzdarwinamd649.28 MiB71d8ac007c9145261fdc2cd35d055fb08e53235ef30c481c3e64f043d1554d94statsd_exporter-0.29.0.darwin-arm64.tar.gzdarwinarm648.69 MiBb429a8f570f7b171fa5251240460c3b43e39b08da95ca2dfed767e08059dee77statsd_exporter-0.29.0.linux-amd64.tar.gzlinuxamd649.03 MiB46e39d834247fcd6219b9076a1c1065973859b70f7316b5159f158ce2da9c2b6statsd_exporter-0.29.0.windows-amd64.zipwindowsamd649.28 MiB23b8942a21242f7923e2d37c65d338cba979f402a6cca6fa39e359f0b0423cea

Show nav
[H1] Getting started
This guide is a "Hello World"-style tutorial which shows how to install,
configure, and use a simple Prometheus instance. You will download and run
Prometheus locally, configure it to scrape itself and an example application,
then work with queries, rules, and graphs to use collected time
series data.
[H2] Downloading and running Prometheus
Download the latest release of Prometheus for
your platform, then extract and run it:
tar xvfz prometheus-*.tar.gz
cd prometheus-*
Before starting Prometheus, let's configure it.
[H2] Configuring Prometheus to monitor itself
Prometheus collects metrics from targets by scraping metrics HTTP
endpoints. Since Prometheus exposes data in the same
manner about itself, it can also scrape and monitor its own health.
While a Prometheus server that collects only data about itself is not very
useful, it is a good starting example. Save the following basic
Prometheus configuration as a file named prometheus.yml:
global:
scrape_interval: 15s # By default, scrape targets every 15 seconds.
# Attach these labels to any time series or alerts when communicating with
# external systems (federation, remote storage, Alertmanager).
external_labels:
monitor: 'codelab-monitor'
# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
# The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
- job_name: 'prometheus'
# Override the global default and scrape targets from this job every 5 seconds.
scrape_interval: 5s
static_configs:
- targets: ['localhost:9090']
For a complete specification of configuration options, see the
configuration documentation.
[H2] Starting Prometheus
To start Prometheus with your newly created configuration file, change to the
directory containing the Prometheus binary and run:
# Start Prometheus.
# By default, Prometheus stores its database in ./data (flag --storage.tsdb.path).
./prometheus --config.file=prometheus.yml
Prometheus should start up. You should also be able to browse to a status page
about itself at localhost:9090 . Give it a couple of
seconds to collect data about itself from its own HTTP metrics endpoint.
You can also verify that Prometheus is serving metrics about itself by
navigating to its metrics endpoint:
localhost:9090/metrics
[H2] Using the expression browser
Let us explore data that Prometheus has collected about itself. To
use Prometheus's built-in expression browser, navigate to
http://localhost:9090/query  and choose the "Graph" tab.
As you can gather from localhost:9090/metrics ,
one metric that Prometheus exports about itself is named
prometheus_target_interval_length_seconds (the actual amount of time between
target scrapes). Enter the below into the expression console and then click "Execute":
prometheus_target_interval_length_seconds
This should return a number of different time series (along with the latest value
recorded for each), each with the metric name
prometheus_target_interval_length_seconds, but with different labels. These
labels designate different latency percentiles and target group intervals.
If we are interested only in 99th percentile latencies, we could use this
query:
prometheus_target_interval_length_seconds{quantile="0.99"}
To count the number of returned time series, you could write:
count(prometheus_target_interval_length_seconds)
For more about the expression language, see the
expression language documentation.
[H2] Using the graphing interface
To graph expressions, navigate to http://localhost:9090/query  and use the "Graph"
tab.
For example, enter the following expression to graph the per-second rate of chunks
being created in the self-scraped Prometheus:
rate(prometheus_tsdb_head_chunks_created_total[1m])
Experiment with the graph range parameters and other settings.
[H2] Starting up some sample targets
Let's add additional targets for Prometheus to scrape.
The Node Exporter is used as an example target, for more information on using it
see these instructions.
tar -xzvf node_exporter-*.*.tar.gz
cd node_exporter-*.*
# Start 3 example targets in separate terminals:
./node_exporter --web.listen-address 127.0.0.1:8080
./node_exporter --web.listen-address 127.0.0.1:8081
./node_exporter --web.listen-address 127.0.0.1:8082
You should now have example targets listening on http://localhost:8080/metrics ,
http://localhost:8081/metrics , and http://localhost:8082/metrics .
[H2] Configure Prometheus to monitor the sample targets
Now we will configure Prometheus to scrape these new targets. Let's group all
three endpoints into one job called node. We will imagine that the
first two endpoints are production targets, while the third one represents a
canary instance. To model this in Prometheus, we can add several groups of
endpoints to a single job, adding extra labels to each group of targets. In
this example, we will add the group="production" label to the first group of
targets, while adding group="canary" to the second.
To achieve this, add the following job definition to the scrape_configs
section in your prometheus.yml and restart your Prometheus instance:
scrape_configs:
- job_name: 'node'
# Override the global default and scrape targets from this job every 5 seconds.
scrape_interval: 5s
static_configs:
- targets: ['localhost:8080', 'localhost:8081']
labels:
group: 'production'
- targets: ['localhost:8082']
labels:
group: 'canary'
Go to the expression browser and verify that Prometheus now has information
about time series that these example endpoints expose, such as node_cpu_seconds_total.
[H2] Configure rules for aggregating scraped data into new time series
Though not a problem in our example, queries that aggregate over thousands of
time series can get slow when computed ad-hoc. To make this more efficient,
Prometheus can prerecord expressions into new persisted
time series via configured recording rules. Let's say we are interested in
recording the per-second rate of cpu time (node_cpu_seconds_total) averaged
over all cpus per instance (but preserving the job, instance and mode
dimensions) as measured over a window of 5 minutes. We could write this as:
avg by (job, instance, mode) (rate(node_cpu_seconds_total[5m]))
Try graphing this expression.
To record the time series resulting from this expression into a new metric
called job_instance_mode:node_cpu_seconds:avg_rate5m, create a file
with the following recording rule and save it as prometheus.rules.yml:
groups:
- name: cpu-node
rules:
- record: job_instance_mode:node_cpu_seconds:avg_rate5m
expr: avg by (job, instance, mode) (rate(node_cpu_seconds_total[5m]))
To make Prometheus pick up this new rule, add a rule_files statement in your prometheus.yml. The config should now
look like this:
global:
scrape_interval: 15s # By default, scrape targets every 15 seconds.
evaluation_interval: 15s # Evaluate rules every 15 seconds.
# Attach these extra labels to all timeseries collected by this Prometheus instance.
external_labels:
monitor: 'codelab-monitor'
rule_files:
- 'prometheus.rules.yml'
scrape_configs:
- job_name: 'prometheus'
# Override the global default and scrape targets from this job every 5 seconds.
scrape_interval: 5s
static_configs:
- targets: ['localhost:9090']
- job_name: 'node'
# Override the global default and scrape targets from this job every 5 seconds.
scrape_interval: 5s
static_configs:
- targets: ['localhost:8080', 'localhost:8081']
labels:
group: 'production'
- targets: ['localhost:8082']
labels:
group: 'canary'
Restart Prometheus with the new configuration and verify that a new time series
with the metric name job_instance_mode:node_cpu_seconds:avg_rate5m
is now available by querying it through the expression browser or graphing it.
[H2] Reloading configuration
As mentioned in the configuration documentation a
Prometheus instance can have its configuration reloaded without restarting the
process by using the SIGHUP signal. If you're running on Linux this can be
performed by using kill -s SIGHUP <PID>, replacing <PID> with your Prometheus
process ID.
[H2] Shutting down your instance gracefully.
While Prometheus does have recovery mechanisms in the case that there is an
abrupt process failure it is recommended to use signals or interrupts for a
clean shutdown of a Prometheus instance. On Linux, this can be done by sending
the SIGTERM or SIGINT signals to the Prometheus process. For example, you
can use kill -s <SIGNAL> <PID>, replacing <SIGNAL> with the signal name
and <PID> with the Prometheus process ID. Alternatively, you can press the
interrupt character at the controlling terminal, which by default is ^C (Control-C).On this page

Show nav
[H1] Data model
Prometheus fundamentally stores all data as time
series: streams of timestamped
values belonging to the same metric and the same set of labeled dimensions.
Besides stored time series, Prometheus may generate temporary derived time series
as the result of queries.
[H2] Metric names and labels
Every time series is uniquely identified by its metric name and optional key-value pairs called labels.
Metric names:
Metric names SHOULD specify the general feature of a system that is measured (e.g. http_requests_total - the total number of HTTP requests received).
Metric names MAY use any UTF-8 characters.
Metric names SHOULD match the regex [a-zA-Z_:][a-zA-Z0-9_:]* for the best experience and compatibility (see the warning below). Metric names outside of that set will require quoting e.g. when used in PromQL (see the UTF-8 guide).
NOTE: Colons (':') are reserved for user-defined recording rules. They SHOULD NOT be used by exporters or direct instrumentation.
Metric labels:
Labels let you capture different instances of the same metric name. For example: all HTTP requests that used the method POST to the /api/tracks handler. We refer to this as Prometheus's "dimensional data model". The query language allows filtering and aggregation based on these dimensions. The change of any label's value, including adding or removing labels, will create a new time series.
Label names MAY use any UTF-8 characters.
Label names beginning with __ (two underscores) MUST be reserved for internal Prometheus use.
Label names SHOULD match the regex [a-zA-Z_][a-zA-Z0-9_]* for the best experience and compatibility (see the warning below). Label names outside of that regex will require quoting e.g. when used in PromQL (see the UTF-8 guide).
Label values MAY contain any UTF-8 characters.
Labels with an empty label value are considered equivalent to labels that do not exist.
WARNING: The UTF-8 support for metric and label names was added relatively recently in Prometheus v3.0.0. It might take time for the wider ecosystem (downstream PromQL compatible projects and vendors, tooling, third-party instrumentation, collectors, etc.) to adopt new quoting mechanisms, relaxed validation etc. For the best compatibility it's recommended to stick to the recommended ("SHOULD") character set.
INFO: See also the best practices for naming metrics and labels.
[H2] Samples
Samples form the actual time series data. Each sample consists of:
a float64 or native histogram value
a millisecond-precision timestamp
[H2] Notation
Given a metric name and a set of labels, time series are frequently identified
using this notation:
<metric name>{<label name>="<label value>", ...}
For example, a time series with the metric name api_http_requests_total and
the labels method="POST" and handler="/messages" could be written like
this:
api_http_requests_total{method="POST", handler="/messages"}
This is the same notation that OpenTSDB  uses.
Names with UTF-8 characters outside the recommended set must be quoted, using
this notation:
{"<metric name>", <label name>="<label value>", ...}
Since metric name are internally represented as a label pair
with a special label name (__name__="<metric name>") one could also use the following notation:
{__name__="<metric name>", <label name>="<label value>", ...}
On this page