Learn
WebsiteLoginFree Trial
  • Incident Management
    • What is Incident Management? Beginner's Guide
    • Severity Levels
    • How to calculate MTTR and Other Common Incident Recovery Metrics
    • On-Call
    • SLA vs SLO vs SLI: What's The Difference?
    • Data Aggregation and Aggregators
  • DevOps
    • Best DevOps Tools for Each Phase of the DevOps Lifecycle
      • Best DevOps Planning Tools
      • Best DevOps Coding Tools
      • Best DevOps Build Tools
      • Best DevOps Testing Tools
      • Best DevOps Release Tools
      • Best DevOps Deployment Tools
      • Best DevOps Operations Tools
      • Best DevOps Monitroing Tools
    • What is DevOps?
      • Best CI/CD Tools
      • DevOps Infrastructure and Automation
      • What is a DevOps Pipeline?
      • DevOps Vs. Agile
      • Top 25 DevOps Interview Questions
      • What Are the Benefits of DevOps?
      • What is CI/CD?
      • What is a DevOps Engineer?
      • What is DevSecOps?
    • What is Observability?
      • USE and RED Method
    • What is Site Reliability Engineering (SRE)?
      • Four Golden Signals: SRE Monitoring
      • What is A Canary Deployment?
      • What is Blue-Green Deployment?
  • Docker
    • Overview
    • Dockerfile
    • Images
    • Containers
    • Storage
    • Network
    • Compose
    • Swarm
    • Resources
  • prometheus
    • Overview
    • Data Model
    • Metric Types
    • PromQL
      • Series Selection
      • Counter Rates & Increases
    • Pushgateway
    • Alertmanager
    • Remote Storage
Powered by GitBook
On this page
  • Why Remote Storage?
  • Architecture
  • Configuration
  • Remote Write
  • Remote Read
  • FAQ
  • What are some remote storage options?
  • What protocol does Prometheus use for remote storage?

Was this helpful?

  1. prometheus

Remote Storage

Integrating Prometheus with remote storage systems allows you to achieve better scalability, durability, outsourcing of storage operations, or additional use cases for your Prometheus metrics.

PreviousAlertmanager

Last updated 1 year ago

Was this helpful?

Remote write allows servers to push to remote storage systems or other systems that can accept Prometheus's data format. This feature is particularly useful for long-term storage, data aggregation, or integrating Prometheus with other monitoring or analytics systems.

Why Remote Storage?

  1. Scalability - By default, Prometheus stores all data on the local filesystem and thus is limited to the size of the single node's filesystem. Remote storage is especially helpful in scenarios with a large volume of metrics or if you need to retain data for a long period of time.

  2. Long-Term Retention - Remote storage solutions are typically designed to handle long-term retention of metrics data efficiently. By using remote storage, you can retain historical metrics data for weeks, months, or even years without impacting the performance of your Prometheus server.

  3. Cost Effectiveness - Some remote storage solutions, such as cloud-based object storage services like or , offer cost-effective storage options. Storing metrics data in these services can be more economical than storing it on local disks or in-memory storage solutions.

  4. High Availability - Remote storage solutions often provide built-in redundancy and high availability features, ensuring your metrics data is resilient to hardware failures or other issues. This improves the reliability of your monitoring infrastructure.

  5. Features - Prometheus is focused on storing its own but doesn't offer a solution for processing data in other formats or not in time series form. Remote storage solutions may offer advanced querying, aggregation, rate limiting, and analysis capabilities that go beyond what Prometheus itself provides.

  6. Multi-Tenant Support - If you're running a multi-tenant environment or providing monitoring services to multiple teams or customers, remote storage solutions can offer features for isolating and managing metrics data for different users or tenants.

Rather than trying to solve all these limitations, Prometheus allows you to integrate with third-party remote storage solutions that can compete to offer the best tradeoffs around storage features, scalability, redundancy, and cost.

Architecture

Configuration

You can configure Prometheus to use remote_write and remote_read at the top-level configuration sections of the Prometheus configuration file.

Remote Write

Additionally, you can configure many additional parameters, such as authentication, relabeling, parallelism, and batch sizes.

prometheus.yml
remote_write:
  - url: "http://localhost:9009/api/prom/push"
    # Only keep metric names starting with "demo_".
    write_relabel_configs:
    - action: keep
      source_labels: [__name__]
      regex: 'demo_.*'

Remote Read

Like the remote write, you can configure additional parameters for filtering and authentication.

prometheus.yml
remote_read:
  - url: "http://localhost:9009/api/prom/api/v1/read"
    # Only query this endpoint for selectors containing the matcher job="demo".
    required_matchers:
      job: "demo"

FAQ

What are some remote storage options?

Remote storage options include and are not limited to the following:

What protocol does Prometheus use for remote storage?

Prometheus allows you to send all, or a subset, of the sampled data to a remote HTTP endpoint using a "remote write" protocol defined by Prometheus. Prometheus only buffers sampled data for a few seconds before forwarding it, so transfer to third-party systems is .

When Prometheus performs a , it uses an adapter to send time series data in a format the third-party storage can understand. The data can then be read back (for use with ) later using the feature.

The section allows for the definition of one or more URLs to which Prometheus should send sample data.

The section allows defining one or more URLs from which to read samples when executing PromQL.

Remote write and read protocols are based on the and sent over HTTP. Requests and responses are to reduce over-the-wire size.

near real-time (NRT)
remote_write
remote_read
Cortex
Thanos
Mimir
S3
VictoriaMetrics
protocol buffer ("protobuf")
snappy encoded
Prometheus
Amazon S3
Google Cloud Storage
labeled data model
remote write
remote read
time series data
Prometheus Remote Storage Architecture
Prometheus Remote Storage Architecture
PromQL