Prometheus Important Configuration

  1. Per-pod Prometheus Annotations

Annotations on pods allow a fine control of the scraping process: (setting up the annotation so that this service will be discovered by Prometheus as a target to be scraped.)

  1. [x] prometheus.io/scrape: The default configuration will scrape all pods (采集所有的pods) and, if set tofalse, this annotation will exclude the pod from the scraping process. 自动采集指标数据开关，默认为false；Prometheus会在k8s集群中自动检测哪些svc是打开了这个开关
  2. [x] prometheus.io/path: If the metrics path is not/metrics, define it with this annotation. 采集指标数据路径，默认为 /metrics

  3. [x] prometheus.io/port: Scrape the pod on the indicated port instead of the pod’s declared ports (default is a port-free target if none are declared). 采集指标端口，默认为svc暴露的端口

  4. <relabel_config>标签重写

    Relabeling is a powerful tool to dynamically rewrite the label set of a target before it gets scraped --- 重新标记是强大的工具，可以在采集目标之前动态重写目标的标签集 match against the configured regular expression -- 与配置的正则表达式匹配
    Regex capture groups are available --- 正则表达式捕获组可用。

Action:

  • replace(Default action): Match regex against the concatenated source_labels. Then, set target_label to replacement, with match group references (${1}, ${2}, ...) in replacement substituted by their value. If regex does not match, no replacement takes place --- replace行为会根据regex的配置匹配source_labels标签的值（多个source_label的值会按照separator进行拼接），并且将匹配到的值写入到target_label当中，如果有多个匹配组，则可以使用${1}, ${2}确定写入的内容。如果没匹配到任何内容则不对target_label进行重新。repalce操作允许用户根据Target的Metadata标签重写或者写入新的标签键值对，在多环境的场景下，可以帮助用户添加与环境相关的特征维度，从而可以更好的对数据进行聚合。

    replace 动作流程
    1 从 source_labels 中获取需要操作的标签列表，并获取他们的值，这些值是一个列表，使用 separator 分割
    2 regex 中的正则表达式会对 值 列表的值进行过滤，为了方便需要使用正则分组
    3 使用分组就会通过从分组获取的匹配字符串 赋值给 $1 等，如果仅仅有一个分组的话，字段 replacement 的默认值为 $1
    4 新的 target_label 的内容会被 replacement 的内容替换掉，也就是说 target_label: replacement

  • keep: Drop targets for whichregexdoes not match the concatenatedsource_labels --- 删除regex与连接的源标签不匹配的目标

  • drop: Drop targets for which regex matches the concatenated source_labels --- 删除regex与连接的源标签匹配的目标。

  • labelmap: Match regex against all label names. Then copy the values of the matching labels to label names given by replacement with match group references (${1}, ${2}, ...) in replacement substituted by their value --- 将regex与所有标签名匹配。然后将匹配标签的值复制到由匹配组引用（${1}，${2}，…）替换而成的标签名，替换为它们的值。 labelmap会根据regex的定义去匹配Target实例所有标签的名称，并且以匹配到的内容为新的标签名称，其值作为新标签的值。

  • hashmod:Set target_label to the modulus of a hash of the concatenated source_labels.

  • labeldrop: Matchregexagainst all label names. Any label that matches will be removed from the set of labels.

  • labelkeep: Matchregexagainst all label names. Any label that does not match will be removed from the set of labels.

replacement: (replacement 字段的作用就是给 target_label 赋值)

Replacement value against which a regex replace is performed if the regular expression matches --- 如果正则表达式匹配，则执行正则表达式替换的替换值

标签重写例子：
例如，在监控Kubernetes下所有的主机节点时，为将这些节点上定义的标签写入到样本中时，可以使用如下relabel_config配置：

- job_name: 'kubernetes-nodes'
  kubernetes_sd_configs:
  - role: node
  relabel_configs:
  - action: labelmap
    regex: __meta_kubernetes_node_label_(.+)

  1. How labels propagate can be a bit tricky to get your head around initially --- 标签是如何传播的，最初理解这一点有点棘手

  2. So as a simple rule of thumb:relabel_confighappens before the scrape, metric_relabel_configshappens after the scrape. And if one doesn't work you can always try the other!

  3. Regular expressions in relabelling are fully anchored, meaning that
    the pattern infra will not match fooinfra or infrabar

  4. relabelconfigs occurs when figuring out what to scrape, 'metrics-relabel-configs' happens after the scrape has occurred. Metric relabelling gives you access to the time series after it is scraped but before it is written to storage.

References

  1. Google Regular Expression re2
  2. Regex Tester - Golang
  3. Prometheus relabeling tricks

  4. Life of Label (Great)

  5. ff

results matching ""

    No results matching ""