Skip to content

node_exporter_metrics diskstats reports incorrect, large float values #7583

Closed as not planned
#12255
Closed as not planned
node_exporter_metrics diskstats reports incorrect, large float values#7583
#12255
Labels
Stalestatus: waiting-for-triage

Description

@spthm
spthm
opened on Jun 17, 2023

Bug Report

Describe the bug
The diskstats metric is reporting incorrect very large floating-point values for most metrics.

To Reproduce
Using the below config,

$ cat $PWD/fluent-bit.conf
[SERVICE]
    flush        1
    daemon       Off
    log_level    info
    http_server  Off

[INPUT]
    name            node_exporter_metrics
    tag             metric.node_exporter
    metrics         diskstats
    scrape_interval 10
    mem_buf_limit   10MB

[OUTPUT]
    name stdout
    match *

I see,

$ /opt/fluent-bit/bin/fluent-bit -c $PWD/fluent-bit.conf
Fluent Bit v3.1.9
* Copyright (C) 2015-2024 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

______ _                  _    ______ _ _           _____  __  
|  ___| |                | |   | ___ (_) |         |____ |/  | 
| |_  | |_   _  ___ _ __ | |_  | |_/ /_| |_  __   __   / /`| | 
|  _| | | | | |/ _ \ '_ \| __| | ___ \ | __| \ \ / /   \ \ | | 
| |   | | |_| |  __/ | | | |_  | |_/ / | |_   \ V /.___/ /_| |_
\_|   |_|\__,_|\___|_| |_|\__| \____/|_|\__|   \_/ \____(_)___/

[2024/10/18 12:00:32] [ info] [fluent bit] version=3.1.9, commit=, pid=28188
[2024/10/18 12:00:32] [ info] [storage] ver=1.5.2, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2024/10/18 12:00:32] [ info] [cmetrics] version=0.9.6
[2024/10/18 12:00:32] [ info] [ctraces ] version=0.5.6
[2024/10/18 12:00:32] [ info] [input:node_exporter_metrics:node_exporter_metrics.0] initializing
[2024/10/18 12:00:32] [ info] [input:node_exporter_metrics:node_exporter_metrics.0] storage_strategy='memory' (memory only)
[2024/10/18 12:00:32] [ info] [input:node_exporter_metrics:node_exporter_metrics.0] path.procfs = /proc
[2024/10/18 12:00:32] [ info] [input:node_exporter_metrics:node_exporter_metrics.0] path.sysfs  = /sys
[2024/10/18 12:00:32] [ info] [input:node_exporter_metrics:node_exporter_metrics.0] thread instance initialized
[2024/10/18 12:00:32] [ info] [sp] stream processor started
[2024/10/18 12:00:32] [ info] [output:stdout:stdout.0] worker #0 started
2024-10-18T11:00:41.549229017Z node_disk_reads_completed_total{device="mmcblk0"} = 3.0025354765217666e+260
2024-10-18T11:00:41.549383444Z node_disk_reads_completed_total{device="mmcblk0p1"} = 7.4500037861315083e+258
2024-10-18T11:00:41.549514642Z node_disk_reads_completed_total{device="mmcblk0p2"} = 2.9114798746912705e+260
2024-10-18T11:00:41.549244225Z node_disk_reads_merged_total{device="mmcblk0"} = 1.3087584577787325e+260
2024-10-18T11:00:41.549390319Z node_disk_reads_merged_total{device="mmcblk0p1"} = 1.3349980751172437e+258
2024-10-18T11:00:41.549521465Z node_disk_reads_merged_total{device="mmcblk0p2"} = 1.29540847702756e+260
2024-10-18T11:00:41.549251100Z node_disk_read_bytes_total{device="mmcblk0"} = 1.4997288792814694e+262
2024-10-18T11:00:41.549396465Z node_disk_read_bytes_total{device="mmcblk0p1"} = 5.3471981257277754e+260
2024-10-18T11:00:41.549527194Z node_disk_read_bytes_total{device="mmcblk0p2"} = 1.4408065858850202e+262
2024-10-18T11:00:41.549257246Z node_disk_read_time_seconds_total{device="mmcblk0"} = 1.6406547429910954e+261
2024-10-18T11:00:41.549401777Z node_disk_read_time_seconds_total{device="mmcblk0p1"} = 2.7433234957431625e+259
2024-10-18T11:00:41.549533027Z node_disk_read_time_seconds_total{device="mmcblk0p2"} = 1.6101835994645522e+261
2024-10-18T11:00:41.549264329Z node_disk_writes_completed_total{device="mmcblk0"} = 7.816627497858684e+262
2024-10-18T11:00:41.549407767Z node_disk_writes_completed_total{device="mmcblk0p1"} = 4.6027543046422402e+256
2024-10-18T11:00:41.549538548Z node_disk_writes_completed_total{device="mmcblk0p2"} = 7.8166228951043804e+262
2024-10-18T11:00:41.549270475Z node_disk_writes_merged_total{device="mmcblk0"} = 2.2760793912247862e+262
2024-10-18T11:00:41.549412559Z node_disk_writes_merged_total{device="mmcblk0p1"} = 0
2024-10-18T11:00:41.549544225Z node_disk_writes_merged_total{device="mmcblk0p2"} = 2.2760793912247862e+262
2024-10-18T11:00:41.549276621Z node_disk_written_bytes_total{device="mmcblk0"} = 4.4877672212591046e+264
2024-10-18T11:00:41.549418027Z node_disk_written_bytes_total{device="mmcblk0p1"} = 4.6024793710790856e+256
2024-10-18T11:00:41.549549538Z node_disk_written_bytes_total{device="mmcblk0p2"} = 4.4877671752343106e+264
2024-10-18T11:00:41.549282610Z node_disk_write_time_seconds_total{device="mmcblk0"} = 8.7072976110370311e+264
2024-10-18T11:00:41.549423079Z node_disk_write_time_seconds_total{device="mmcblk0p1"} = 4.6044039055078484e+256
2024-10-18T11:00:41.549555319Z node_disk_write_time_seconds_total{device="mmcblk0p2"} = 8.7072975189489542e+264
2024-10-18T11:00:41.549295788Z node_disk_io_time_seconds_total{device="mmcblk0"} = 3.3366413226191617e+263
2024-10-18T11:00:41.549432611Z node_disk_io_time_seconds_total{device="mmcblk0p1"} = 7.7767962245363142e+259
2024-10-18T11:00:41.549564902Z node_disk_io_time_seconds_total{device="mmcblk0p2"} = 3.3358636429967082e+263
2024-10-18T11:00:41.549302350Z node_disk_io_time_weighted_seconds_total{device="mmcblk0"} = 8.7036313823846841e+264
2024-10-18T11:00:41.549438236Z node_disk_io_time_weighted_seconds_total{device="mmcblk0p1"} = 8.1766516154001194e+259
2024-10-18T11:00:41.549570059Z node_disk_io_time_weighted_seconds_total{device="mmcblk0p2"} = 8.7035465789579701e+264
2024-10-18T11:00:41.549308548Z node_disk_discards_completed_total{device="mmcblk0"} = 8.2822359069269866e+259
2024-10-18T11:00:41.549443184Z node_disk_discards_completed_total{device="mmcblk0p1"} = 2.3926459286677962e+258
2024-10-18T11:00:41.549575215Z node_disk_discards_completed_total{device="mmcblk0p2"} = 8.0429713140602068e+259
2024-10-18T11:00:41.549314381Z node_disk_discards_merged_total{device="mmcblk0"} = 5.0656649279781151e+259
2024-10-18T11:00:41.549448652Z node_disk_discards_merged_total{device="mmcblk0p1"} = 2.3925029632594185e+258
2024-10-18T11:00:41.549580215Z node_disk_discards_merged_total{device="mmcblk0p2"} = 4.8264146316521734e+259
2024-10-18T11:00:41.549320892Z node_disk_discarded_sectors_total{device="mmcblk0"} = 6.6337873789088068e+264
2024-10-18T11:00:41.549454173Z node_disk_discarded_sectors_total{device="mmcblk0p1"} = 9.9782575218404259e+263
2024-10-18T11:00:41.549585528Z node_disk_discarded_sectors_total{device="mmcblk0p2"} = 5.6359616267247645e+264
2024-10-18T11:00:41.549326881Z node_disk_discard_time_seconds_total{device="mmcblk0"} = 8970
2024-10-18T11:00:41.549460059Z node_disk_discard_time_seconds_total{device="mmcblk0p1"} = 1179
2024-10-18T11:00:41.549590684Z node_disk_discard_time_seconds_total{device="mmcblk0p2"} = 7791
2024-10-18T11:00:41.549331829Z node_disk_flush_requests_total{device="mmcblk0"} = 0
2024-10-18T11:00:41.549464381Z node_disk_flush_requests_total{device="mmcblk0p1"} = 0
2024-10-18T11:00:41.549595215Z node_disk_flush_requests_total{device="mmcblk0p2"} = 0
2024-10-18T11:00:41.549336829Z node_disk_flush_requests_time_seconds_total{device="mmcblk0"} = 0
2024-10-18T11:00:41.549469121Z node_disk_flush_requests_time_seconds_total{device="mmcblk0p1"} = 0
2024-10-18T11:00:41.549599330Z node_disk_flush_requests_time_seconds_total{device="mmcblk0p2"} = 0
2024-10-18T11:00:41.549289173Z node_disk_io_now{device="mmcblk0"} = 0
2024-10-18T11:00:41.549427819Z node_disk_io_now{device="mmcblk0p1"} = 0
2024-10-18T11:00:41.549560059Z node_disk_io_now{device="mmcblk0p2"} = 0

But /proc/diskstats values look OK,

$ cat /proc/diskstats 
   1       0 ram0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
   1       1 ram1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
   1       2 ram2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
   1       3 ram3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
   1       4 ram4 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
   1       5 ram5 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
   1       6 ram6 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
   1       7 ram7 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
   1       8 ram8 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
   1       9 ram9 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
   1      10 ram10 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
   1      11 ram11 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
   1      12 ram12 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
   1      13 ram13 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
   1      14 ram14 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
   1      15 ram15 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
   7       0 loop0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
   7       1 loop1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
   7       2 loop2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
   7       3 loop3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
   7       4 loop4 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
   7       5 loop5 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
   7       6 loop6 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
   7       7 loop7 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
 179       0 mmcblk0 6529 2843 325794 35644 1698898 538174 97725977 189439565 0 7261280 189484180 1800 1101 144186760 8970 0 0
 179       1 mmcblk0p1 162 29 11616 596 1 0 1 1 0 1690 1777 52 52 21687952 1179 0 0
 179       2 mmcblk0p2 6331 2814 312994 34982 1698897 538174 97725976 189439563 0 7259590 189482337 1748 1049 122498808 7791 0 0

Expected behavior
diskstats metric values should match those seen a /proc/diskstats.

Your Environment

  • Version used: 3.1.9 installed from https://packages.fluentbit.io/ (originally I hit this in 2.1.4)
  • Configuration: see above
  • Environment name and version: Raspberry Pi running current Raspbian bullseye, fluent-bit directly as binary
  • Server type and version: Raspberry Pi 2 Model B Rev 1.1
  • Operating System and version: Raspbian GNU/Linux 11 (bullseye), 32-bit, Linux kernel 6.1.21-v7+ armv7l
  • Filters and plugins: node_metrics_exporter

Metadata

Metadata

Assignees

No one assigned

    Labels

    Stalestatus: waiting-for-triage

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions