I have 2 lambda functions pointing to the same gateway. the first lambda sends:

• requests_num_total{LAMBDA_NAME="test_function"}

the second lambda sends:

• requests_num_total{job="test"}

when I curl the gateway endpoint i see only:

requests_num_total{LAMBDA_NAME="test_function"} 1
requests_num_total{LAMBDA_NAME="test"} 1

where the label job has been overridden by LAMBDA_NAME.

How to reproduce

• run the gateway locally
• send the first metric

echo '# TYPE requests_num_total counter
requests_num_total{LAMBDA_NAME="test_function"} 1' | curl --data-binary @- 127.0.0.1:4278/metrics

• send the second metric

echo '# TYPE requests_num_total counter
requests_num_total{job="test"} 1' | curl --data-binary @- 127.0.0.1:4278/metrics

curl the gateway endpoint curl 127.0.0.1:4278/metrics:

# TYPE requests_num_total counter
requests_num_total{LAMBDA_NAME="test_function"} 1
requests_num_total{LAMBDA_NAME="test"} 1

as you can see the label job has been renamed to LAMBDA_NAME the first label sent always overrides any other following labels.

Hi,

thanks for developing Gravel Gateway!

I have an issue in pushing the same metric with different label values. I'm not sure whether it's something I'm missing, but here's how to reproduce:

• clone master@06e50df0210797b47a7ca13ccf8fb0f4b8bc159e
• cargo run (rustc 1.57.0 (f1edd0429 2021-11-29))
• The following curl returns correctly with 200 OK

echo '# HELP mymetric My Metric
> # TYPE mymetric gauge
> mymetric{clearmode="aggregate",label="value1",job="my_metric"} 1.0' | curl -v -H 'Authorization: x' --data-binary @- localhost:4278/metrics

• Multiple curl invocations as above yield the expected results (summed values for the metric).
• But the following curl returns with 500 with message Unhandled rejection: AggregationError(ParseError(InvalidMetric("Cannot add a sample with 3 labels into a family with 2")))* (see label="value2")

echo '# HELP mymetric My Metric
> # TYPE mymetric gauge
> mymetric{clearmode="aggregate",label="value2",job="my_metric"} 1.0' | curl -v -H 'Authorization: x' --data-binary @- localhost:4278/metrics

Is this something to be expected?

Please let me know if I can be of further help.

Thanks!

The result of using mean5m seems not correct. To reproduce run the gateway locally and send:

echo '# TYPE test_value gauge
test_value{clearmode="mean5m"} 22' | curl --data-binary @- localhost:4278/metrics/job/testjob

expected 22, returns 0

curl localhost:4278/metrics
# TYPE test_value gauge
test_value{job="testjob"} 0

the result is wrong for all successive submissions.

Hello @sinkingpoint, i'd like to ask few questions about clustering:

• is this feature using any membership algorithm (like serf for example) so that if a node of the cluster is down the metric gets re-assigned to another node?
• is the metric proxy best effort? If a node is down or temp unavailable the data point just discarded?
• can the cluster be scaled in-out? is the data re-organized when this happens?

From what i'm reading the current clustering feature is just best-effort and depending on the requirements not production ready?

Trying to use pebbles with {clearmode="mean5m"} and it doesn't seem to work, maybe i'm getting the doc wrong. To replicate, run the gateway locally and perform:

echo '# TYPE value2 gauge
value2{clearmode="mean5m"} 1' | curl --data-binary @- localhost:4278/metrics/job/lttest

this POST will successfully complete, by perform a GET /metrics we get:

# TYPE value2 gauge
value2{job="lttest"} 0

let's now trying to push another value for the same gauge metric:

echo '# TYPE value2 gauge
value2{clearmode="mean5m"} 10' | curl --data-binary @- localhost:4278/metrics/job/lttest

will receive the following error:

Unhandled rejection: AggregationError(Error("invalid push - new push has different label names than the existing family"))

Found some unsupported items example:

# HELP jvm_memory_bytes_used Used bytes of a given JVM memory area
# TYPE jvm_memory_bytes_used gauge
jvm_memory_bytes_used{area="heap",} 9.4094392E12


# Service=myapp, date=Fri May 27 11:34:23 UTC 2022

1. coma at the end of labels list is unsupported
2. exponent uppercase letter "E" is not supported, but just lowercase "e"
3. 2 and more blank lines after the last metric is not supported

Tested with:

docker build -t prometheus-gravel-gateway .
docker run -it --rm -e RUST_BACKTRACE=1 -p 4278:4278 prometheus-gravel-gateway

The only problem with the dockerized app is that doesn't manager the INT signal, so pressing CTRL-C does not interrupt the daemon, the service has to be stopped with a docker kill

The new version of openmetrics-parser loosens the prometheus grammar a bit and thus should make us more compliant with actual Prometheus text exposition in the wild

In theory this solves #5

Hello, when trialing out this project I encountered a bug when a job ran that didn't have a value recorded for one of the metrics. Any metrics detailed in the PUT after the empty metric would cause the server to throw back a 400 error, the Prometheus pushgateway handles empty metrics without errors.

Sample PUT:

# HELP metric_without_values_total This metric does not always have values
# TYPE metric_without_values_total counter
# HELP metric_with_values_total This metric will always have values
# TYPE metric_with_values_total counter
metric_with_values_total{a_label="label_value",another_label="a_value"} 1.0
# HELP metric_with_values_created This metric will always have values
# TYPE metric_with_values_created gauge
metric_with_values_created{a_label="label_value",another_label="a_value"} 1.665577650707084e+09

Gives the following error back:

Invalid metric name in family. Family name is metric_without_values_total, but got a metric called metric_with_values_total

Code to reproduce (Python):

from prometheus_client import CollectorRegistry, Counter, push_to_gateway

registry = CollectorRegistry()

no_values = Counter(
    "metric_without_values",
    "This metric does not always have values",
    ["label"],
    registry=registry
)

has_values = Counter(
    "metric_with_values",
    "This metric will always have values",
    ["a_label", "another_label"],
    registry=registry
)

has_values.labels(a_label="label_value", another_label="a_value").inc()
push_to_gateway("localhost:4278", job="test_job", registry=registry)

I understand the comment and intention here:

You'll note that we don't base64 the authorization header, so it's not technically Basic Auth, but I don't like Base64ing it because I believe that gives a false sense of security. Instead, you should enable TLS

However, the basic auth spec themselves mention that base64 has nothing to do with security and should be used in conjuction with TLS https://www.rfc-editor.org/rfc/rfc7617#section-1

The downside of not following the spec is integration efforts with libraries. For example, I want to use prom-client to push to gravel gateway but now I can't use the methods meant for that because they will obviously base64 the authorization header for me.

I would request the change to expect a base64 encoded authorization header to improve client integration. Or maybe just accept both ways for backwards compatibility

In order to deploy the gravel-gateway within our K8s cluster, a helm chart would be helpful. This would make operations like ha scaling easier. We tried a drop in replacement in the pushgateway chart but needed to disable the probes for that. If a Helm chart is not feasible, could you add a section in the Readme which probes (Liveness, Readiness) should be configured.

All prometheus_client example uses push_to_gateway function to push metrics to the prometheus gateway. https://github.com/prometheus/client_python#exporting-to-a-pushgateway

The push_to_gateway method uses PUT as http method: https://github.com/prometheus/client_python/blob/v0.14.1/prometheus_client/exposition.py#L448

meanwhile pushadd_to_gateway uses POST as HTTP method: https://github.com/prometheus/client_python/blob/v0.14.1/prometheus_client/exposition.py#L479

would be nice to add support to PUT method as well so that developers can switch gateways with no code change.

Hi

I saw your talk on KubeCon EU and found this project quite interesting, one thing I was wondering how you are tackling in your use case (if it is needed) is persistent state.

As far as I can see in the readme or the code, there isnt anything written to disk and it is "more or less" stateless.

I assume this would mean that if the gateway is restarted then it will loose the metrics that was already located in there and you "risk" having a scrape with absent metrics?

How do you tackle this issue in your setup?, and is persisting the state something that you would consider useful? The state could for example be written to blob/s3 like storage periodically and allow it to start up again with data from the last session.