feat: metricsV2 + oTel + prometheus sample and Grafana dashboard

[csviri]

Goal of this PR is to provide a OTel + Prometheus + Grafana setup. So we:

1. verify integration with OTel
2. Provide a default Grafana dashboard for metrics.
3. Adds a new operator and E2E test for h OTel + Prometheus + Grafana handling verification. This should be easy to done by a users, so dey can check and validate the dashboard.
4. new metrics implementations

Notes on new metrics implementation:

• only static gauges (lives until the operator lives) - that makes much easir to manage them
• no manual counter removal (should not be done)
• sanitized names and labels according to best practices
• added some metrics counter for last attempt retries.

[csviri]

JVM metrics:

[csviri]

JODSK metrics:

Added TODOs to improve those, like we should not have controller name as suffix, rather as a tag:

operator_sdk_reconciliations_executions_webpagestandalonedependentsreconciler

[xstefank]

is "active" really a good name for this?

[csviri]

see below.

[xstefank]

queue would be better name

[csviri]

The thing is that queue is a bit misleading, because this is actually the queue + ongoing reconiliations, that is the reason I added active. (ExecutorService has a queue). Can expand the docs on this

[csviri]

Actually made an adjustment so we have now active and queue where queue is that are submitted to executor service but not started yet, the active is the number of actually running threads with reconiliation logic. This seems to be more intuitive and useful.

[metacosm]

This would technically be an API break and would require a new major version.

[csviri]

Strinctly sepaking yes, but such minor API changes we do some times, see the migration document. As other frameworks sometimes. It is basically I think a better choice in terms of tradeoff, since because we don't really want to increase the major verion that often and we on the other hand we have quite an amount of APIs, that sometimes better to evolve this way IMO.

I also was trying to do backwards compatible, we still could. But at the end it looked like that it would be more confusing, that just having a table to be able to easily migrate from current impl. If that makes sense.

[metacosm]

Installing helm should be made optional. People might not want to have helm automatically installed or want to install it themselves some other way.

[csviri]

What this does, it checks if helm command works (so already installed), if not, downloads it as a binary, but just locally in the project dir (so does not install it), and uses that binary directly. So this is not very invasive at the end IMO, usually it is done this way in such scripts.

[metacosm]

SLO acronym should be used expanded first before.

[csviri]

SLO I consider a well known term, but actually we can just remote that from here. done.

[metacosm]

Shouldn't that be in sync with what's done for MDC, i.e. use a default value instead of null for clustered resources?

[csviri]

This way at the end we don't add the namespace tag (if null), that is according the guidelines.

[metacosm]

Then I'd argue that MDC should follow the same guidelines, otherwise it's just confusing to have telemetry report one thing while MDC does something else, especially if you're trying to correlate logs and telemetry.

[csviri]

I don't see why should we unify those completely unrelated systems.

[metacosm]

This shouldn't be a List but a Set.

[csviri]

That is really an implementation detail IMO

[csviri]

Hmm, was thinking about this, actually, should not be a Set: basically, what we really want is just a list of Tags in memory, that we later iterate through. While Set might create some secondary data structures like indexes for fast access, what we don't want in this case. So using list is a better both memory and mental model IMO :)

[metacosm]

We don't want duplicated tags, nor do we care about their order, that's what set are for and that matches what people would expect. As a user, I don't want duplicated tags to show up, which is a possibility with the current implementation.

[csviri]

If you take the look on Micrometer API:

 public Counter counter(String name, Iterable<Tag> tags) {
        return this.counter(name, Tags.of(tags));
    }

    public Counter counter(String name, String... tags) {
        return this.counter(name, Tags.of(tags));
    }

    public Counter counter(String name, Tags tags) {
        return this.counter(new Meter.Id(name, tags, (String)null, (String)null, Type.COUNTER));
    }

Does not work with Sets either.

Set's usually maintains an addiitonal index in memory, we don't want to have that memory overhead. While adding an additional tag should not be our responsibility filter out, that is an usage error.

[metacosm]

I'm fine with using Iterable. The semantics should not be a list, though, because this implies order, which does not exist.

[csviri]

They have also an array of tags api, that implies order... I don't think that is a bad approach; the other way to look at this, is that we just want to have a list of elemnts in memory (not a linked list, not a set, since those have memory overhead).

Is there order? yes
Do we care about the order? no

[csviri]

But will take a look how it would look like with iterable.

[csviri]

Hmm, checking, but those are 2 liner methods, internal APIs, there is no memory overhead. This is completely fine this way.

[metacosm]

How does the retry number get propagated? Seems like the counter that is incremented is completely decorrelated from the actual retry number?

[csviri]

The idea is that we measure the number of retries, so we increase this number if the actual reconiliation is a retry. What is interesting at the end in the metrics is the rate of the retries.

[metacosm]

Why was the todo removed? This still needs to be done.

[csviri]

Ahh should not be removed in this PR, will put it back.

Wanted to remove these TODOs for which we already have issues, so that when go through TODOs I see just relevant, in general it is a good practice to remove all the TODO's from the code before we merge and rather create issue, since it pollutes the code.