Skip to content

Add Java instrumentation agent skill - #4624

Open
hegerchr wants to merge 1 commit into
elastic:mainfrom
hegerchr:add-java-agent-instrumentation-skill
Open

hegerchr wants to merge 1 commit into
elastic:mainfrom
hegerchr:add-java-agent-instrumentation-skill

Conversation

@hegerchr

Copy link
Copy Markdown
Contributor

What does this PR do?

Adds a reusable agent skill for designing, implementing, validating, documenting, and reviewing built-in Elastic APM Java instrumentation plugins.

The skill is discoverable under .agents/skills/create-java-agent-instrumentation-plugin/ and captures repository-specific guidance for:

  • Selecting stable, non-duplicating instrumentation points.
  • Defining supported-version boundaries and classloader guards.
  • Handling nullability and agent/application classloader separation.
  • Implementing synchronous, asynchronous, and reactive span lifecycles.
  • Applying semantic conventions with reliable metadata provenance.
  • Capturing sensitive content safely.
  • Adding metrics without unnecessarily changing agent core.
  • Choosing the smallest effective testing topology.
  • Verifying plugin packaging and full-agent assembly.
  • Completing documentation and the new-plugin PR checklist.

The guidance incorporates lessons learned while implementing and reviewing the Spring AI instrumentation, but is intentionally technology-independent.

This PR changes only agent-development guidance. It does not modify production agent behavior, APIs, configuration, or supported technologies.

Checklist

  • This is an enhancement of existing features, or a new feature in existing plugins
    • I have updated CHANGELOG.next-release.md
    • I have added tests that prove my fix is effective or that my feature works
    • Added an API method or config option? Document in which version this will be introduced
    • I have made corresponding changes to the documentation
  • This is a bugfix
  • This is a new plugin
    • I have updated CHANGELOG.next-release.md
    • My code follows the style guidelines of this project
    • I have made corresponding changes to the documentation
    • I have added tests that prove my fix is effective or that my feature works
    • New and existing unit tests pass locally with my changes
    • I have updated supported-technologies.md
    • Added an API method or config option? Document in which version this will be introduced
    • Added an instrumentation plugin? Describe how you made sure that old, non-supported versions are not instrumented by accident.
  • This is something else
    • I have updated CHANGELOG.next-release.md — Not applicable; this adds development guidance without changing agent behavior.

@hegerchr
hegerchr requested a review from a team as a code owner November 13, 2026 08:55
Copilot AI balanced review requested due to automatic review settings November 13, 2026 08:55
@mergify

mergify Bot commented Nov 13, 2026

Copy link
Copy Markdown
Contributor

Tick the box to add this pull request to the merge queue (same as @mergifyio queue). If it does not match the queue conditions yet, it enters the queue once it does.

  • Queue this pull request

@github-actions

Copy link
Copy Markdown

🤖 GitHub comments

Just comment with:

  • run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The skill guide still recommends invalid span-type fallbacks, a JAR glob that prevents packaged-agent inspection, and a deprecated test runner incompatible with JUnit Jupiter suites.

3 open issues
What changed in this PR

Adds a reusable skill for developing and reviewing built-in Elastic APM Java instrumentation plugins. This PR changes development guidance only, not production behavior, APIs, or configuration.

Changes:

  • Documents instrumentation selection, compatibility boundaries, class-loader safety, and span lifecycles.
  • Covers telemetry semantics, sensitive-content handling, and metrics.
  • Defines testing, agent assembly verification, documentation, and PR completion guidance.
File Description
.agents/​skills/​create-java-agent-instrumentation-plugin/​SKILL.md Adds end-to-end guidance and checklists for built-in instrumentation plugin development.

Review effort: Balanced


Do not invent a second field for data already represented by a standard attribute without a documented compatibility reason.

Keep semantic attributes separate from the agent's internal span taxonomy. External library values may be appropriate for attributes such as `db.system` or service-target metadata but invalid as APM `span.type`, `span.subtype`, or `span.action`. Check the repository's allowed span-type schema; use a schema-defined generic value such as `unknown` when no reliable mapping exists, and preserve the raw value only in an unconstrained semantic field.
Then verify that the packaged agent contains the transformed plugin classes/service entries and launch it once:

```bash
unzip -l elastic-apm-agent/target/elastic-apm-agent-*.jar | rg '<plugin-package|plugin-artifact>'
The repository uses several complementary patterns:

1. **In-process instrumentation tests.** Most library plugins extend `AbstractInstrumentationTest`, execute the real library API with Byte Buddy instrumentation active, and assert against `MockReporter`. Prefer this for matcher/advice behavior, span fields, context propagation, errors, configuration changes, and library calls that do not require packaging boundaries.
2. **Isolated dependency-version tests.** Use the repository's dependency-loading test runner, such as `JUnit4TestClassWithDependencyRunner`, when the same behavioral suite must run against multiple library versions or when the application class path could hide class-loader problems. Explicitly include split transitive artifacts required by newer versions.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, using TestClassWithDependencyRunner is preferred, migrating existing usages of the junit4 runner is not necessary.

- It is high enough in the stack to avoid provider-specific duplication, but low enough to run for all intended entry paths.
- It runs once per logical operation. Watch for overload delegation, default methods, wrappers, retries, and implementations calling other instrumented methods.
- It preserves downstream work beneath the new span when a nested HTTP, database, or messaging span should be its child.
- It can be matched narrowly and cheaply.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For example, ideally the matching should be on FQN class/method name, use heuristics on name to make matching efficient, and rely on class hierarchy only when the number of classes has been reduced to a small number.


Prefer a stable framework abstraction or canonical implementation method over:

- controllers or application code, which miss non-web use;

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what does "non-web use" mean ?

Prefer a stable framework abstraction or canonical implementation method over:

- controllers or application code, which miss non-web use;
- low-level network calls, which lose framework semantics and duplicate existing HTTP instrumentation;

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- low-level network calls, which lose framework semantics and duplicate existing HTTP instrumentation;
- low-level network calls, which lose framework semantics and might duplicate existing framework instrumentation like HTTP or database clients.

- constructors, getters, builders, or hot utility methods without a logical operation boundary;
- internal implementation classes that vary by provider or patch release.

When an interface declares the operation, instrument concrete implementations with `hasSuperType(...)` and exclude interfaces. Add a cheap name-based prefilter when practical. Match the exact method name, argument count, argument types, return type, and relevant modifiers so unrelated overloads are not transformed.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

when the implementation uses a nested method call, prefer to instrument the overload that is being delegated to as it will allow to instrument all overloads at once.


When an interface declares the operation, instrument concrete implementations with `hasSuperType(...)` and exclude interfaces. Add a cheap name-based prefilter when practical. Match the exact method name, argument count, argument types, return type, and relevant modifiers so unrelated overloads are not transformed.

Test that one application call creates exactly one intended span.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

where does this requirement comes from ?


Test that one application call creates exactly one intended span.

Do not add call-depth suppression merely because wrappers might exist. First reproduce a wrapper or overload delegating into another matched method and assert the duplicate spans. If duplication is real, add a shared `CallDepth` guard, decrement it on every exit path including exceptions and no-span paths, and keep the regression test proving that only the outer logical call is reported.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the case where nested method calls should be avoided as they produce duplicate or nested spans, use the CallDepth to guard and only create a span in the outer method call.


The marker must distinguish the supported line from older versions that contain the common API. Put shared guards in a common base instrumentation so every instrumentation in the plugin uses the same boundary.

Add isolated compatibility tests that load dependency jars with a dedicated `URLClassLoader`, preferably with a `null` parent, and assert:

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use OpenTelemetryVersionIT test class as an example how to create tests to test multiple versions in a isolated way.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants