New Plugin

Phase 1: Understand How the Library Is Used

Read the target library's documentation, quickstart guides, or sample code. Understand the user-facing API — how developers create clients, make calls, and handle responses. This tells you:

• What objects are long-lived (clients, connections, pools) vs. per-request (requests, commands)
• Where configuration lives (server address, credentials, timeouts)
• Whether the library is sync, async, or reactive
• Whether it uses callbacks, futures, or blocking calls

Example thought process for a Redis client:

User creates: RedisClient client = RedisClient.create("redis://localhost:6379");
User connects: StatefulRedisConnection conn = client.connect();
User executes: conn.sync().get("key");  // or conn.async().get("key")

This tells you: connection holds the server address, commands are executed on the connection.

Phase 2: Trace the Execution Flow

Starting from the user-facing API, trace inward through the library source code to understand the execution workflow:

1. What happens when the user calls the API method?
2. Where does the request object get built?
3. Where is the actual network I/O or dispatch?
4. Where is the response available?
5. For RPC/MQ: where are headers/metadata accessible for inject/extract?

Key question at each point: What data is directly accessible as method arguments, return values, or fields on this? You want interception points where you can read the data you need without reflection.

Phase 3: Choose Interception Points

Pick interception points based on these principles:

Principle 1: Data accessibility without reflection. Choose methods where the information you need (peer address, operation name, request/response details, headers for inject/extract) is directly available as method arguments, return values, or accessible through the this object's public API. Never use reflection to read private fields. If the data is not accessible at one method, look at a different point in the execution flow.

If the target class is package-private (e.g., final class without public), you cannot import or cast to it. Same-package helper classes do NOT work because the agent and application use different classloaders — Java treats them as different runtime packages even with the same package name (IllegalAccessError). Use setAccessible reflection to call public methods:

try {
    java.lang.reflect.Method method = objInst.getClass().getMethod("publicMethodName");
    method.setAccessible(true);  // Required for package-private class
    Object result = method.invoke(objInst);
} catch (Exception e) {
    LOGGER.warn("Failed to access method", e);
}

Principle 2: Use EnhancedInstance dynamic field to propagate context inside the library. This is the primary mechanism for passing data between interception points. The agent adds a dynamic field to every enhanced class via EnhancedInstance. Use it to:

• Store server address (peer) at connection/client creation time, retrieve it at command execution time
• Store request info at request-build time, retrieve it at send time
• Pass span references from the initiating method to the completion callback

Do NOT use Map or other caches to store per-instance context. Always use the dynamic field on the relevant EnhancedInstance. Maps introduce memory leaks, concurrency issues, and are slower than the direct field access that EnhancedInstance provides.

Principle 3: Intercept the minimal set of methods. Prefer one well-chosen interception point over many surface-level ones. If a library has 20 command methods that all flow through a single dispatch() method internally, intercept dispatch() — not all 20.

Principle 4: Pick points where you can do inject/extract for cross-process propagation. For RPC/HTTP/MQ plugins, you need to inject trace context into outgoing requests (ExitSpan) or extract from incoming requests (EntrySpan). The interception point MUST be where headers/metadata are writable (inject) or readable (extract). If headers are not accessible at the execution method, look for:

• A request builder/decorator stage where headers can be added
• A channel/transport layer where metadata is attached
• A message properties object accessible from the method arguments

Principle 5: Consider the span lifecycle across threads. If the library dispatches work asynchronously:

• Identify where the work is submitted (original thread) and where the result arrives (callback/future thread)
• You may need interception points in both threads
• Use EnhancedInstance dynamic field on the task/callback/future object to carry the span or ContextSnapshot across the thread boundary
• Use prepareForAsync() / asyncFinish() if the span must stay open across threads

Phase 4: Map Out the Interception Plan

Before writing code, create a clear plan listing:

For each interception point, verify:

• The data I need is readable from method args, this, or EnhancedInstance dynamic field — no reflection needed
• For inject: I can write headers/metadata through a public API on a method argument
• For extract: I can read headers/metadata through a public API on a method argument
• The this object (or a method argument) will be enhanced as EnhancedInstance, so I can use the dynamic field

Choosing Span Type

For Meter Plugins

Meter plugins follow the same understand-then-intercept process, but the goal is to find objects that expose numeric state:

• Gauges: Intercept the creation of pool/executor/connection-manager objects. Register a MeterFactory.gauge() with a supplier lambda that calls the object's own getter methods (e.g., pool.getActiveCount()). Store the gauge reference in the dynamic field if needed.
• Counters: Intercept execution methods and call counter.increment() on each invocation.
• Histograms: Intercept methods where duration or size is computable (measure between before/after, or read from response).

Check Existing Plugins for Reference

Before writing a new plugin, check if a similar library already has a plugin:

apm-sniffer/apm-sdk-plugin/          # 70+ standard plugins
apm-sniffer/optional-plugins/         # Optional plugins
apm-sniffer/bootstrap-plugins/        # JDK-level plugins

Similar libraries often share execution patterns. Study how an existing plugin for a similar library solved the same problems — especially how it chains dynamic fields across multiple interception points and where it does inject/extract.

Verify Against Actual Source Code — Never Speculate

This applies to both new plugin development AND extending existing plugins to newer library versions.

When assessing whether a plugin works with a new library version, or when choosing interception points for a new plugin, you MUST read the actual source code of the target library at the specific version. Do NOT rely on:

• Version number assumptions ("it's still 4.x so it should be compatible")
• Changelog summaries (they don't list every internal class rename or method removal)
• General knowledge about the library's public API (plugins intercept internal classes, which change without notice)

What to verify for each intercepted class/method:

1. Does the target class still exist at the exact FQCN? (internal classes get renamed, extracted, or removed between minor versions)
2. Does the intercepted method still exist with a compatible signature? (parameters may be added/removed/reordered)
3. Do the witness classes still correctly distinguish versions? (a witness class that exists in both old and new versions won't prevent the plugin from loading on an incompatible version)
4. Do the runtime APIs called by the interceptor still exist? (e.g., calling cluster.getDescription() will crash if that method was removed, even if the plugin loaded successfully)

How to verify — clone the library source locally:

# Clone specific version tag to /tmp for easy source code inspection
cd /tmp && git clone --depth 1 --branch {tag} https://github.com/{org}/{repo}.git {local-name}

# Examples:
git clone --depth 1 --branch r4.9.0 https://github.com/mongodb/mongo-java-driver.git mongo-4.9
git clone --depth 1 --branch v2.4.13.RELEASE https://github.com/spring-projects/spring-kafka.git spring-kafka-2.4

# Then grep/read the actual source files to check class/method existence
grep -rn "getFilter\|getWriteRequests" /tmp/mongo-4.9/driver-core/src/main/com/mongodb/internal/operation/

This is faster and more reliable than fetching individual files via raw GitHub URLs. You can grep, diff between versions, and trace the full execution path.

Also check for import-time class loading failures: If a plugin helper class (not just the instrumentation class) imports a library class that was removed, the entire helper class will fail to load with NoClassDefFoundError at runtime. This silently breaks ALL functionality in that helper — not just the code paths using the removed class. Verify that every import statement in plugin support classes resolves to an existing class in the target version.

Real examples of why this matters:

• MongoDB driver 4.9 removed InsertOperation, DeleteOperation, UpdateOperation — MongoOperationHelper imported all three, causing the entire class to fail loading with NoClassDefFoundError, silently losing ALL db.bind_vars tags even for operations that still exist (like FindOperation, AggregateOperation)
• MongoDB driver 4.11 removed Cluster.getDescription() — the plugin loads (witness classes pass) but crashes at runtime with NoSuchMethodError
• Feign 12.2 moved ReflectiveFeign$BuildTemplateByResolvingArgs to RequestTemplateFactoryResolver$BuildTemplateByResolvingArgs — the path variable interception silently stops working
• MariaDB 3.0 renamed every JDBC wrapper class (MariaDbConnection → Connection) — none of the plugin's byName matchers match anything

When extending support-version.list to add newer versions: Before adding a version, verify that every class and method the plugin intercepts still exists in that version's source. A plugin test passing does not mean everything works — it only means the test scenario's specific code path exercised the intercepted methods. Missing interception points may go undetected if the test doesn't cover them.

Step 2 - Create Plugin Module

Directory Structure

SDK plugin (most common):

apm-sniffer/apm-sdk-plugin/{framework}-{version}-plugin/
  pom.xml
  src/main/java/org/apache/skywalking/apm/plugin/{framework}/v{N}/
    define/
      {Target}Instrumentation.java          # One per target class
    {Target}Interceptor.java                # One per interception concern
    {Target}ConstructorInterceptor.java     # If intercepting constructors
    {PluginName}PluginConfig.java           # If plugin needs configuration
  src/main/resources/
    skywalking-plugin.def                   # Plugin registration
  src/test/java/org/apache/skywalking/apm/plugin/{framework}/v{N}/
    {Target}InterceptorTest.java            # Unit tests

Bootstrap plugin (for JDK classes):

apm-sniffer/bootstrap-plugins/{name}-plugin/
  (same structure, but instrumentation class overrides isBootstrapInstrumentation)

Optional plugin:

apm-sniffer/optional-plugins/{name}-plugin/
  (same structure as SDK plugin)

pom.xml Template

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <parent>
        <groupId>org.apache.skywalking</groupId>
        <artifactId>apm-sdk-plugin</artifactId>  <!-- or bootstrap-plugins / optional-plugins -->
        <version>${revision}</version>
    </parent>

    <artifactId>{framework}-{version}-plugin</artifactId>
    <packaging>jar</packaging>

    <properties>
        <target-library.version>X.Y.Z</target-library.version>
    </properties>

    <dependencies>
        <!-- Target library - MUST be provided scope -->
        <dependency>
            <groupId>com.example</groupId>
            <artifactId>target-library</artifactId>
            <version>${target-library.version}</version>
            <scope>provided</scope>
        </dependency>
    </dependencies>
</project>

CRITICAL dependency rules:

• Target library: always provided scope (supplied by the application at runtime)
• apm-agent-core: inherited from parent POM as provided
• apm-util: inherited from parent POM as provided
• Never bundle target library classes into the plugin JAR
• If the plugin needs a 3rd-party utility not already in agent-core, discuss with maintainers first

CRITICAL compiler level rule:

• Do NOT set maven.compiler.release or maven.compiler.source/target unless the plugin source code itself uses JDK 9+ language features (e.g., var, records, sealed classes, java.net.http.HttpClient).
• A provided-scope dependency targeting a higher JDK (e.g., Spring AI requires JDK 17+) does NOT require raising the compiler level — the plugin only references the library's API at compile time.
• Bootstrap plugins like jdk-httpclient-plugin (which uses JDK 11 HttpClient API directly in plugin source code) legitimately need a higher compiler target. SDK plugins and optional plugins generally should not.

Register in Parent POM

Add the new module to the parent pom.xml:

<modules>
    ...
    <module>{framework}-{version}-plugin</module>
</modules>

Step 3 - Implement Instrumentation Class (V2 API)

ALWAYS use V2 API for new plugins. V1 is legacy.

Import Rules (Enforced by Checkstyle)

Plugins may ONLY import from:

• java.* - Java standard library
• org.apache.skywalking.* - SkyWalking modules
• net.bytebuddy.* - ByteBuddy (for matchers in instrumentation classes)

No other 3rd-party imports are allowed in instrumentation/activation files. This is enforced by apm-checkstyle/importControl.xml. Interceptor classes CAN reference target library classes (they're loaded after the target library).

Class Matching

CRITICAL: NEVER use .class references in instrumentation definitions. Always use string literals.

// WRONG - breaks agent if class doesn't exist at runtime
byName(SomeThirdPartyClass.class.getName())
takesArgument(0, SomeThirdPartyClass.class)

// CORRECT - safe string literals
byName("com.example.SomeThirdPartyClass")
takesArgumentWithType(0, "com.example.SomeThirdPartyClass")

Available ClassMatch types (from org.apache.skywalking.apm.agent.core.plugin.match):

Prefer NameMatch.byName() whenever possible. It uses a fast HashMap lookup. All other matchers require linear scanning.

Method Matching (ByteBuddy ElementMatcher API)

Common matchers from net.bytebuddy.matcher.ElementMatchers:

// By name
named("methodName")

// By argument count
takesArguments(2)
takesArguments(0)  // no-arg methods

// By argument type (use SkyWalking's helper - string-based, safe)
import static org.apache.skywalking.apm.agent.core.plugin.bytebuddy.ArgumentTypeNameMatch.takesArgumentWithType;
takesArgumentWithType(0, "com.example.SomeType")  // arg at index 0

// By return type (SkyWalking helper)
import static org.apache.skywalking.apm.agent.core.plugin.bytebuddy.ReturnTypeNameMatch.returnsWithType;
returnsWithType("java.util.List")

// By annotation (SkyWalking helper - string-based)
import static org.apache.skywalking.apm.agent.core.plugin.bytebuddy.AnnotationTypeNameMatch.isAnnotatedWithType;
isAnnotatedWithType("org.springframework.web.bind.annotation.RequestMapping")

// Visibility
isPublic()
isPrivate()

// Composition
named("execute").and(takesArguments(1))
named("method1").or(named("method2"))
not(isDeclaredBy(Object.class))

// Match any (use sparingly)
any()

Instrumentation Template - Instance Methods

package org.apache.skywalking.apm.plugin.xxx.define;

import net.bytebuddy.description.method.MethodDescription;
import net.bytebuddy.matcher.ElementMatcher;
import org.apache.skywalking.apm.agent.core.plugin.interceptor.ConstructorInterceptPoint;
import org.apache.skywalking.apm.agent.core.plugin.interceptor.v2.InstanceMethodsInterceptV2Point;
import org.apache.skywalking.apm.agent.core.plugin.interceptor.enhance.v2.ClassInstanceMethodsEnhancePluginDefineV2;
import org.apache.skywalking.apm.agent.core.plugin.match.ClassMatch;
import org.apache.skywalking.apm.agent.core.plugin.match.NameMatch;

import static net.bytebuddy.matcher.ElementMatchers.named;

public class XxxInstrumentation extends ClassInstanceMethodsEnhancePluginDefineV2 {

    private static final String ENHANCE_CLASS = "com.example.TargetClass";
    private static final String INTERCEPTOR_CLASS = "org.apache.skywalking.apm.plugin.xxx.XxxInterceptor";

    @Override
    protected ClassMatch enhanceClass() {
        return NameMatch.byName(ENHANCE_CLASS);
    }

    @Override
    public ConstructorInterceptPoint[] getConstructorsInterceptPoints() {
        return null;  // null or empty array if not intercepting constructors
    }

    @Override
    public InstanceMethodsInterceptV2Point[] getInstanceMethodsInterceptV2Points() {
        return new InstanceMethodsInterceptV2Point[] {
            new InstanceMethodsInterceptV2Point() {
                @Override
                public ElementMatcher<MethodDescription> getMethodsMatcher() {
                    return named("targetMethod");
                }

                @Override
                public String getMethodsInterceptorV2() {
                    return INTERCEPTOR_CLASS;
                }

                @Override
                public boolean isOverrideArgs() {
                    return false;
                }
            }
        };
    }
}

Instrumentation Template - Static Methods

import org.apache.skywalking.apm.agent.core.plugin.interceptor.StaticMethodsInterceptPoint;
import org.apache.skywalking.apm.agent.core.plugin.interceptor.enhance.v2.ClassStaticMethodsEnhancePluginDefineV2;

public class XxxStaticInstrumentation extends ClassStaticMethodsEnhancePluginDefineV2 {
    @Override
    public StaticMethodsInterceptPoint[] getStaticMethodsInterceptPoints() {
        return new StaticMethodsInterceptPoint[] {
            new StaticMethodsInterceptPoint() {
                @Override
                public ElementMatcher<MethodDescription> getMethodsMatcher() {
                    return named("factoryMethod").and(takesArguments(2));
                }

                @Override
                public String getMethodsInterceptor() {
                    return INTERCEPTOR_CLASS;
                }

                @Override
                public boolean isOverrideArgs() {
                    return false;
                }
            }
        };
    }

    @Override
    protected ClassMatch enhanceClass() {
        return NameMatch.byName(ENHANCE_CLASS);
    }
}

Instrumentation Template - Both Instance + Static Methods

Extend ClassEnhancePluginDefineV2 and implement all four methods:

• enhanceClass()
• getConstructorsInterceptPoints()
• getInstanceMethodsInterceptV2Points()
• getStaticMethodsInterceptPoints()

Matching Interface/Abstract Class Implementations

Use HierarchyMatch when you need to intercept all implementations of an interface:

import org.apache.skywalking.apm.agent.core.plugin.match.HierarchyMatch;

@Override
protected ClassMatch enhanceClass() {
    return HierarchyMatch.byHierarchyMatch("com.example.SomeInterface");
}

When to use HierarchyMatch:

• The library has an interface/abstract class with multiple implementations
• You cannot enumerate all implementation class names
• Example: intercepting all javax.servlet.Servlet implementations

Performance warning: HierarchyMatch checks every loaded class against the hierarchy. Prefer NameMatch or MultiClassNameMatch if you know the concrete class names.

Combining with other matchers:

import org.apache.skywalking.apm.agent.core.plugin.match.logical.LogicalMatchOperation;

@Override
protected ClassMatch enhanceClass() {
    return LogicalMatchOperation.and(
        PrefixMatch.nameStartsWith("com.example"),
        HierarchyMatch.byHierarchyMatch("java.lang.Runnable")
    );
}

Witness Classes for Version Detection

Override witnessClasses() or witnessMethods() to activate the plugin only for specific library versions:

@Override
protected String[] witnessClasses() {
    // Plugin only loads if this class exists in the application
    return new String[] {"com.example.VersionSpecificClass"};
}

@Override
protected List<WitnessMethod> witnessMethods() {
    return Collections.singletonList(
        new WitnessMethod("com.example.SomeClass", ElementMatchers.named("methodAddedInV2"))
    );
}

Bootstrap Plugin Override

For bootstrap plugins (instrumenting JDK classes), add:

@Override
public boolean isBootstrapInstrumentation() {
    return true;
}

Bootstrap plugin rules:

• Only for JDK core classes (java., javax., sun.*)
• Minimal interception scope (performance-critical paths)
• Extra care with class loading (bootstrap classloader visibility)
• Test with runningMode: with_bootstrap

Step 4 - Implement Interceptor Class (V2 API)

Available Interceptor Interfaces

Core APIs Available in Interceptors

ContextManager - Central tracing API (ThreadLocal-based):

CRITICAL threading rule: All span lifecycle APIs (createEntrySpan, createExitSpan, createLocalSpan, activeSpan, stopSpan) operate on a per-thread context via ThreadLocal. By default, createXxxSpan and stopSpan MUST be called in the same thread. There are only two ways to work across threads:

1. ContextSnapshot (capture/continued) — snapshot the context in thread A, then continued() in thread B to link a NEW span in thread B back to the parent trace. Each thread manages its own span lifecycle independently.
2. Async mode (prepareForAsync/asyncFinish) — keeps a single span alive beyond the creating thread. Call prepareForAsync() in the original thread (before stopSpan), then asyncFinish() from any thread when the async work completes. Between prepareForAsync and asyncFinish, you may call tag/log/error on the span from any thread, but you must NOT call ContextManager.stopSpan() for that span again.

import org.apache.skywalking.apm.agent.core.context.ContextManager;

// Create spans (must stopSpan in the SAME thread, unless async mode)
AbstractSpan span = ContextManager.createEntrySpan(operationName, contextCarrier);
AbstractSpan span = ContextManager.createLocalSpan(operationName);
AbstractSpan span = ContextManager.createExitSpan(operationName, contextCarrier, remotePeer);
AbstractSpan span = ContextManager.createExitSpan(operationName, remotePeer);

// Span lifecycle (same thread as create, unless async mode)
ContextManager.activeSpan();      // Get current span in THIS thread
ContextManager.stopSpan();        // Stop current span in THIS thread
ContextManager.isActive();        // Check if context exists in THIS thread

// Cross-process propagation (inject/extract ContextCarrier into headers/metadata)
ContextManager.inject(carrier);   // Inject into outgoing carrier
ContextManager.extract(carrier);  // Extract from incoming carrier

// Cross-thread propagation (ContextSnapshot — link spans across threads)
ContextManager.capture();         // Capture snapshot in originating thread
ContextManager.continued(snapshot); // Continue from snapshot in receiving thread

// Trace metadata
ContextManager.getGlobalTraceId();
ContextManager.getSegmentId();
ContextManager.getSpanId();

AbstractSpan - Span configuration:

span.setComponent(ComponentsDefine.YOUR_COMPONENT);  // Required for Entry/Exit
span.setLayer(SpanLayer.HTTP);                        // Required for Entry/Exit
span.setOperationName("GET:/api/users");
span.setPeer("host:port");                            // Required for Exit spans

// Tags
span.tag(Tags.URL, url);
span.tag(Tags.HTTP_RESPONSE_STATUS_CODE, statusCode);
span.tag(Tags.DB_TYPE, "sql");
span.tag(Tags.DB_STATEMENT, sql);
span.tag(Tags.ofKey("custom.key"), value);

// Error handling
span.errorOccurred();
span.log(throwable);

// Async support
span.prepareForAsync();  // Must call in original thread
span.asyncFinish();      // Call in async thread when done

SpanLayer values: DB, RPC_FRAMEWORK, HTTP, MQ, CACHE, GEN_AI

Standard Tags (from org.apache.skywalking.apm.agent.core.context.tag.Tags):

EnhancedInstance - Dynamic field for cross-interceptor data:

// Store data (e.g., in constructor interceptor)
objInst.setSkyWalkingDynamicField(connectionInfo);

// Retrieve data (e.g., in method interceptor)
ConnectionInfo info = (ConnectionInfo) objInst.getSkyWalkingDynamicField();

Logging - Agent-internal logging (NOT application logging):

import org.apache.skywalking.apm.agent.core.logging.api.ILog;
import org.apache.skywalking.apm.agent.core.logging.api.LogManager;

private static final ILog LOGGER = LogManager.getLogger(MyInterceptor.class);
LOGGER.info("message: {}", value);
LOGGER.error("error", throwable);

MeterFactory - For meter plugins:

import org.apache.skywalking.apm.toolkit.meter.MeterFactory;
import org.apache.skywalking.apm.toolkit.meter.Counter;
import org.apache.skywalking.apm.toolkit.meter.Gauge;
import org.apache.skywalking.apm.toolkit.meter.Histogram;

Counter counter = MeterFactory.counter("metric_name")
    .tag("key", "value")
    .mode(Counter.Mode.INCREMENT)
    .build();
counter.increment(1.0);

Gauge gauge = MeterFactory.gauge("metric_name", () -> pool.getActiveCount())
    .tag("pool_name", name)
    .build();

Histogram histogram = MeterFactory.histogram("metric_name")
    .steps(Arrays.asList(10.0, 50.0, 100.0, 500.0))
    .minValue(0)
    .build();
histogram.addValue(latencyMs);

Interceptor Template - ExitSpan (Client/Producer)

package org.apache.skywalking.apm.plugin.xxx;

import java.lang.reflect.Method;
import org.apache.skywalking.apm.agent.core.context.CarrierItem;
import org.apache.skywalking.apm.agent.core.context.ContextCarrier;
import org.apache.skywalking.apm.agent.core.context.ContextManager;
import org.apache.skywalking.apm.agent.core.context.tag.Tags;
import org.apache.skywalking.apm.agent.core.context.trace.AbstractSpan;
import org.apache.skywalking.apm.agent.core.context.trace.SpanLayer;
import org.apache.skywalking.apm.agent.core.plugin.interceptor.enhance.EnhancedInstance;
import org.apache.skywalking.apm.agent.core.plugin.interceptor.enhance.v2.InstanceMethodsAroundInterceptorV2;
import org.apache.skywalking.apm.agent.core.plugin.interceptor.enhance.v2.MethodInvocationContext;
import org.apache.skywalking.apm.network.trace.component.ComponentsDefine;

public class XxxClientInterceptor implements InstanceMethodsAroundInterceptorV2 {

    @Override
    public void beforeMethod(EnhancedInstance objInst, Method method, Object[] allArguments,
                             Class<?>[] argumentsTypes, MethodInvocationContext context) throws Throwable {
        // 1. Build peer address from stored connection info
        String remotePeer = (String) objInst.getSkyWalkingDynamicField();

        // 2. Create ExitSpan with ContextCarrier for cross-process propagation
        ContextCarrier contextCarrier = new ContextCarrier();
        AbstractSpan span = ContextManager.createExitSpan("operation/name", contextCarrier, remotePeer);
        span.setComponent(ComponentsDefine.YOUR_COMPONENT);
        SpanLayer.asHttp(span);  // or asDB, asMQ, asRPCFramework, asCache

        // 3. Inject trace context into outgoing request headers
        // The request object is typically one of the method arguments
        CarrierItem next = contextCarrier.items();
        while (next.hasNext()) {
            next = next.next();
            // Set header on the outgoing request:
            // request.setHeader(next.getHeadKey(), next.getHeadValue());
        }

        // 4. Set tags
        Tags.URL.set(span, url);

        // 5. Store span in context for afterMethod
        context.setContext(span);
    }

    @Override
    public Object afterMethod(EnhancedInstance objInst, Method method, Object[] allArguments,
                              Class<?>[] argumentsTypes, Object ret, MethodInvocationContext context) throws Throwable {
        // Check response status, set tags/errors
        AbstractSpan span = (AbstractSpan) context.getContext();
        // Example: Tags.HTTP_RESPONSE_STATUS_CODE.set(span, statusCode);
        // if (statusCode >= 400) span.errorOccurred();

        ContextManager.stopSpan();
        return ret;
    }

    @Override
    public void handleMethodException(EnhancedInstance objInst, Method method, Object[] allArguments,
                                      Class<?>[] argumentsTypes, Throwable t, MethodInvocationContext context) {
        ContextManager.activeSpan().log(t);
        ContextManager.activeSpan().errorOccurred();
    }
}

Interceptor Template - EntrySpan (Server/Consumer)

public class XxxServerInterceptor implements InstanceMethodsAroundInterceptorV2 {

    @Override
    public void beforeMethod(EnhancedInstance objInst, Method method, Object[] allArguments,
                             Class<?>[] argumentsTypes, MethodInvocationContext context) throws Throwable {
        // 1. Extract trace context from incoming request headers
        ContextCarrier contextCarrier = new ContextCarrier();
        CarrierItem next = contextCarrier.items();
        while (next.hasNext()) {
            next = next.next();
            // Read header from incoming request:
            // next.setHeadValue(request.getHeader(next.getHeadKey()));
        }

        // 2. Create EntrySpan (extracts context automatically)
        AbstractSpan span = ContextManager.createEntrySpan("operation/name", contextCarrier);
        span.setComponent(ComponentsDefine.YOUR_COMPONENT);
        SpanLayer.asHttp(span);  // or asMQ, asRPCFramework
        span.setPeer(clientAddress);  // Optional: client address

        context.setContext(span);
    }

    @Override
    public Object afterMethod(EnhancedInstance objInst, Method method, Object[] allArguments,
                              Class<?>[] argumentsTypes, Object ret, MethodInvocationContext context) throws Throwable {
        ContextManager.stopSpan();
        return ret;
    }

    @Override
    public void handleMethodException(EnhancedInstance objInst, Method method, Object[] allArguments,
                                      Class<?>[] argumentsTypes, Throwable t, MethodInvocationContext context) {
        ContextManager.activeSpan().log(t);
        ContextManager.activeSpan().errorOccurred();
    }
}

Interceptor Template - Constructor (Store Connection Info)

import org.apache.skywalking.apm.agent.core.plugin.interceptor.enhance.InstanceConstructorInterceptor;

public class XxxConstructorInterceptor implements InstanceConstructorInterceptor {

    @Override
    public void onConstruct(EnhancedInstance objInst, Object[] allArguments) {
        // Store connection info for later use by method interceptors
        String host = (String) allArguments[0];
        int port = (int) allArguments[1];
        objInst.setSkyWalkingDynamicField(host + ":" + port);
    }
}

Cross-Thread Context Propagation (ContextSnapshot)

Use ContextSnapshot when the library dispatches work to another thread and you want the new thread's spans to be linked to the parent trace. Each thread creates and stops its OWN spans — the snapshot only provides the link.

// Thread A (originating thread) — create span, capture snapshot, stop span (all same thread)
@Override
public void beforeMethod(..., MethodInvocationContext context) {
    AbstractSpan span = ContextManager.createLocalSpan("async/dispatch");
    // Capture context snapshot BEFORE handing off to another thread
    ContextSnapshot snapshot = ContextManager.capture();
    // Store snapshot on the task object via EnhancedInstance dynamic field
    ((EnhancedInstance) allArguments[0]).setSkyWalkingDynamicField(snapshot);
    ContextManager.stopSpan();  // Stop span in THIS thread (same thread as create)
}

// Thread B (receiving thread) — create its OWN span, link to parent via continued()
@Override
public void beforeMethod(EnhancedInstance objInst, ...) {
    ContextSnapshot snapshot = (ContextSnapshot) objInst.getSkyWalkingDynamicField();
    if (snapshot != null) {
        AbstractSpan span = ContextManager.createLocalSpan("async/execute");
        ContextManager.continued(snapshot);  // Link this span to the parent trace
    }
}

@Override
public Object afterMethod(...) {
    if (ContextManager.isActive()) {
        ContextManager.stopSpan();  // Stop span in THIS thread (same thread as create)
    }
    return ret;
}

Async Span Pattern (prepareForAsync / asyncFinish)

Use this when a single span needs to stay open across thread boundaries — e.g., an ExitSpan created before an async call, finished when the callback fires in another thread. The key difference from ContextSnapshot: here one span lives across threads instead of each thread having its own span.

// Thread A — create span, mark async, stop context (all same thread)
@Override
public void beforeMethod(..., MethodInvocationContext context) {
    AbstractSpan span = ContextManager.createExitSpan("async/call", remotePeer);
    span.setComponent(ComponentsDefine.YOUR_COMPONENT);
    SpanLayer.asHttp(span);

    span.prepareForAsync();       // Mark: this span will finish in another thread
    ContextManager.stopSpan();    // Detach from THIS thread's context (required, same thread as create)

    // Store span reference on the callback object's dynamic field
    ((EnhancedInstance) callback).setSkyWalkingDynamicField(span);
}

// Thread B (callback/completion handler) — finish the async span
@Override
public void beforeMethod(EnhancedInstance objInst, ...) {
    AbstractSpan span = (AbstractSpan) objInst.getSkyWalkingDynamicField();
    if (span != null) {
        // Add response info to the span (tag/log/error are thread-safe after prepareForAsync)
        span.tag(Tags.HTTP_RESPONSE_STATUS_CODE, statusCode);
        if (isError) span.errorOccurred();
        span.asyncFinish();  // Must match prepareForAsync count
    }
}

Plugin Configuration (Optional)

import org.apache.skywalking.apm.agent.core.boot.PluginConfig;

public class XxxPluginConfig {
    public static class Plugin {
        @PluginConfig(root = XxxPluginConfig.class)
        public static class Xxx {
            // Config key: plugin.xxx.trace_param
            public static boolean TRACE_PARAM = false;
            // Config key: plugin.xxx.max_length
            public static int MAX_LENGTH = 256;
        }
    }
}

Step 5 - Register Plugin

Create src/main/resources/skywalking-plugin.def:

plugin-name=org.apache.skywalking.apm.plugin.xxx.define.XxxInstrumentation
plugin-name=org.apache.skywalking.apm.plugin.xxx.define.XxxOtherInstrumentation

Format: {plugin-id}={fully.qualified.InstrumentationClassName} (one line per instrumentation class, all sharing the same plugin-id prefix).

Step 6 - Write Unit Tests

Test Setup Pattern

import org.junit.Before;
import org.junit.Rule;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.mockito.Mock;
import org.mockito.junit.MockitoJUnit;
import org.mockito.junit.MockitoRule;
import org.apache.skywalking.apm.agent.core.plugin.interceptor.enhance.EnhancedInstance;
import org.apache.skywalking.apm.agent.test.tools.AgentServiceRule;
import org.apache.skywalking.apm.agent.test.tools.SegmentStorage;
import org.apache.skywalking.apm.agent.test.tools.SegmentStoragePoint;
import org.apache.skywalking.apm.agent.test.tools.TracingSegmentRunner;

@RunWith(TracingSegmentRunner.class)
public class XxxInterceptorTest {

    @SegmentStoragePoint
    private SegmentStorage segmentStorage;

    @Rule
    public AgentServiceRule agentServiceRule = new AgentServiceRule();

    @Rule
    public MockitoRule rule = MockitoJUnit.rule();

    @Mock
    private EnhancedInstance enhancedInstance;

    private XxxInterceptor interceptor;

    @Before
    public void setUp() {
        interceptor = new XxxInterceptor();
        // Setup mocks
    }

    @Test
    public void testNormalRequest() throws Throwable {
        // Arrange
        Object[] allArguments = new Object[] { /* mock args */ };
        Class[] argumentsTypes = new Class[] { /* arg types */ };

        // Act
        interceptor.beforeMethod(enhancedInstance, null, allArguments, argumentsTypes, null);
        interceptor.afterMethod(enhancedInstance, null, allArguments, argumentsTypes, mockResponse, null);

        // Assert spans
        assertThat(segmentStorage.getTraceSegments().size(), is(1));
        TraceSegment segment = segmentStorage.getTraceSegments().get(0);
        List<AbstractTracingSpan> spans = SegmentHelper.getSpans(segment);
        assertThat(spans.size(), is(1));
        // Verify span properties...
    }
}

Step 7 - Write E2E Plugin Tests

Test Scenario Structure

test/plugin/scenarios/{framework}-{version}-scenario/
  bin/startup.sh
  config/expectedData.yaml
  src/main/java/org/apache/skywalking/apm/testcase/{framework}/
    controller/CaseController.java    # HTTP endpoints
  pom.xml
  configuration.yml
  support-version.list

When copying an existing scenario to create a new one, update the scenario name in ALL of these files:

• pom.xml — artifactId, name, finalName
• src/main/assembly/assembly.xml — JAR filename reference
• bin/startup.sh — JAR filename in java -jar command
• config/expectedData.yaml — serviceName field AND parentService in refs (but NOT URL paths — those are the app context path)
• support-version.list — new versions
• For JDK 17+ scenarios: update compiler.version to 17, spring.boot.version to 3.x, change javax.annotation imports to jakarta.annotation in Java source
• Add the scenario to the appropriate CI workflow (plugins-test.*.yaml for JDK 8, plugins-jdk17-test.*.yaml for JDK 17)

configuration.yml