[Bug]: KafkaConnect maven builds ignore scopes which cause errors with plugins that have older Kafka dependencies

[dalelane]

Bug Description

Some Kafka Connect plugins include Connect artifacts as dependencies.

For example, I include this in my KafkaConnect custom resource:

spec:
  build:
    plugins:
      - name: myconnector
        artifacts:
          - artifact: debezium-scripting
            group: io.debezium
            type: maven
            version: 3.2.1.Final

And my resulting plugins folder contains:

7751d6fd:
total 103228
-rw-r--r--. 1 kafka root     4149 May 30 14:30 annotations-1.4.0.jar
-rw-r--r--. 1 kafka root   126113 Mar 29 15:13 asm-9.8.jar
-rw-r--r--. 1 kafka root    35151 Mar 29 15:13 asm-analysis-9.8.jar
-rw-r--r--. 1 kafka root    73498 Mar 29 15:13 asm-commons-9.8.jar
-rw-r--r--. 1 kafka root    51934 Mar 29 15:13 asm-tree-9.8.jar
-rw-r--r--. 1 kafka root    94559 Mar 29 15:13 asm-util-9.8.jar
-rw-r--r--. 1 kafka root  1396119 Jan 18  2025 assertj-core-3.27.3.jar
-rw-r--r--. 1 kafka root  8469021 Dec 15  2024 byte-buddy-1.15.11.jar
-rw-r--r--. 1 kafka root    88470 May 30 14:30 compiler-1.4.0.jar
-rw-r--r--. 1 kafka root   109977 May 12 05:10 connect-api-3.9.1.jar
-rw-r--r--. 1 kafka root   118601 May 12 05:12 connect-transforms-3.9.1.jar
-rw-r--r--. 1 kafka root    30976 Aug 13 09:24 debezium-api-3.2.1.Final.jar
-rw-r--r--. 1 kafka root    14816 Aug 13 09:24 debezium-common-3.2.1.Final.jar
-rw-r--r--. 1 kafka root   430640 Aug 13 09:24 debezium-core-3.2.1.Final-tests.jar
-rw-r--r--. 1 kafka root  1305726 Aug 13 09:24 debezium-core-3.2.1.Final.jar
-rw-r--r--. 1 kafka root    14683 Aug 13 09:24 debezium-openlineage-api-3.2.1.Final.jar
-rw-r--r--. 1 kafka root    32888 Sep  1 15:15 debezium-scripting-3.2.1.Final.jar
-rw-r--r--. 1 kafka root     1629 Aug 13 09:24 debezium-scripting-languages-3.2.1.Final.pom
-rw-r--r--. 1 kafka root   813240 Oct 16  2023 graal-sdk-23.0.2.jar
-rw-r--r--. 1 kafka root  7597629 Dec 19  2023 groovy-4.0.17.jar
-rw-r--r--. 1 kafka root   129920 Dec 19  2023 groovy-json-4.0.17.jar
-rw-r--r--. 1 kafka root    16924 Dec 19  2023 groovy-jsr223-4.0.17.jar
-rw-r--r--. 1 kafka root    45024 Jul  9  2012 hamcrest-core-1.3.jar
-rw-r--r--. 1 kafka root 14386028 Oct 19  2022 icu4j-72.1.jar
-rw-r--r--. 1 kafka root    78481 Mar  9  2024 jackson-annotations-2.16.2.jar
-rw-r--r--. 1 kafka root   582784 Mar  9  2024 jackson-core-2.16.2.jar
-rw-r--r--. 1 kafka root  1638583 Mar  9  2024 jackson-databind-2.16.2.jar
-rw-r--r--. 1 kafka root   127530 Mar  9  2024 jackson-datatype-jsr310-2.16.2.jar
-rw-r--r--. 1 kafka root   126898 Aug 24  2018 javax.ws.rs-api-2.1.1.jar
-rw-r--r--. 1 kafka root 27387378 Oct 16  2023 js-23.0.2.jar
-rw-r--r--. 1 kafka root    75769 Oct 16  2023 js-scriptengine-23.0.2.jar
-rw-r--r--. 1 kafka root   382708 Oct 11  2020 junit-4.13.1.jar
-rw-r--r--. 1 kafka root  9215194 May 12 05:08 kafka-clients-3.9.1.jar
-rw-r--r--. 1 kafka root   232073 Dec  1  2023 logback-classic-1.2.13.jar
-rw-r--r--. 1 kafka root   449638 Dec  1  2023 logback-core-1.2.13.jar
-rw-r--r--. 1 kafka root   682804 Jun 19  2021 lz4-java-1.8.0.jar
-rw-r--r--. 1 kafka root  3097882 Oct 16  2023 regex-23.0.2.jar
-rw-r--r--. 1 kafka root   104459 May 30 14:30 runtime-1.4.0.jar
-rw-r--r--. 1 kafka root    41125 Feb  8  2022 slf4j-api-1.7.36.jar
-rw-r--r--. 1 kafka root  2299797 Sep 27  2023 snappy-java-1.1.10.5.jar
-rw-r--r--. 1 kafka root 16210116 Oct 16  2023 truffle-api-23.0.2.jar
-rw-r--r--. 1 kafka root   172837 May 30 14:30 wasm-1.4.0.jar
-rw-r--r--. 1 kafka root  7325504 Sep 22  2024 zstd-jni-1.5.6-6.jar

The connect jars are the problem:

-rw-r--r--. 1 kafka root   109977 May 12 05:10 connect-api-3.9.1.jar
-rw-r--r--. 1 kafka root   118601 May 12 05:12 connect-transforms-3.9.1.jar

This is because I'm running Kafka 4.x and the mismatch results in a runtime error

java.lang.NoSuchMethodError: org/apache/kafka/common/utils/ConfigUtils.translateDeprecatedConfigs(Ljava/util/Map;[[Ljava/lang/String;)Ljava/util/Map;
        (loaded from file:/opt/kafka/libs/kafka-clients-4.0.0.jar by jdk.internal.loader.ClassLoaders$AppClassLoader@ef1275b)
        called from class org.apache.kafka.connect.transforms.ReplaceField (loaded
        from file:/opt/kafka/plugins/myconnector/7751d6fd/connect-transforms-3.9.1.jar by
        PluginClassLoader{pluginLocation=file:/opt/kafka/plugins/myconnector/}).\n\tat org.apache.kafka.connect.transforms.ReplaceField.configure(ReplaceField.java:101)\n\tat
        org.apache.kafka.connect.runtime.ConnectorConfig.transformationStages(ConnectorConfig.java:316)\n\tat
        org.apache.kafka.connect.runtime.Worker$TaskBuilder.build(Worker.java:1793)\n\tat
        org.apache.kafka.connect.runtime.Worker.startTask(Worker.java:702)\n\tat org.apache.kafka.connect.runtime.Worker.startSourceTask(Worker.java:591)\n\tat
        org.apache.kafka.connect.runtime.distributed.DistributedHerder.startTask(DistributedHerder.java:2042)\n\tat
        org.apache.kafka.connect.runtime.distributed.DistributedHerder.lambda$getTaskStartingCallable$39(DistributedHerder.java:2059)\n\tat
        java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\tat
        java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat
        java.base/java.lang.Thread.run(Thread.java:853)

The maven artifact does have the Connect dependencies in a "provided" scope (ref) so it feels reasonable that we could've known to exclude them from the build.

Looking at the BuildConfig that the Strimzi Operator generated, the dockerfile contains:

      RUN 'curl' '-f' '-L' '--create-dirs' '--output' '/tmp/myconnector/7751d6fd/pom.xml' 'https://repo1.maven.org/maven2/io/debezium/debezium-scripting/3.2.1.Final/debezium-scripting-3.2.1.Final.pom' \
            && 'echo' '<settings xmlns="http://maven.apache.org/SETTINGS/1.0.0"><profiles><profile><id>download</id><repositories><repository><id>custom-repo</id><url>https://repo1.maven.org/maven2/</url></repository></repositories></profile></profiles><activeProfiles><activeProfile>download</activeProfile></activeProfiles></settings>' > '/tmp/7751d6fd.xml' \
            && 'mvn' 'dependency:copy-dependencies' '-s' '/tmp/7751d6fd.xml' '-DoutputDirectory=/tmp/artifacts/myconnector/7751d6fd' '-f' '/tmp/myconnector/7751d6fd/pom.xml' \
            && 'curl' '-f' '-L' '--create-dirs' '--output' '/tmp/artifacts/myconnector/7751d6fd/debezium-scripting-3.2.1.Final.jar' 'https://repo1.maven.org/maven2/io/debezium/debezium-scripting/3.2.1.Final/debezium-scripting-3.2.1.Final.jar'

We're copying all dependencies, irregardless of scope.

I think if we added -DincludeScope=runtime to the mvn dependency:copy-dependencies command, it'd be more appropriate. This would have excluded the provided dependencies, so there wouldn't be duplicate / conflicting Kafka jars in the plugins path. (It'd also exclude test dependencies which isn't a bad thing as it'll reduce the size of the resulting image).

I've seen this today with Debezium plugins, but I've seen the same problem previously with Apicurio plugins

e.g. you can recreate java.lang.NoSuchMethodError errors by creating a KafkaConnect with:

- artifact: apicurio-registry-serdes-avro-serde
  group: io.apicurio
  type: maven
  version: <VERSION>
- artifact: apicurio-registry-utils-converter
  group: io.apicurio
  type: maven
  version: <VERSION>

Does it sound like I'm on the right track here?

Steps to reproduce

1. Create a KafkaConnect custom resource using Kafka 4.x
2. Add artifacts to the build spec that have provided scope dependencies of Kafka 3.x (e.g. Apicurio, Debezium, etc.)
3. Start a connector that uses one of those dependencies

Expected behavior

The connector should start without a java.lang.NoSuchMethodError

Strimzi version

main

Kubernetes version

OpenShift 4.19

Installation method

No response

Infrastructure

No response

Configuration files and logs

No response

Additional context

No response

[dalelane]

Does it sound like I'm on the right track here?

btw - happy to contribute a fix to this, if there is agreement on the overall diagnosis / suggestion.

[scholzj]

I did not had time to double check your findings. But I think it does make sense to use only runtime dependencies. One of the things I was wondering about is whether this can break something. But I guess if it does, the affected users would need to add addtional dependencies 🤷.

[dalelane]

Digging into the details a little...

The Debezium plugin that caused my initial problem has connect-api as a provided dependency so my suggested change of adding -DincludeScope=runtime would have worked. includeScope=runtime includes only the runtime and compile dependencies, not provided.

I had assumed this would also fix problems I'd heard of with Apicurio plugins, but now I look more carefully I see that they have connect-api and connect-json as compile dependencies. includeScope=runtime would not avoid them, as it gives runtime and compile dependencies

So a full fix that also avoids java.lang.NoSuchMethodError with Apicurio plugins is going to be more complex than I'd hoped. Adding excludeArtifactIds with a list of Kafka Connect artifacts is my first brute-force fix, but I need to read into mvn behaviour a bit more to see if there is anything more elegant :)

[scholzj]

I'm not sure that invalidates this issue. I think this rather indicates some other issues as well.

I ran into the connect-json absence in the Apache Camel connectors. And I think I saw the connect-api problem somewhere as well. However, I do not think this means that using the runtime scope dependencies only is a wrong idea:

• Clearly, there was some change in the Kafka classpath that caused these issues (I guess with Kafka 4.0)?
• Was this change intentional in Kafka?
• If it was an intentional change on the Kadka side, is it now a mistake that the connectors have these dependencies as provider and not runtime and should it be fixed there?

In any case, I think it means that as I suspected, there might be some impact of just treating this as a bug and simply switching to runtime scope only. It can be still worked around. You can just add second artifact:

          - type: maven
            group: org.apache.kafka
            artifact: connect-json
            version: 4.0.0

But if it breaks things, it would definitely need more noise about this change to warn the users. We can also consider making it configurable. I.e. something like adding the scope field to configure the scope. (but as an API change that might need a proposal)

I personally would be against adding some excluded artifacts. I think it would be too complicated and not worth it. And to be honest, I do not see anymore the Kafka Connect Build as the future. I would instead hope that overtime we move to Image Volumes and things like this would be solved when creating the OCI artifact. But for example, adding the scope option as mentioned above seems to be pretty easy to add, so I would nto see that as an issue 🤷‍♂️.

@mimaison Out of curiosity, are you aware of any changes to Connect classpath that woudl cause issues with the things such as the connect-api or connect-json? I don't think I saw these issues before.

[mimaison]

I can't recall of any changes related to Connect classpath in 4.0. I wonder if we just notice this now as 4.0 introduced changes in ConfigUtils. Is Debezium shipping the Kafka JARs?

[dalelane]

@mimaison They're not intending to, but they declare it as a dependency. Because the KafkaConnect maven build is fetching all maven dependencies (regardless of scope) we end up with both 3.x and 4.x Kafka jars.

[neeraj-laad]

I think changing Strimzi to only get runtime dependencies is a good thing anyway. It will also make the build process quicker and images smaller. I've raised a PR with the changes that I think might be needed, but was unsure on the docs side. Any pointers would be appreciated.

[scholzj]

@neeraj-laad I do not think it is that simple given it seems it breaks many connectors.