Fix Solr 10 migration by dpol1 · Pull Request #1833 · apache/stormcrawler

dpol1 · 2026-03-20T16:25:06Z

Fixes #1831

…-PARTY

…Solr 10

rzo1 · 2026-03-20T19:49:55Z

Hi @dpol1

thanks for the PR !

Can you check the pipeline status? Guess it needs auto formatting. Thanks

dpol1 · 2026-03-20T19:59:03Z

Oops, my bad 😅

…olr 10

rzo1

For me, it looks good but I am not deep into SOLR.

rzo1 · 2026-03-21T17:25:15Z

Thanks for the PR, @dpol1

external/solr/configsets/docs/conf/solrconfig.xml

mvolikas · 2026-03-22T15:32:47Z

external/solr/src/test/java/org/apache/stormcrawler/solr/SolrContainerTest.java

+                "--name",
                collectionName,
-                "-n",
+                "--conf-name",


Some options (like --conf-name) are not documented in the Solr Control Script Reference, but are shown when running bin/solr create --help so I guess are ok to use.

external/solr/src/main/java/org/apache/stormcrawler/solr/SolrConnection.java

- Drop pseudo-async CompletableFuture.runAsync() around blocking cloudClient.request(). - Use NIO-based LBAsyncSolrClient.requestAsync() for Solr 10. - Route writes via LBSolrClient.Endpoint built from leader baseUrl/coreName. - Access CloudHttp2SolrClient#getLbClient() via explicit cast, with a note on potential SolrJ API changes.

mvolikas · 2026-03-23T16:42:19Z

Thanks for the changes @dpol1. Unfortunately I'm currently sick, so I'll be able to review earliest tomorrow.

external/solr/src/main/java/org/apache/stormcrawler/solr/SolrConnection.java

mvolikas · 2026-03-25T17:10:50Z

Thanks for the patience 😇. @dpol1 I've left some more comments.

…oad balancing

mvolikas

We also need to update the commands in the archetype scripts. See here.

mvolikas · 2026-03-29T18:20:23Z

I tried to run from this branch using a local Solr cluster in Linux.

With Storm 2.8.2, I was getting the following exception:

18:50:11.431 [Thread-37-status-executor[5, 5]] ERROR o.a.s.u.Utils - Async loop died!
java.lang.NoSuchMethodError: 'boolean org.eclipse.jetty.http2.client.HTTP2Client.installBean(java.lang.Object)'
        at org.eclipse.jetty.http2.client.HTTP2Client.<init>(HTTP2Client.java:133)
        at org.apache.solr.client.solrj.jetty.HttpJettySolrClient.createHttpClient(HttpJettySolrClient.java:292)
        at org.apache.solr.client.solrj.jetty.HttpJettySolrClient.<init>(HttpJettySolrClient.java:152)
        at org.apache.solr.client.solrj.jetty.HttpJettySolrClient$Builder.build(HttpJettySolrClient.java:987)
        at org.apache.stormcrawler.solr.SolrConnection.getConnection(SolrConnection.java:364)
        at org.apache.stormcrawler.solr.persistence.StatusUpdaterBolt.prepare(StatusUpdaterBolt.java:89)
        at org.apache.storm.executor.bolt.BoltExecutor.init(BoltExecutor.java:128)
        at org.apache.storm.executor.bolt.BoltExecutor.call(BoltExecutor.java:138)
        at org.apache.storm.executor.bolt.BoltExecutor.call(BoltExecutor.java:54)
        at org.apache.storm.utils.Utils$1.run(Utils.java:393)
        at java.base/java.lang.Thread.run(Thread.java:840)

Switched to Storm 2.8.5 and then this was gone but was getting the following instead:

19:03:37.318 [SLOT_1027] ERROR o.a.s.d.s.Slot - Error when processing event
java.lang.NoClassDefFoundError: org/apache/storm/shade/org/apache/commons/lang/StringUtils
        at org.apache.stormcrawler.util.URLStreamGrouping.prepare(URLStreamGrouping.java:74)
        at org.apache.storm.daemon.GrouperFactory.mkGrouper(GrouperFactory.java:113)
        at org.apache.storm.executor.Executor.outboundComponents(Executor.java:572)
        at org.apache.storm.executor.Executor.<init>(Executor.java:164)
        at org.apache.storm.executor.bolt.BoltExecutor.<init>(BoltExecutor.java:66)
        at org.apache.storm.executor.Executor.mkExecutor(Executor.java:205)
        at org.apache.storm.executor.LocalExecutor.mkExecutor(LocalExecutor.java:29)
        at org.apache.storm.daemon.worker.Worker.loadWorker(Worker.java:257)
        at org.apache.storm.daemon.worker.Worker.lambda$start$1(Worker.java:206)
        at java.base/java.security.AccessController.doPrivileged(AccessController.java:712)
        at java.base/javax.security.auth.Subject.doAs(Subject.java:439)
        at org.apache.storm.daemon.worker.Worker.start(Worker.java:205)
        at org.apache.storm.daemon.supervisor.LocalContainer.launch(LocalContainer.java:61)
        at org.apache.storm.daemon.supervisor.LocalContainerLauncher.launchContainer(LocalContainerLauncher.java:52)
        at org.apache.storm.daemon.supervisor.Slot.handleWaitingForBlobLocalization(Slot.java:459)
        at org.apache.storm.daemon.supervisor.Slot.stateMachineStep(Slot.java:194)
        at org.apache.storm.daemon.supervisor.Slot.run(Slot.java:1051)
Caused by: java.lang.ClassNotFoundException: org.apache.storm.shade.org.apache.commons.lang.StringUtils
        at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641)
        at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:525)
        ... 17 more

This was because of imports import org.apache.storm.shade.org.apache.commons.lang.StringUtils;.
The issue is resolved if you switch to import org.apache.commons.lang3.StringUtils;.

Then another Jetty related exception causing runtime issues:

19:30:57.629 [Thread-42] ERROR o.a.s.m.MetricsConsumerBolt - Exception occurred during handle metrics
java.lang.NoSuchMethodError: 'org.eclipse.jetty.io.Transport org.eclipse.jetty.io.ClientConnector.newTransport()'
        at org.eclipse.jetty.client.HttpClient.createOrigin(HttpClient.java:478)
        at org.eclipse.jetty.http2.client.transport.HttpClientTransportOverHTTP2.newOrigin(HttpClientTransportOverHTTP2.java:114)
        at org.eclipse.jetty.client.HttpClient.resolveDestination(HttpClient.java:455)
        at org.eclipse.jetty.client.transport.HttpRequest.send(HttpRequest.java:750)
        at org.apache.solr.client.solrj.jetty.HttpJettySolrClient.requestAsync(HttpJettySolrClient.java:407)
        at org.apache.solr.client.solrj.impl.HttpSolrClientBase.requestAsync(HttpSolrClientBase.java:359)
        at org.apache.solr.client.solrj.jetty.LBJettySolrClient.lambda$requestAsyncWithUrl$0(LBJettySolrClient.java:51)
        at org.apache.solr.client.solrj.jetty.HttpJettySolrClient.requestWithBaseUrl(HttpJettySolrClient.java:537)
        at org.apache.solr.client.solrj.jetty.LBJettySolrClient.requestAsyncWithUrl(LBJettySolrClient.java:51)
        at org.apache.solr.client.solrj.impl.LBAsyncSolrClient.doAsyncRequest(LBAsyncSolrClient.java:143)
        at org.apache.solr.client.solrj.impl.LBAsyncSolrClient.requestAsync(LBAsyncSolrClient.java:104)
        at org.apache.stormcrawler.solr.SolrConnection.flushUpdates(SolrConnection.java:235)
        at org.apache.stormcrawler.solr.SolrConnection.updateAsync(SolrConnection.java:177)
        at org.apache.stormcrawler.solr.SolrConnection.addAsync(SolrConnection.java:254)
        at org.apache.stormcrawler.solr.metrics.MetricsConsumer.indexDataPoint(MetricsConsumer.java:114)
        at org.apache.stormcrawler.solr.metrics.MetricsConsumer.handleDataPoints(MetricsConsumer.java:81)
        at org.apache.stormcrawler.solr.metrics.MetricsConsumer.handleDataPoints(MetricsConsumer.java:85)
        at org.apache.stormcrawler.solr.metrics.MetricsConsumer.handleDataPoints(MetricsConsumer.java:74)
        at org.apache.storm.metric.MetricsConsumerBolt$MetricsHandlerRunnable.run(MetricsConsumerBolt.java:131)
        at java.base/java.lang.Thread.run(Thread.java:840)

I found this was because Storm 2.8.5 brings Jetty 12.1.6 and Solr depends on Jetty 12.0.27.
I resolved it by pinning specific versions (12.1.6) in the Solr archetype POM, but I'm not sure whether this is a good idea or there are better ways around this. @rzo1 ?

After all these changes I managed to run and didn't notice any unexpected behavior while stopping and restarting one of the two Solr nodes :-).

rzo1 · 2026-03-29T18:28:00Z

I resolved it by pinning specific versions (12.1.6) in the Solr archetype POM, but I'm not sure whether this is a good idea or there are better ways around this. @rzo1 ?

Thanks for tracking this down and documenting all the steps.

Pinning Jetty to 12.1.6 in the Solr archetype POM works for now, but it tightly couples us to whichever Jetty version Storm happens to ship, so the next Storm upgrade could silently break things again.

The more robust fix (imho) would be to shade the Solr client with its own relocated Jetty (e.g. under org.apache.stormcrawler.solr.shaded.jetty) using the Maven Shade plugin. That way Storm's Jetty and Solr's Jetty coexist on the classpath without interfering with each other, regardless of what either project ships. The downside is that shading is known to confuse CVE scanners, which won't be able to match the relocated classes against vulnerability databases, so any Jetty CVEs would need to be tracked manually.

If we keep the pinning approach, it would be worth adding a comment in the POM to make the coupling explicit: otherwise it's easy to miss during a future Storm upgrade. The StringUtils stuff is fixed in #1847 .

mvolikas · 2026-03-29T18:33:41Z

Thanks @rzo1! I think it's worth trying the approach with the shading.

The StringUtils stuff is fixed in #1847 .

Hmm, I think I was still getting the error after integrating the changes of #1847, and it was resolved only after changing the imports, but maybe I missed something.

rzo1 · 2026-03-29T18:44:04Z

@mvolikas It was done in #1835

rzo1 · 2026-03-29T18:46:37Z

Regarding shading: We are doing something similar in TomEE: https://github.com/apache/tomee/blob/main/deps/commons-dbcp2-shade/pom.xml so might be an inspiration.

dpol1 · 2026-03-31T07:57:01Z

Pinning Jetty to 12.1.6 in the Solr archetype POM works for now, but it tightly couples us to whichever Jetty version Storm happens to ship, so the next Storm upgrade could silently break things again.

The more robust fix (imho) would be to shade the Solr client with its own relocated Jetty (e.g. under org.apache.stormcrawler.solr.shaded.jetty) using the Maven Shade plugin. That way Storm's Jetty and Solr's Jetty coexist on the classpath without interfering with each other, regardless of what either project ships. The downside is that shading is known to confuse CVE scanners, which won't be able to match the relocated classes against vulnerability databases, so any Jetty CVEs would need to be tracked manually.

I'll proceed with the shading-based approach - I'll still be able to implement it in that PR. However, I'll open a ticket to document this - is that what you had in mind, @rzo1? Or should we refine it later?

If we keep the pinning approach, it would be worth adding a comment in the POM to make the coupling explicit: otherwise it's easy to miss during a future Storm upgrade. The StringUtils stuff is fixed in #1847 .

Agreed, good practice for both

- Adjusts Solr command-line arguments in scripts and tests to use the new long-form syntax (e.g., --conf-name, --zk-host). - Introduces Maven shading for Jetty classes to resolve runtime version conflicts with Apache Storm. This prevents NoSuchMethodError by relocating Solr's Jetty dependencies, allowing coexistence with Storm's bundled Jetty version. FIxes apache#1859

rzo1 · 2026-03-31T14:41:57Z

Yep can be done here. The Issue is great, so it appears in the changelog.

rzo1 · 2026-04-02T08:17:28Z

external/solr/pom.xml

The shade plugin configuration has two issues that will prevent it from achieving the goal of embedding Jetty with no transitive Jetty deps:

createDependencyReducedPom must be true

Currently set to false, which means the published POM still declares org.eclipse.jetty:* and org.eclipse.jetty.http2:* as transitive dependencies. Consumers will pull in both the shaded
Jetty (inside the jar) and the original Jetty jars defeating the purpose.

solr-solrj-jetty must be included in the artifactSet

solr-solrj-jetty provides HttpJettySolrClient and ConcurrentUpdateJettySolrClient, which reference org.eclipse.jetty classes directly. Since it is not in the artifactSet, its classes are
not pulled into the shaded jar and its bytecode is not rewritten. At runtime it will try to load the original (unshaded) org.eclipse.jetty.* either causing ClassNotFoundException or hitting Storm's conflicting Jetty 12.1.x.

The two wildcard includes already cover all Jetty artifacts in the dependency tree (solr-solrj and solr-solrj-zookeeper have zero Jetty transitive deps), so no additional includes are
needed beyond solr-solrj-jetty.

--- a/external/solr/pom.xml +++ b/external/solr/pom.xml @@ -59,7 +59,8 @@ <createSourcesJar>true</createSourcesJar> <useBaseVersion>true</useBaseVersion> - <createDependencyReducedPom>false</createDependencyReducedPom> + <createDependencyReducedPom>true</createDependencyReducedPom> <artifactSet> <includes> + <include>org.apache.solr:solr-solrj-jetty</include> <include>org.eclipse.jetty:*</include> <include>org.eclipse.jetty.http2:*</include>

Thanks for the feedback, now should be fine, changed!

- Adjusts Solr command-line arguments in scripts and tests to use the new long-form syntax (e.g., --conf-name, --zk-host). - Introduces Maven shading for Jetty classes to resolve runtime version conflicts with Apache Storm. This prevents NoSuchMethodError by relocating Solr's Jetty dependencies, allowing coexistence with Storm's bundled Jetty version. FIxes apache#1859

dpol1 added 2 commits March 20, 2026 17:06

apache#1831 Bump Solr dependencies aligned to 10.0.0 and update THIRD…

367b5e8

…-PARTY

apache#1831 Migrate HTTP clients to Jetty and update URL parsing for …

deeb71a

…Solr 10

rzo1 requested a review from mvolikas March 20, 2026 19:45

rzo1 added this to the 3.5.2 milestone Mar 20, 2026

rzo1 added enhancement dependency dependency upgrade labels Mar 20, 2026

apache#1831 Update Testcontainers, CLI flags and Docker configs for S…

721d892

…olr 10

dpol1 force-pushed the fix/solr-10-migration branch from 956a76f to 721d892 Compare March 20, 2026 20:29

rzo1 approved these changes Mar 21, 2026

View reviewed changes

mvolikas requested changes Mar 22, 2026

View reviewed changes

dpol1 requested a review from mvolikas March 23, 2026 10:13

mvolikas requested changes Mar 25, 2026

View reviewed changes

apache#1831 Enhance Solr connection handling for async requests and l…

feb415e

…oad balancing

dpol1 requested a review from mvolikas March 26, 2026 17:14

rzo1 mentioned this pull request Mar 28, 2026

Bump org.apache.solr:solr-solrj from 9.10.1 to 10.0.0 #1822

Closed

mvolikas requested changes Mar 29, 2026

View reviewed changes

mvolikas mentioned this pull request Mar 29, 2026

Upgrade to Apache Storm 2.8.5 #1847

Merged

Merge branch 'main' into fix/solr-10-migration

e73adbf

dpol1 force-pushed the fix/solr-10-migration branch from 2b89d34 to e73adbf Compare March 30, 2026 18:05

dpol1 mentioned this pull request Mar 31, 2026

[Improvement] Shade Jetty in the Solr module to avoid classpath conflicts with Storm #1859

Open

dpol1 requested a review from mvolikas April 1, 2026 07:31

rzo1 requested changes Apr 2, 2026

View reviewed changes

dpol1 force-pushed the fix/solr-10-migration branch from 9835b91 to c71b174 Compare April 2, 2026 16:46

Conversation

dpol1 commented Mar 20, 2026

Uh oh!

rzo1 commented Mar 20, 2026

Uh oh!

dpol1 commented Mar 20, 2026

Uh oh!

rzo1 left a comment

Choose a reason for hiding this comment

Uh oh!

rzo1 commented Mar 21, 2026

Uh oh!

Uh oh!

mvolikas Mar 22, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

mvolikas commented Mar 23, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mvolikas commented Mar 25, 2026

Uh oh!

mvolikas left a comment

Choose a reason for hiding this comment

Uh oh!

mvolikas commented Mar 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rzo1 commented Mar 29, 2026

Uh oh!

mvolikas commented Mar 29, 2026

Uh oh!

rzo1 commented Mar 29, 2026

Uh oh!

rzo1 commented Mar 29, 2026

Uh oh!

dpol1 commented Mar 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rzo1 commented Mar 31, 2026

Uh oh!

rzo1 Apr 2, 2026

Choose a reason for hiding this comment

Uh oh!

dpol1 Apr 2, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

mvolikas commented Mar 29, 2026 •

edited

Loading

dpol1 commented Mar 31, 2026 •

edited

Loading