Skip to content

[Improvement] Shade Jetty in the Solr module to avoid classpath conflicts with Storm #1859

@dpol1

Description

@dpol1

What would you like to be improved?

Context

As discovered during the Solr 10 migration (#1833), the Solr module's Jetty dependencies conflict with the Jetty version shipped by Apache Storm at runtime.

Solr 10 (via solrj) depends on Jetty 12.0.27, while Storm 2.8.5 ships Jetty 12.1.6. Because both end up on the same classpath, the version mismatch causes NoSuchMethodError at runtime:

18:50:11.431 [Thread-37-status-executor[5, 5]] ERROR o.a.s.u.Utils - Async loop died!
java.lang.NoSuchMethodError: 'boolean org.eclipse.jetty.http2.client.HTTP2Client.installBean(java.lang.Object)'
        at org.eclipse.jetty.http2.client.HTTP2Client.<init>(HTTP2Client.java:133)
        at org.apache.solr.client.solrj.jetty.HttpJettySolrClient.createHttpClient(HttpJettySolrClient.java:292)
        at org.apache.solr.client.solrj.jetty.HttpJettySolrClient.<init>(HttpJettySolrClient.java:152)
        at org.apache.solr.client.solrj.jetty.HttpJettySolrClient$Builder.build(HttpJettySolrClient.java:987)
        at org.apache.stormcrawler.solr.SolrConnection.getConnection(SolrConnection.java:364)
        at org.apache.stormcrawler.solr.persistence.StatusUpdaterBolt.prepare(StatusUpdaterBolt.java:89)
        at org.apache.storm.executor.bolt.BoltExecutor.init(BoltExecutor.java:128)
        at org.apache.storm.executor.bolt.BoltExecutor.call(BoltExecutor.java:138)
        at org.apache.storm.executor.bolt.BoltExecutor.call(BoltExecutor.java:54)
        at org.apache.storm.utils.Utils$1.run(Utils.java:393)
        at java.base/java.lang.Thread.run(Thread.java:840)

How should we improve?

Proposed Solution

Use the Maven Shade plugin to relocate the Solr client's Jetty dependencies under a StormCrawler-specific package (e.g. org.apache.stormcrawler.solr.shaded.jetty).

This way Storm's Jetty and Solr's Jetty coexist on the classpath without interfering with each other, regardless of what either project ships.

A similar approach is used in Apache Tomee: tomee/deps/commons-dbcp2-shade/pom.xml.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions