Maven Dependency Hell

maven

https://maven.apache.org/guides/introduction/introduction-to-dependency-mechanism.html
https://carlosbecker.com/posts/maven-dependency-hell/ - done reading
https://stackoverflow.com/questions/33907162/systematic-approach-with-maven-to-deal-with-dependency-hell - done reading
http://cupofjava.de/blog/2013/02/01/fight-dependency-hell-in-maven/
https://coderwall.com/p/tbxihg/fight-dependency-hell-in-maven
https://en.wikipedia.org/wiki/Dependency_hell
http://martin-toshev.com/index.php/software-engineering/tips-and-tricks/82-solving-the-jar-hell-problem-in-maven
https://github.com/dsyer/dependency-hell
https://www.reddit.com/r/programming/comments/176o53/why_everyone_eventually_hates_or_leaves_maven/
https://olemortenamundsen.wordpress.com/2008/05/22/maven-dependency-hell/
https://maven.apache.org/guides/introduction/introduction-to-dependency-mechanism.html
http://www.tutorialspoint.com/maven/maven_manage_dependencies.htm - printed
http://www.kyleblaney.com/maven-best-practices/ - printed

https://github.com/basepom/duplicate-finder-maven-plugin/wiki
The duplicate-finder-plugin is mostly useful for the detection of "shaded jars" - it checks SHA sum of class files and reports difference
Many projects provide a BOM POM to avoid ${spring.version} everywhere
Use Maven Helper plugin to easily resolve all conflict by excluding old versions of dependencies.
Maven Helper plugin really helps, especially for those who prefer GUI over CLI. I am not quite sure that excluding old versions of dependencies is the best way to go in long term. If I understand it correctly, it can be useful in case that some transitive dependencies are unused.
You can either fork and start to maintain more things, or exclude old dependencies, or do some magic so that you can have multiple versions.

https://maven.apache.org/plugins/maven-shade-plugin
http://github.com/basepom/duplicate-finder-maven-plugin/wiki

I am not sure how to get the list of duplicated classes from the maven.apache.org/plugins/maven-shade-plugin On the other hand, the duplicate-finder-plugin github.com/basepom/duplicate-finder-maven-plugin/wiki does this job quite well.

How can we avoid or get rid of dependency hell?

Use maven-depedency-plugin to find the obvious and easy stuff to fix:

mvn dependency:analyze-duplicate

Just remove what’s duplicated.

mvn dependency:analyze

This will take a while and may warn you about a lot of depedency errors. You will be able to remove dependencies that are not used anymore and add dependencies that are directly used. This might show some false-positives.

Multiple versions of the same depedency. This is the hard because maven doesn’t have a plugin to find them easily. So, I used a little bit of bash trickery to do the job for me:

mvn dependency:list -Dsort=true | # list all deps
  grep "^\[INFO\]    " |          # grep for the deps list only
  awk '{print $2}' |              # remove the INFO prefix
  cut -f1-4 -d: |                 # removes the dep scope
  sort |                          # sort (duh)
  uniq |                          # remove duplicates
  cut -f1-3 -d: |                 # removes the version
  uniq -c |                       # count line groups
  grep -v '^ *1 '                 # grep groups that repeat

You’ll end up with a list like this:

    2 com.foo:foo-modules-commons:jar
    3 com.fasterxml.jackson.core:jackson-annotations:jar
    3 com.fasterxml.jackson.core:jackson-core:jar
    2 com.fasterxml.jackson.core:jackson-databind:jar
    2 com.squareup.okio:okio:jar
    2 commons-collections:commons-collections:jar
    2 commons-fileupload:commons-fileupload:jar
    2 joda-time:joda-time:jar
    2 org.mapstruct:mapstruct:jar

Now, for each of these deps, you’ll have to run:

mvn dependency:tree -Dincludes=DEP

For example:

mvn dependency:tree -Dincludes=org.mapstruct:mapstruct

This will show you all the places this dependency is being used, so all you need to do is fix it.

Duplicated classes. This is something that I have seen a lot in Apache Commons libraries. As far as I know, the easiest way to find these problems is by using Jboss’s Tattletale Maven Plugin. While the plugin seems to be abandoned, it still works. Just add it to your parent pom and run it to get the reports. These kind of problems may require manually removing .class files from jars, excluding dependencies or, sometimes, just ignoring some of them.

Real solutions:

1. Run maven-dependency-plugin and tattletale-maven in the build for every pull request to block problematic changes. You can tweak the script to find dependencies with multiple versions to exit 1 in thoses cases, and just add it to the build too.

What are different dependency hell scenarios?

1. Multiple versions of the same depedency. Developer may not be aware that a different version of the same dependency is already being used, or maybe we need a specific functionality in this particular version, but we are not ready to upgrade the entire application (or other applications) to use the newer version yet. This can also be caused by the fact that we use multiple open-sourced libraries, and they themselves may depends on different versions.
2. Duplicated classes. Perhaps this is caused by using open-sourced libraries. If a library need a class (but not everything else) from another library but instead of declaring the other library as a dependency, this library may just take the one class from the other library and bundle it for its own release / distribution.

What are different reasons why dependency hell happen?

1. Multiple versions of the same depedency. Developer may not be aware that a different version of the same dependency is already being used, or maybe we need a specific functionality in this particular version, but we are not ready to upgrade the entire application (or other applications) to use the newer version yet.
2. Duplicated classes. Perhaps this is caused by using open-sourced libraries. If a library need a class (but not everything else) from another library but instead of declaring the other library as a dependency, this library may just take the one class from the other library and bundle it for its own release / distribution.

What are the solutions for dealing with dependency hell?

Dealing with dependency hell is harder after the fact. Preventing is the best option. To prevent it from happening:

1. Run maven-dependency-plugin and tattletale-maven in the build for every pull request to block problematic changes. You can tweak the script to find dependencies with multiple versions to exit 1 in thoses cases, and just add it to the build too. Push back to the development team. Just do not let it happens in the first place. Either upgrade entire application, or other applications to use the new version, or implement our own code that provide the same feature as the new version if that is possible.
2. Duplicated classes. The easiest way to find these problems is by using Jboss’s Tattletale Maven Plugin. While the plugin seems to be abandoned, it still works. Just add it to your parent pom and run it to get the reports. These kind of problems may require manually removing .class files from jars, excluding dependencies or, sometimes, just ignoring some of them.
3. This can also be caused by the fact that we use multiple open-sourced libraries, and they themselves may depends on different versions. We may have to find versions of these libraries that depends on the same version.
4. Can we use different class loader? For example, for this application, use a class loader that load the new version?