Monday, 9 July 2018

Upgrading to Eclipse Photon

I use Eclipse as my Java IDE. And the new release, Photon is now out.

Photon is a large release, with lots of new features. The most important is the separation of the test and main classpaths, which has always been a point of pain in the IDE. Now it just works as you would expect, and the Maven plugin M2E correctly sets it up:

Note the darker colour of the src/test classpath elements.

Support for Java 9 (modules) and Java 10 (local variable type inferenece) is also present, ready for Java 11 in September. You can also use JUnit 5. It even tries to help you reach 100% code coverage!

All in all, I feel this is a release where upgrading will make a difference to everyday coding.

I've upgraded my own Eclipse installations, and it all went pretty well. You can either start from a clean install (I prefer the basic IDE without plugins so I can choose which ones to add). Or you can add Photon as an update site, and let Eclipse update itself.

One problem I had was the plugin that connects Maven (M2E) to Checkstyle (Eclipse-CS), known as m2e-code-quality. Fortunately, the team at GEBIT have been maintaining a fork of the original plugin. However, they don't release it in binary form. As such, I had to build the plugin locally (no big deal - its a simple build).

To simplify the process however, I've created a repository on GitHub with my Eclipse setup files, and a binary zip of the GEBIT forked plugin.

To use just the m2e-code-quality GEBIT fork, download the zip file and add it as an update site. Here are some instructions.

Thank you Eclipse team for a great release!


PS. I won't be answering "how to" questions about upgrading Eclipse or the eclipse-setup repository. There are plenty of other places to ask questions, such as Stack Overflow or the Eclipse Forums.

Thursday, 22 March 2018

JPMS modules for library developers - negative benefits

Java 9 introduced a major new feature - JPMS, the Java Platform Module System. After six months I've come to the conclusion that JPMS currently offers "negative benefits" to open source library developers. Read on to understand why.

Modules for library developers

Java 8 is probably the most successful Java release ever. It is widely used and widely liked. As such, almost all open source libraries run on Java 8 (as library authors want their code to be used!). Some libraries with a long history also still run on older versions. Joda-Convert has a Java 6 baseline, while Joda-Time has a Java 5 baseline. Others have a Java 8 baseline, such as ThreeTen-Extra.

Java 9 was released in September 2017, but it is not a release that will be supported for a number of years. Instead, it had a lifetime of six months and is now obsolete because Java 10 is out. And in six months time Java 11 will be out making Java 10 obsolete, and so on.

While most releases last six months, some are luckier. Java 11 will be a "long term support" (LTS) release with security and bug support for a few years (Java 8 is also an LTS release). Thus, even though Java 10 is out, Java 8 is still the sensible Java version for open source library developers to target right now because it is the current LTS release.

But what happens when Java 11 comes out? Since Java 8 will be unsupported relatively soon after Java 11 is released, you'd think that the sensible baseline would be 11. Unfortunately I believe many companies will be sticking with Java 8 for a long time. An aggressive open source project might move quickly to a Java 11 baseline, but doing so would be a risky strategy for adoption.

The module-path

Before discussing the JPMS options for open source library developers, it is important to cover the distinction between the class-path and the module-path. The class-path that we all know and love is still present in Java 9+, and it mostly works in the same way.

The module-path is new. When a jar file is on the module-path any module-info is used to apply the new stricter JPMS rules. For example, a public method is no longer callable unless it has been exported from the module it is contained in (and required by the caller's module).

The basic idea is simple, you put old fashioned non-modular jar files on the class-path, while you put modular jar files on the module-path. Nothing enforces this however, and it turns out this is a bit of a problem. There are thus four possibilities:

  • modular jar on the module-path
  • modular jar on the class-path
  • classic non-modular jar on the module-path
  • classic non-modular jar on the class-path

To be sure your library works correctly, you need to test it both on the class-path and on the module-path. For example, service loading is very different on the module-path compared to the class-path. And some resource lookup methods also work completely differently.

To complicate this further, JPMS has no explicit support for testing. In order to test a module on the module-path (which is a tightly locked down set of packages) you have to use the --patch-module command line flag. This flag effectively extends the module, adding the testing packages into the same module as the classes under test. (If you only test the public API, you can do this without using patch-module, but in Maven you'd need a whole new project and pom.xml to achieve that approach, so its likely to be rare.)

In the latest Maven surefire plugin (v2.21.0 and later) the patch-module flag is used, but if your module has optional dependencies, or you have additional testing dependencies, you may have to manually add them, see this issue and this issue.

Given all this, what should an open source library developer do?

Option 1, do nothing

In most cases, but not all, code that is compiled on Java 8 or earlier will run just fine on the class-path in Java 9+. So, you can do nothing and ignore JPMS.

The problem is that other projects will depend on your library. By not adopting JPMS at all, you block those projects from progressing in their modularization. (A project can only choose to fully modularize once all of its dependencies are modularized.)

Of course if your code doesn't run on Java 9+ because you've used sun.misc.Unsafe or something else you shouldn't have done then you've got other things to fix.

And don't forget that a user could put your jar file on the class-path or the module-path. Have you tested both? ie. The truth is that "do nothing" is not possible - at a minimum you have extra testing to do, even if it just to document that your project does't work on the module-path.

Option 2, add a module name

Java 9+ recognises a new entry in the MANIFEST.MF file. The Automatic-Module-Name entry allows a jar file to declare what name it will use if/when it is turned into a proper modular jar file. Here is how you can use Maven to add it:

 <plugin>
  <groupId>org.apache.maven.plugins</groupId>
  <artifactId>maven-jar-plugin</artifactId>
  <configuration>
   <archive>
    <manifestEntries>
     <Automatic-Module-Name>org.foo.bar</Automatic-Module-Name>
    </manifestEntries>
   </archive>
  </configuration>
 </plugin>

This is a nice simple way to move forward. It reserves the module name and allows other projects that depend on your jar file to fully modularize if they wish.

But because its so simple, its easy to forget the testing aspect. Again, your jar file might be placed on the class-path or on the module-path, and the two can behave quite differently. In fact, now that it has some module information, tools may treat it differently.

When Maven sees an Automatic-Module-Name it will normally place the classes on the module-path instead of the class-path. This may have no effect, or it may show up a bug where your code works on the class-path but not on the module-path. With Maven right now, you have to use surefire plugin v2.20.1 to test on the class-path (an old version that doesn't know about the module-path). To test on the module-path, use v2.21.0. Swapping between these two versions is of course a manual process, see this issue for a request to improve this.

While upgrading some of my projects I added Automatic-Module-Name without testing on the module-path. When I did eventually test on the module-path the tests failed, as the code simply didn't work on the module-path. Unfortunately, I now have some releases on Maven-Central that have Automatic-Module-Name but don't work on the module-path, happy days...

To emphasise this, just adding something to the MANIFEST.MF file can have an effect on how the project is run and tested. You need to test on both the class-path and module-path.

Option 3, add module-info.java

This is the full modularization approach described in numerous web pages and tutorials on JPMS.

 module org.foo.bar {
   requires org.threeten.extra;
   exports org.foo.bar;
   exports org.foo.bar.util;
 }

So, what are the implications of doing this to the open source project?

Unlike option 2, your code now has a baseline of Java 9+. The Java 8 compiler won't understand the file. What we really want is a jar file that contains Java 8 class files, but with just the module-info.java file compiled under Java 9+. In theory, when running on Java 8 the module-info.class file will be ignored if it is not used.

Maven has a technique to achieve this. While the technique works OK, it turns out to be nowhere near sufficient to achieve the goal. To actually get a single jar file that works on both Java 8 and 9+ you need:

  • use the release flag on Java 9+ to build for Java 8
  • add an OSGi require capability filter to inform it that its still Java 8 compatible
  • exclude module-info.java from maven-javadoc-plugin when building on Java 8
  • use maven-javadoc-plugin v3.0.0-M1 (not later), manually copy dependencies to a directory and refer to them using additional Javadoc command line arguments, see this issue
  • exclude module-info.java from maven-checkstyle-plugin
  • exclude module-info.java from maven-pmd-plugin
  • manually swap the version of maven-surefire-plugin to test both the module-path and the class-path

And probably some more I've forgotten about. Here is one pom.xml before integrating Java 9. Here it is after integrating Java 9. An increase from 650 to 862 lines, with lots of complexity, profiles and workarounds.

With a Java 11 baseline, the project would be simpler again, but that baseline isn't going to happen for a number of years. Note that my comments should not be interpreted as anti-Maven. A small team there is working hard to do the best they can - JPMS is complex.

And just for kicks, your project can no longer be used by Android (as the team there seems to be very slow in adding a simple "ignore module-info" rule). And many tools with older versions of bytecode libraries like ASM will fail too - I had a report that a particular version of Tomcat/TomEE could not load the modular jar file. I've ended up having to release a "classic" non-modular jar file to cope with these situations, something which is profoundly depressing.

While I've added module-info.java to some of my projects, I cannot recommend others to do so - its a very painful and time-consuming process. The time to consider it would appear to be once Java 11 or beyond is widely adopted and the baseline of your project.

Negative benefits

Now for the controversial part.

It is my conclusion that JPMS, as currently designed, has "negative benefits" for open source libraries.

As explained above, the cost of full modularization is high for library developers. The need to retain Java 8 compatibility makes JPMS really hard to use (module information should have been textual, not a class file). The tooling is still incomplete/buggy. Many older projects can't cope with the new jar files if you do go for it. Much of this will improve over time, but we're talking a number of years before Java 11 is widely adopted. But don't be lulled into just believing waiting will solve the key problem.

The split (bifurcation) of the module-path from the class-path is an absolute nightmare. At a stroke, there are now two different ways that your library can be run, and the two environments have quite different qualities. Code that compiles and runs on the class-path will often not compile or not run on the module-path. And vice versa. As a library author, you cannot control whether the class-path or module-path is used. You have no choice - you must test both, which you probably won't think to do. (And Maven currently provides no way to test both in one pom.xml)

Given all this effort and extra complexity, we should be getting some great benefits, right? Well no.

JPMS is supposed to ensure reliable configuration (that all your dependencies are available at startup) and strong encapsulation (that other code can't see or use packages that you want to keep hidden). But since there is no way to stop your modular jar file being used on the class-path, you get none of these benefits.

Did you put lots of effort into choosing which packages to hide? Meaningless, as the user can just put the jar file on class-path and call your internal packages. Did you believe that the JVM will check all your dependent modules are available before starting? Afraid not, no checks performed when the user puts the jar file on class-path.

Since we get none of the claimed benefits of JPMS, but get lots of extra work in testing and complexity in the build tools, I feel "negative benefits" is a pretty accurate summary.

Summary

As of today, JPMS is a pain for library authors. The split of the module-path from the class-path is at the heart of the problem. You really can't afford to release a library without testing on both module-path and class-path - there are subtle corner cases where the environments differ.

What is desperately needed is a small change to JPMS. There needs to be a way for a library author to insist that the modular jar file they are producing can only be run on the module-path (with any attempt to use it on the class-path preventing application startup). This would eliminate the need for testing both class-path and module-path. Together with the passage of time, JPMS might yet achieve its goals and go from negative to positive benefits.

Monday, 5 February 2018

Java 9 has six weeks to live

Java 9 is obsolete in just six weeks (20th March 2018). What? You haven't upgraded yet? Well, Java 10 is only going to last six months before it is obsolete too.

Update 2018-03-20: Java 10 is released. Java 9 is obsolete.

Release train impact

The new Java release train means that there will be a new release of Java every six months. And when the next release comes out, the previous release is obsolete.

What do I mean by obsolete?

In practical terms it means that there are no more security updates from Oracle. (Theoretically, the OpenJDK community could release security updates, but there is no sign of this yet). And since you don't want to run your software without the latest security updates, you are expected to upgrade to Java 10 as soon as it is released.

As a user of Java, here are three possible ways to approach the release train:

  1. Stay on Java 8, the current LTS (long term support) release, until the next LTS release occurs (Java 11)
  2. Move from Java 9 to Java 10 to Java 11, making sure you update rapidly to get the security updates
  3. Stay on Java 9 (or Java 10) and don't worry about security updates

If you have already moved to Java 9, you have effectively committed to option 2 or 3. If you care about security updates, you need to be prepared to switch to Java 10 shortly after it is release on 20th March. To do this, you probably should be testing with a Java 10 pre-release now. If you find that to be a challenge, you have to stop caring about security, or consider going back to Java 8 LTS.

However you look at it, being on the release train is a big commitment.

  • Will your dependencies work on the next version?
  • Will your IDE be ready?
  • Will your build tool (Maven, Gradle etc.) be ready?
  • Will your other tools (spotbugs, checkstyle, PMD etc.) be ready?
  • How fast are you going to be able to update when the release you are on is obsolete?

Lots to consider. And given the number of external tools/dependencies to consider, I think its fair to say that its a bold choice to use Java 9 or 10.

Summary

With a release every six months, it is important to decide on an approach to the release train. If you want to upgrade every six months, great! But you'll need to test pre-releases of Java with your whole toolchain in advance of the release to ensure you don't get stuck on an unpatched obsolete release of Java.