Sun, Apache & IP - in pictures!

This blog explains the JCP IP process in pictures. And how Sun have managed to use it to block Apache Harmony.

IP in the JCP - how it should work

Within the expert group, a number of experts make contributions. Each member agrees, as part of the JSPA agreement, to grant their IP to the spec Lead. The spec lead will also add IP in the same way. This produces a 'pot' of collective IP owned by the spec lead on behalf of the expert group.

The expert group produces two outputs in addition to the IP - the specification itself and the testing kit.

An implementor of the specification uses the specification document (which may well include Javadocs) to develop the implementation. The implementor will probably have their own unit tests at this stage.

When the implementor feels that the development is nearly complete, they request the testing kit (TCK) from the spec lead. They then use that to test the implementation (and fix any bugs).

Once the testing kit has been passed, there is another stage that isn't found in a typical project. This is the stage of 'obtaining the collective IP' from the spec lead. This should be automatic if the testing kit is passed and the implementation is complete and valid.

Finally, the implementation can be released. It can now claim to be a compatible implementation of the JSR. And the collective IP has been granted to avoid any unpleasant surprises down the road for users.

What about Apache Harmony?

Well, its important to realise that Sun didn't expect that Harmony would succeed. They thought that the task was too big. As such, they assumed that they would never need to worry about providing a Java SE testing kit, even though they'd agreed to do so.

Instead of keeping to the agreement they had signed, Sun used its lawyers to come up with a cunning way of blocking Harmony:

What Sun did was add a 'Field of Use' clause (FOU) to the license for the testing kit. This clause infected the tested code, and thereby infected the release.

The precise details of the FOU are confidential at the behest of Sun, however in summary they state (rather absurdly) that the tested code cannot be run on a PC in an enclosed environment. Thus, you could run a tested version of Harmony on your PC providing it is running on your desktop. But if you pick the machine up and place it in an enclosed cabinet, such as inside an X-Ray machine, or an shopping mall information kiosk, then you would be breaking the FOU clause.

Sounds absurd? Well it is yes.

But remember, Sun didn't need something sensible. They just needed something, anything, to stop Harmony.

By adding this clause, it meant that the tested code would need to contain an additional clause over and above the standard Apache license. A clause that would be invalid for any open source software as defined by the OSI. To be clear, this FOU clause would be an issue for any open source group trying to implement the Java SE specification.

Since Apache only releases open source licensed code, Sun could therefore effectively block the release of Harmony. And in the meantime, it could spend all its efforts fooling everyone that Java is an open standard using the Open JDK initiative - an open implementation, not an open standard.

Summary

Sometimes a picture tells the story clearer than text. Hopefully, these pictures will help illuminate you. If not, you can always read the legal agreement.

A question of IP

My post about the lack of an open specification for Java SE 7 raised some eyebrows yesterday. One point I wanted to clarify in a little more detail is how the JCP treats IP.

Warning! I am not a lawyer. This is written to the best of my knowledge, but I may well have something messed up. Feel free to tell me I'm wrong, but please provide a link explaining why!

The JCP and IP

IP (Intellectual Property) is one of the aspects which any good specification needs to define. For example, here are the relevant terms for the W3C. Put simply, the IP, and associated, restrictions describe what rights you are granted.

For the JCP (Java Community Process), the relevant document is the JSPA. I'm going to try and outline some key sections here.

4. Intellectual Property; In-Bound.
A. Contributions to the Spec Lead. You hereby grant to each Spec Lead ... with respect to the Output of the JSR ... a perpetual, non-exclusive, worldwide, royalty-free, fully paid-up ... irrevocable, license, with the right to sublicense:
- Copyrights and Trade Secrets ....
- Patents ....
B. Grants to Other Expert Group Members. You hereby grant to Member represented on any Expert Group ... Your applicable patents, copyrights and trade secret rights which You currently have or may acquire in the future a perpetual, non-exclusive, worldwide, royalty free, fully paid-up, irrevocable license to use Your Contributions for research and development purposes related to the activities of such Expert Group. Similarly, Sun makes the same grant to You with respect to its Contributions concerning Expert Groups on which You are represented. These grants, from You to other Members and from Sun to You, shall not include distribution to third parties of early access implementations of the Spec under development by Your Expert Group until the draft Spec has been released for Public Review.
C. Contributions from Sun and Ex-Spec Leads. For Contributions from Sun under JSRs for which You are the Spec Lead, Sun hereby grants to You the same license with respect to Sun’s Contributions ...

This refers to expert group members contributing to the JSR. I've snipped quite a bit to avoid too much legality. The key elements are:

• the members of each JSR license their IP (copyrights, trade secrets, patents) primarily to the spec lead
• the same grant is also made to other JSR expert group members, for R&D prior to the JSR being finalised
• the granted rights cannot be used publicly until the JSR is in public review
• Sun grants the same

One thing to note is that Sun's participation is explicit in the JSPA. Sun is not "just another participant" in the JCP - it has special rights.

The next section is an interesting revoke section:

D. Withdrawal of Contributions due to Change in Announced License Terms. If the Spec Lead for an Expert Group in which You are participating makes significant changes to the terms and conditions of any license granted pursuant to Sections 5.B or 5.F below after those terms and conditions are first disclosed pursuant to the Process, then You may, upon giving notice to the Specification Lead and the PMO, withdraw any copyright or patent licenses concerning Your Contributions granted pursuant to Section 4.A.I or 4.A.II above (but not patent licenses granted by You pursuant to Section 6 below).

Put simply, if any spec lead breaks the license terms that they have already agreed to, then the granted IP from other expert group members can be revoked.

The next section is about what happens when the spec is published:

Intellectual Property; Out-Bound.
A. Ownership; Obligation to Publish Spec.
... the Spec Lead(s) for a particular JSR at the time of the final release of the Specification shall own the copyright to the final Spec generated pursuant to that JSR under United States copyright law. Promptly after its completion ... such Spec will be published by the PMO at the JCP Web Site.

For example, for the Java SE specs, Sun owns the copyright, as it is the spec lead.

B. License to Create Independent Implementations.
... the Spec Lead for such JSR shall offer to grant a perpetual, non-exclusive, worldwide, fully paid-up, royalty free, irrevocable license under its licensable copyrights in and patent claims covering the Specification (including rights licensed to the Spec Lead pursuant to Section 4.A and 4.C) to anyone who wishes to create and/or distribute an Independent Implementation of the Spec. Such license will authorize the creation and distribution of Independent Implementations provided such Independent Implementations:
(a) fully implement the Spec(s) including all its required interfaces and functionality;
(b) do not modify, subset, superset or otherwise extend the Licensor Name Space, or include any public or protected packages, classes, Java interfaces, fields or methods within the Licensor Name Space other than those required/authorized by the Spec or Specs being implemented; and
(c) pass the TCK for such Spec.
...

Here is the key part! The spec lead grants the IP to everyone that implements the spec. But only if they meet the three criteria.

• the implementation must fully implement the spec
• the implementation must not modify the spec
• the implementation must pass the testing kit

So, everyone grants their IP to the spec lead. And then the spec lead grants that IP to the implementor once the three conditions are met. Of course, to meet the three conditions, you have to have access to the testing kit...

Since Apache Harmony is prevented from passing the Java SE specification testing kit, it cannot meet the three criteria. Since it cannot meet the three criteria, it cannot receive the IP grants of copyrights, trade secrets and patents.

And as a not-for-profit, there is no way for Apache to take any legal risks. It certainly can't just assume that there is no IP to be granted (there almost certainly is).

A section on licensing of the testing kit:

F. Licensing of RI and TCK.
I. The Spec Lead (including Sun) for a JSR approved by the JCP shall offer to any interested party licenses concerning the RI and TCK -- and also the TCK separately when developed under any JSR submitted hereafter -- on terms and conditions that are non-discriminatory, fair and reasonable. Except as otherwise set forth in this Section 5.F, such terms and conditions shall be determined by the Spec Lead in its reasonable discretion.

So, the testing kit (TCK) has to be licensed in a "non-discriminatory, fair and reasonable" manner.

III. With respect to the TCK when licensed separately from the RI, for a Qualified Not-for- Profit or Qualified Individual there shall be no charge for the license offered by the Spec Lead.

The JSPA is clear that "Qualified Not-for-Profit" groups, like Apache, are fully entitled to the testing kit free of charge. Sun haven't disputed that.

V. No provision of this Section 5.F shall be understood to require that the Spec Lead license the RI (or TCK) in source code form...

The JSPA is clear that the testing kit does not need to be open source, or even source code available. Thats not in question by Apache or Sun.

Finally, a section on trademarks:

8. Use of Trademarks. Subject to any other rights and obligations ... You may refer to Sun's Java technology or programming language to the same extent as the general public, provided that such reference is not misleading or likely to cause confusion....

Thus, the JSPA grants no special privileges with respect to trademarks. Apache and Sun agree on this. Apache isn't seeking to use the Java trademark.

Another (simpler) summary is the original Apache FAQ on the JSPA and associated issues. Take a read if you're still confused.

Summary

All legal agreements are complex, and this is no exception. But the principle is simple.

Each Expert Group member licenses their IP (copyrights, trade secrets and patents) to the spec lead. The spec lead then licenses that 'collective IP' to anyone that successfully implements the specification.

In the case of Apache Harmony, because Sun is preventing Apache from receiving the testing kit under an appropriate license (see the Apache FAQ), it is impossible for Apache to verify that the Java SE specification has been met. As such, Sun cannot and will not provide the required IP grant to Apache, and thus Apache cannot release Harmony.

As I said yesterday, for Java 7, the easy solution for Sun may well be to simply not have an open Java SE 7 specification.

No more Java 7

"No more Java 7". Yes, I know its a strong statement, but I don't mean it in the way you probably think.

Update: I've followed up this blog with a whole series of articles:
* a description of how IP works in the JCP (pretty picture!)
* a deep dive into the behind-the-scenes world of the JCP to provide a clearer explanation of the lack of a Java SE 7 specification and how this affects the whole Java industry, not just Apache and Sun
* more info from the ASF board meeting minutes
* an exploration of how the JCP doesn't really exist
* a take on the Oracle-Sun deal

What is Java?

Well, the term 'Java' is a complex and overloaded one, including:

• the stock symbol of Sun Microsystems
• the trademark of Sun Microsystems
• the term for a raft of technologies based around the JVM
• a platform (Java SE) based around the Java programming language (a programming language)
• the specification for the Java SE platform (the specification for the programming language)

This blog post is about the last of these - the Java SE platform specification.

Back in 1997 Sun talked about making the Java programming language into a formal open standard. At that time, the ECMA was the standards body approached. However, Sun withdrew from the standards process and created the Java Community Process instead. Everyone breathed a little easier, as Java now appeared to be an effectively open standard.

The job of the JCP is to "... develop and revise the Java technology specifications, reference implementations, and test suites, the Java Community Process (JCP) program has fostered the evolution of the Java platform in cooperation with the international Java developer community."

Over the years, and with a few hiccups, the process hasn't gone too badly. There are multiple implementations of many of the JSRs, creating a healthy eco-system of competition and improvement. That is after all a key goal of any standard - to define what is necessary to produce a conforming implementation.

The JCP is, in fact, an unusual standards body in that it goes further than most to ensure that every specification must be tested before it is officially allowed to be released as an implementation of the specification. Each JSR (the individual standard) must provide an appropriate specification document, reference implementation and, crucially, a testing kit.

One group that has implemented many JSR specification is Apache. From Tomcat to Geronimo to JMS to the whole of Java EE. (And back in 2002, Apache had to fight to allow any Open Source implementations of JSR specifications.)

So, what has this to do with the end of Java SE 7? Well, as I said, I'm specifically referring to Java the specification. Now many people don't realise this, but as well as the JSRs for Servlets, EJBs and JMS, there are also JSRs for the whole of Java EE and for the whole of Java SE.

The relevant specifications are JSR-176 for Java SE 5 and JSR-270 for Java SE 6. These specifications are no different to any other JSR. The specification documents exist (mostly Javadoc). The testing kit exists. As does the reference implementation (the Sun JDK).

So, is it an open specification and open standard, as promised back in 1998?

Well, the Java SE specification has been implemented by other companies - IBM has a JDK amongst others. However, IBM pay money to Sun for the rights to do this. The question is whether Apache can implement it, just like they implement 25 other JSRs.

Apache's implementation of the Java SE 5 JSR specification is Apache Harmony. However, when Apache came to obtain the testing kit for the specification a whole political game started. Instead of Sun offering the testing kit as normal for each of the other 25 JSRs, they offered a testing kit where the results of the tested code would not be Open Source

Obviously, Apache couldn't accept this limitation, which is against the legal agreement signed between Sun and Apache. Apache complained 2 years ago, but has yet to receive an acceptable response. And no, there is no way a charity not-for-profit like Apache is going to sue a multi-national corporation - who do you think would get the better lawyers?

The key point is that Sun's tactics here were quite deliberate. They knew exactly what they were doing by offering a testing kit with a restrictive license. They wanted to ensure that Apache Harmony couldn't be certified as complete. Sun was going out of its way to ensure that there was no competition to its own JDK.

In the meantime, Sun played the OpenJDK card. It announced the release of the JDK under the GPL license.

Lots in the community (notably those from a GPL background) suddenly became best friends with Sun. Some joined Sun, some now work closely with Sun. Many of these people now believe that there is no danger around Java, and that Richard Stallman's Java trap is over. I think this shows a gross lack of perspective - the code may now be GPL open source, but the specification is now no longer open. Which is more important?

If you can't freely implement a specification*, then I'd argue its not worth the paper its written on

* There are two ways you can implement the Java specification - (1) pay Sun money, or (2) ensure your code is "mostly derived from the OpenJDK".

What about Java SE 7? Everyone isn't noticing the obvious

I started with a contentious statement - that there may not be a Java SE 7 (open specification). So, lets look at the evidence.

Blogs, blogs and more blogs - Every blog from a key Sun employee in recent times has referred to JDK 7, not Java 7. Read them carefully!!! Still think there is guaranteed to be a Java 7?

To emphasise the point - JDK 7 is the implementation, Java SE 7 is the specification. But everyone at Sun is talking about the JDK (Open JDK), not the specification (Java) and this isn't an accident.

Project Jigsaw - Modularisation of the core of Java - No JSR planned. Funny, how such a major piece of infrastructure has managed to slide by without becoming an open specification...

To be fair, Mark Reinhold's recent blog is more positive - "The JDK 7 Project is creating a prototype of what might-or might not-wind up in the Java SE 7 Platform Specification. When the SE 7 Platform JSR is submitted then the features under development in JDK 7 will be proposed for inclusion therein, except for those that are VM-level or implementation-specific." (He uses the 'when the JSR is submitted phrase', but note the 'implementation-specific' phrase which conveniently covers Jigsaw at a minimum).

However, unless anyone is being particularly naive, it should be clear that if there is to be a Java SE 7 specification then it would be fully defined by what is in Open JDK, not by the JSR process. Do you think that Sun would remove things already in OpenJDK were the JCP to so decide? Er, no.

And look at the timeline. The full JDK contents are defined by September, yet there is no JSR for Java SE 7 yet. The first two milestones have already happened for goodness sake!!!

If the JSR happens it will be a rubber stamp. So why bother?

Next release - JDK 7, not Java 7

Yes, this is what I mean by the title of this blog. The evidence I see, and have shown here, is that the next release will be JDK 7, not Java 7. ie. there won't be an open specification.

Of course, since Sun own the Java trademark, Sun puts in all the investment and Sun has all the mindshare, the vast majority of developers won't notice this. In fact, since its now Open Source as the OpenJDK they might even think all is well. Sun's strategy to quosh all competition is working well isn't it?

What won't be realised is how 10 years of open standards have been lost. What won't be realised is how everyone is now locked into a new Java trap - a single vendor of anything that can truly be certified as Java. Its just a trap that Richard Stallman isn't interested in arguing against.

Personally, I would have thought that given all our experiences with the benefits of open Standards on the internet we would be demanding it of Java too - especially since it was promised back in 1997. We need to fight to retain Java as an open standard.

Summary

All the evidence I see is that Java SE is no longer an open standard and the next release will be JDK 7, not Java 7. That is something that should matter to all of us who care about the Java eco-system.

After all, if Sun can get away with making Java SE not an open standard, then why not Java EE, or Servlets or JMS? Its time to stand up and be counted and demand our open standard back!

Feel free to tell me why you think my analysis is wrong... I suspect you won't hold back...

For the record, I'm a member of the Apache Software Foundation, speaking personally here to point out something someone else really should have noticed...

Update 1: Please tag any follow up blogs or tweets with 'nojava7' so the feedback can be seen!

Update 2: For info, I've never contributed to either Apache Harmony or the Open JDK, although I have used the OpenJDK in Kijaro and work on JSR-310. As such, I'm pretty dispassionate about the quality or usefulness of Harmony/OpenJDK. I care more about the lack of an open specification for Java 7, which is what this blog is primarily about.

Update 3: I haven't seen any evidence that the engineering team at Sun don't want a Java SE specification. The evidence I see is that this is driven from a higher business/management level.

Update 4: I've clarified the initial 'meanings of Java' section to place the emphasis on the Java Platform, as explained here. (The licensing terms are well worth a read - particularly number 7)

Update 5: I've updated Java 7 to Java SE 7 in a few places to aid clarity. Most people probably understand that Java EE and Java ME are different, but it helped in a couple of places to be extra sure.

Update 6: Fixed "offering Sun the testing kit" to "Sun offering the testing kit"

Speaking on Fantom tonight in London

Quick note to say I'm speaking on the Fan Fantom programming language tonight in London. The evening will also include a brief intro to OSGi.

Its at the JavaWUG user group, hosted by Skills Matter, registration required to attend.

Renaming the Fan programming language?

I've recently started a poll to discuss the possibility of renaming the Fan programming language. The reason for a possible name change is simple - you can't google "Fan" and get hits about the language, and you never will (unlike Java).

If you're interested in this new language at all, please take the poll. We're also gathering alternative names, which you can leave in comments here if you prefer.

JDK 7 language changes - Everyone votes!

A third vote on language changes has just completed, this time it was online. I'll summarise the results from the three votes in this post.

Three votes

The three votes in question were:

• Devoxx
• JavaEdge
• Online

At Devoxx, the topics voted on were:

• Properties
• Multi-catch of exceptions
• Null handling
• List/Map access using []
• Extension methods
• Method pointers
• Multi line strings
• Infer generics on the RHS

At JavaEdge and Online the features voted on were:

• For-each loop over a Map
• Control over for-each loops (index/remove)
• List/Map access using []
• Infer generics on the RHS
• Multi-catch of exceptions
• String switch
• String interpolation
• Multi line strings
• Resource management
• Null handling

The later polls excluded properties as they are specifically excluded for Java 7. They also excluded extension methods and method pointers as these were the least popular options at Devoxx.

Here are the first preference votes for Devoxx, JavaEdge (ranked), JavaEdge, (not-ranked) and Online:

Considering the first to fourth preference votes (first at the bottom in dark green):

Totals

Unfortunately, we can't add the Devoxx figures directly to the rest as the data is incomplete. However, it is useful to add all JavaEdge and Online votes together:

So, what do these results tell us? As always, absolute answers are difficult, because not all voters followed the same rules (online voters clearly didn't rank properly), and Devoxx was asked a different question. However, some things can be established, based not only on total votes but also how the preferences went from 1 to 10.

• Devoxx voters favoured Infer-generics, Multi-catch and Null-handling.
• JavaEdge voters favoured Null-handling, String-switch and Multi-catch.
• Online voters favoured Multi-catch, Null-handling, Infer-generics and String-switch.

So, these four options polled consistently highly in each of the different votes (except Infer-generics which did badly at JavaEdge). This now lends quite significant weight to the question of 'what does the community want' and 'what pain points need addressing'. The small language changes for Java 7 project now has some solid data to work from.

My take is that Multi-catch looks like a strong favourite for inclusion, as does String-switch. Infer-generics had a slightly more mixed reception in JavaEdge, but is popular elsewhere.

The big question is over Null-handling. Three separate votes has ranked it highly, with it taking the highest number of first preference votes in three of the four measured graphs above. There is a clear desire for change and a clear proposal. I hope the opportunity is not missed.

Summary

As more votes stack up, the statistics become more secure. We can now say that out of the ten polled items, the top four are Null-handling, Multi-catch, String-switch and Infer-generics.

Any feedback or thoughts are welcome!

Java 7 - Null-default and Null-safe operators

The most popular small language change request is better handling of nulls. As a result, I've updated my null-handling proposal.

Enhanced null-handling

The votes from Devoxx and JavaEdge were clear. Ordinary developers find handling nulls to be a pain and they would like to see language change to address it. At JavaEdge in particular almost a third of the first preferences and two thirds of the top four preferences went to null-handling.

I blogged my original thoughts two years ago. The updated null-default and null-safe invocation proposal is now available (v0.2).

The proposal covers two new operators, which follow Groovy for syntax (and thus consistency).

The null-default operator ?: returns the LHS unless that is null in which case it returns the RHS:

// today
  String str = getStringMayBeNull();
  str = (str != null ? str : "");
  
  // with null-default operator
  String str = getStringMayBeNull() ?: "";

The null-default operator also works particularly well with auto-unboxing.

Integer value = getIntegerMayBeNull();
  int val = value ?: -1;

In fact, I hope that tools (IDEs and static analysis) would combine to effectively make the null-default operator mandatory when unsafe unboxing is occuring.

The null-safe operator ?. is an alternative form of method/field invocation. If the LHS is null, then the result of the whole method/field invocation is null:

// today
  String result = null;
  Foo foo = getFooMayBeNull();
  if (foo != null) {
    Bar bar = foo.getBarMayBeNull();
    if (bar != null) {
      result = bar.getResult();
    }
  }
  
  // with null-safe operator
  String result = getFooMayBeNull()?.getBarMayBeNull()?.getResult();

There is an interesting case if the last segment of the field/method expression returns a primitive. However, this can be handled by combining the two new operators (hence why its one proposal):

// today
  int dotIndex = -1;
  if (str != null) {
    dotIndex = str.indexOf(".");
  }
  
  // with null-safe and null-default operators
  int dotIndex = str?.indexOf(".") ?: -1;

I considered mandating this when a primitive occurs, but it would have undesirable side effects, so this is one best left to the tools to handle.

More details on all of this in the proposal.

Summary

Nulls remain a pain in Java. There simply isn't enough language support for this most common task. Every day, developers write logic to check and handle nulls and this obscures the meaning of the real code. And every day production systems go down due to NullPointerExceptions that weren't caught. This proposal provides a simple enhancement to Java that tackles the heart of the null issue.

Opinions welcome.