Saturday, 28 March 2009

Shedding new light on No Java SE 7 JSR

There is a direct connection between the 'Sun vs Apache Harmony' dispute and the lack of a Java SE 7 platform JSR. Using newly available evidence I hope to shed new light on what that link is.

Back in the day

Its interesting to read again how Harmony was welcomed when it first started:

Part of the reason these Apache projects are possible is that the JCP has been working over the last few years to clarify the role of open source projects and also to improve overall transparency. For example there were changes to the intellectual property rules in JCP 2.5 specifically designed to allow open source implementations. There have also been a number of transparency improvements in JCP 2.6 to allow earlier access to specification information. In addition to those JCP updates, Sun created a scholarship program under which we have provided Apache (and other not-for-profit groups) with free access and free support for Sun's Java compatibility test suites.
...
The licensing rules for J2SE 5.0 were carefully designed to allow independent, compatible open-source implementations of the J2SE specification. Personally, I am not entirely sure if the world really needs a second J2SE implementation, but at the same time I am also glad to see that all the effort we put into getting the rules and the licensing issues straightened out is actually proving useful!

Graham Hamilton's blog VP and Fellow at Sun responsible for Java at the time, written May 7, 2005.
(My emphasis)

And lets look at how that was reiterated in the business terms for Java SE 6:

JSR 270: JavaTM SE 6 Release Contents
...
2006.10.24:
2.18 Please provide a description of the business terms for the Specification, RI and TCK that will apply when this JSR is final.
...
7. Nothing in the licensing terms will prevent open source projects from creating and distributing their own compatible open source implementations of Java SE 6, using standard open source licenses. (Yes, you can create your own open source implementation of Java SE 6 if you really want to. But we're also doing everything we can to make it easy for you to use the original RI sources! See http://jdk6.dev.java.net.)

JSR-270
(My emphasis)

How times change. Update: The 'how times change' reference was a comparison of both of the above quotes to the dispute that followed, not a comparison of the two quotes. Sorry for the confusion.

Trying to resolve the dispute

As is clear, Apache claim that Sun is in violation of the JSPA contract for both Java SE 5 and Java SE 6 platform specifications. The comments above also indicate how matters changed over the course of time.

Given the dispute, it is clear that Apache would object to there being a Java SE 7 platform JSR without changes being made to resolve the issue. However, Apache is just one voice in the industry. The power in the JCP is held in two 16 member Executive Committes - one for Java SE/EE and one for Java ME. Clearly, therefore, Apache could be outvoted 15-1 on the SE/EE committee, and it has no vote on the ME committee.

So, why hasn't Sun just submitted the Java SE 7 platform JSR and accepted/ignored Apache's No vote?

Well, until recently, I thought that the reasons were all private information. However, it turns out that since September 2008, the Executive Committee meeting minutes have been openly published. As such, we can see how the Executive Committees have tried to resolve the issue. As far as I know, no journalist or blog has yet picked up on this.

Given the controversial nature of this, I strongly encourage anyone following this to read the minutes themselves. Its a good insight into the the way in which the JCP operates (a style probably typical of any commercial based standards body). In this section I will extract some key elements, and make some minimal comment, but please read the full documents to make up your own mind.

Meeting summary - February 27th 2007
Geir Magnusson presented on a hypothetical situation regarding a company pursuing a certain Java-related business opportunity and whether the JSPA allows licenses that could affect that opportunity. During the meeting Geir asked for a straw poll on the following statement:
"The Executive Committee of the JCP supports the Apache Software Foundation's assertion that placing limitations on field of use for compatible implementations of Java specifications is contrary to the letter of the JSPA and the spirit of the JCP as a creator of open specifications."
Voting "yes": 17
Voting "no": 2
Voting "abstain": 3

(Geir Magnusson is the Apache representative on the Executive Committee)
(Note that the official document status is draft and unapproved, however it is publicly linked from the JCP website)

Note the 'straw poll' explicitly mentions the field of use restriction that Sun used against Harmony. This vote represents the collective wisdom of the two Executive Committees - Java ME and Java SE/EE - 32 votes in total, however there were 10 absentees.

Note that there were two 'no' votes.
Also note that Sun sits on both executive committees and thus has two votes.
But there is no other information to tie those two facts together.

Meeting summary - December 7th 2007
Resolution 1 (proposed by Oracle, seconded by BEA)
"It is the sense of the Executive Committee that the JCP become an open independent vendor-neutral Standards Organization where all members participate on a level playing field with the following characteristics:
* members fund development and management expenses
* a legal entity with by-laws, governing body, membership, etc.
* a new, simplified IPR Policy that permits the broadest number of implementations
* stringent compatibility requirements
* dedicated to promoting the Java programming model
Furthermore, the EC shall put a plan in place to make such transition as soon as practical with minimal disruption to the Java Community."


Vote tally from SE/EE Executive Committee:
Voting "yes": 13 (Apache, BEA, Fujitsu, Google, HP, IBM, Intel, Doug Lea, Nortel, Oracle, Red Hat, SAP, Hani Suleiman)
Voting "abstain": 1 (Sun)
Vote tally from ME Executive Committee:
Voting "yes": 13 (BenQ, Jean-Marie Dautelle, Ericsson, IBM, Intel, Nokia, Orange, RIM, Samsung, Siemens, Sony-Ericsson, Time Warner, Vodafone)
Voting "abstain": 1 (Sun)

Note how the proposal is explicit about the JCP becoming 'independent' and 'vendor-neutral'. Note also how the proposal accepts that the costs of such a body would have to be shared by its members. Again, the two Executive Committees - Java ME and Java SE/EE - clearly voted in favour of a resolution to further open the JCP. The sole exception was Sun, who abstained.

Meeting summary - December 7th 2007
Resolution 2 (proposed by BEA, seconded by Intel)
"The SEEE EC requests Sun to submit a Java SE7 JSR with the following characteristics:
* EG members will share costs by contributing to the TCK and RI submitted under common licensing terms that will allow any implementer to use them;
* accompanied by public spec, tck, ri licenses at JSR initiation;
* licenses do not contain Field of Use restrictions on spec implementations.
* the JSR is operated by the Spec Lead under democratic principles (e.g. Java SE6)
Furthermore, from the time of submission, TCK license(s) for Java SE5 and later will be offered without field of use restrictions on spec implementations enabling the TCK to be used by organizations including Apache."


Vote tally from SE/EE Executive Committee:
Voting "yes": 10 (Apache, BEA, Fujitsu, Google, Intel, Doug Lea, Nortel, Oracle, SAP, Hani Suleiman)
Voting "abstain": 4 (Sun, HP, IBM, Red Hat)

So, the SE/EE Executive Committee clearly voted in favour (10-4) of a resolution in favour of an open Java SE 7 specification, and for Apache to receive the testing kit for Java SE 5 and 6. There is no information available as to the details of the discussion, nor or the reasons why individual participants voted for or against.

Meeting summary - April 8th 2008
Sun's commitments on JCP reform
Onno Kluyt gave a presentation in which he outlined a series of JCP reforms that Sun is willing to commit to. The presentation was followed by an extensive discussion.
Java SE 7 JSR
Roberto Chinnici summarized Sun's plans for the Java SE 7 JSR, expressing the hope that this would be submitted and approved soon.

We can note that back in April 2008 Sun were making efforts to resolve the issues. Also note that Sun expressed the desire to submit the Java SE 7 JSR 'soon'. I'd also note that April 2008 is just before JavaOne 2008, which was in May - a time when Sun frequently makes big announcements.

Meeting summary - June 24th 2008
Geir Magnusson and Onno Kluyt reported that Sun and Apache have failed to reach agreement on the terms under which the JCK would be licensed to Apache for use in the Harmony project. They also reported that no further discussions are planned. Onno stated that Sun has no immediate plans to submit the JSR for Java SE 7. Rather, Sun plans to continue SE 7 development work through the OpenJDK project and through component JSRs that are already in progress. He expressed the hope that when the JSR is submitted at some point in the future the EC will approve it.

EC members expressed their disappointment at this outcome, and discussed the possible implications without reaching any firm conclusions.

Sun now have 'no immediate plans' to submit the Java SE 7 platform JSR. It is fair to ask what happened between April and June to cause the submission of the JSR to go from 'soon' to 'no immediate plans'. Sadly, there is no published information to answer that question, so I'll leave that open to debate (I can't comment further).

Further, it can be seen that Sun are beginning to focus on working towards version 7 using Open JDK instead.

Meeting minutes - September 25th 2008
Patrick gave an update on the upcoming elections, and informed the ECs of Sun's nominations:
...
Several members expressed concern that they weren't consulted about Sun's nominations. Patrick said that Sun was unwilling to make concessions on "control points" (such as the right to nominate for EC elections) without something in return. When members asked what concessions Sun wanted, Patrick responded that Sun expressed the desire to feel confident that the Java SE7 JSR would be considered by EC members without allowing the dispute about TCK licensing between Apache and Sun to obstruct it.

This passage shows Sun as being concerned about whether the Java SE 7 JSR would be considered without the TCK dispute being resolved. Further, they link certain aspects of Sun control in the JCP (ie. who gets a guaranteed seat on the JCP) to the Java SE 7 debate. I'm not going to speculate on the details of what is clearly a politically-charged passage.

Meeting minutes - September 25th 2008
Private Session
The ECs then went into private session for a discussion on the negotiations between Apache and Sun.
After these discussions it was agreed that Apache and Sun would explore the possibility of arbitration to resolve their differences.

Arbitration is mentioned as one possible solution.

Compatibility Rules discussed at the meeting on September 25th 2008
Slide 13:
Compatibility is a contractual obligation.
- You must not ship incompatible products.
-- Compatibility is binary.
- You can't be "almost compatible" or "a little bit incompatible".
- The JCK contains 120,000 test cases.
-- If you pass 119,999 you are not compatible.
-- If you pass 120,000 and don't meet all the other requirements you are not compatible.

This hammers home the point that any implementation of any JSR must be fully compatible. And that it is a contractual obligation. It also shows the size of the Java SE platform JSR testing kit.

Meeting minutes - January 13th 2009
Private session
The ECs then went into private session to discuss the status of the Sun-Apache negotations. During this session members had a lively discussion, and made some proposals that will be brought to the attention of Sun's management

Again, discussions to resolve the issue by the Executive Committee are still ongoing in January 2009.


There are no further minutes publicly available - January 2009 is the last published date. However, the next meeting is on April 9th 2009, so we should expect to see the minutes from February 25th approved at the April meeting very soon.

Any clearer?

I hope that these quotes, and the original minutes, provide a deeper insight into the dispute, and into how the Java SE 7 is affected. I'll draw out three points:

Firstly, that this isn't just Apache vs Sun. Other members of the Executive Committee have been willing to side with Apache and vote against Sun (February and December 2007). Bear in mind that those members have access to even more information than is public here in order to judge which way to vote. Remember though that those votes are not-binding. Due to the structure of the JCP, the other Executive Committee members cannot actually intercede on behalf of Apache.

Secondly, from the September 2008 meeting, there is distinct evidence (but not proof) that Sun feels unable to get a Java SE 7 JSR passed until the Harmony dispute is resolved. We can also note the difference in tone between April 2008 and June 2008, where a Java SE 7 platform JSR went from 'soon' to 'no immediate plans', and Sun's focus became clearer around using Open JDK instead of a JSR.

Finally, it is clear that discussions have continued in one form or another for the entire time of the dispute. It isn't the case that the Apache open letter was published and then forgotten or ignored.

Despite all the efforts, there has been no solution. And as such there is still no Java SE 7 platform specification. Thus, I hope its a little clearer why I'm arguing that, in all likelihood, we will get a JDK 7 implementation, but not a Java SE 7 platform open specification.

Summary

Personally, I'm glad these meeting minutes are now public. Everyone can now see the behind-the-scenes discussion for themselves and make up their own minds as to the true state of the debate. In addition, you can freely judge if my blogging has been factual or not.

Finally, for anyone still wondering why Apache Harmony might be of any value at all, consider this:

Apache Harmony: Thanks for the TreeMap Work!
I'd like to thank Apache Harmony for their JDK library performance efforts. We were given a tip that the Harmony folks were doing some interesting work with the TreeMap collection class, and low and behold they were.
...
Controlled measurements showed the "fat" TreeMap significantly faster with in order traversals, and improved our SPECjbb2005 score by a solid 3-5% depending on the platform.
...
Considering the potential performance gain we started the effort of porting the Apache Harmony TreeMap code to JDK 6.
...
The new TreeMap is included in JDK 6 Update 6 Performance Release

David Dagastine

So, Sun's JDK runs faster today as a direct result of code written in Apache Harmony. If you use Sun's JDK 6 update 6 or later, Apache Harmony has already benefited you. And that just one of the benefits of open specifications and competing implementations.


Stephen Colebourne
Apache Software Foundation member, speaking personally
Sun Java Champion, speaking personally
Not a committer on Harmony or OpenJDK

19 comments:

  1. Agreed, wow. But, why do apache care if they can call their implementation "java"?

    I certainly don't want apache-java running on my local nuclear reactor. This is a general problem for most open source licences in areas such as medical software, RT software etc.

    The question is, why is it not good enough to have harmony be harmony, pass the TCK but not be called java? Why would we want it called java anyway? What benefit does this give us or them?

    ReplyDelete
  2. *Very* interesting post!
    I had no idea there was so much politics in the JCP.

    Thanks a lot for that.

    ReplyDelete
  3. Rectum, damn near killed him!

    ReplyDelete
  4. Rodrigo Hartmann30 March 2009 17:21

    I wonder what IBM's true position is. Have they always considered a Sun's purchase while sitting on the JCP? How truly open can the JCP be when one of its EC members buy out one of the others? I think that's a point of interest for every Java developer out there right now.

    ReplyDelete
  5. Walter Harley30 March 2009 17:52

    Of those 120k test cases in the TCK, some proportion are intentionally or inadvertently crafted to pass against the Sun implementation, rather than against the spec per se. That is, there are bugs in the TCK that mirror bugs in the Sun implementation, relative to the spec.

    Realistically, this means that it is not possible to produce a compliant implementation without interacting with the owners of the TCK (ie Sun): you write your implementation, discover failing TCK tests, consider whether they represent bugs in your implementation or in the TCK, and if the latter, report them to Sun and hope for an exclusion or for a fix in the next TCK release.

    A consequence of this dependency is that the owners of the TCK have powerful leverage on the release schedule of any would-be-compliant implementation.

    ReplyDelete
  6. Stephen Colebourne30 March 2009 23:44

    @Ben, Harmony cannot be released until it receives an IP grant, and it can't get it without being officially tested. See the previous blog post. ie. Harmony doesn't just 'want' to be called a compatible implementation of the Java SE 5 platform specification, it HAS to be according to the rules (the same rules Sun are breaking).

    @Walter, You are probably right that some tests encode bugs in the Sun JDK. However, the JSPA does have a test appeals process, whereby invalid tests must be chaged or revoked.

    Bear in mind that you can run Eclipse on Harmony, so it must be pretty close to the Sun JDK.

    ReplyDelete
  7. Dmitri Trembovetski31 March 2009 00:28

    > Bear in mind that you can run Eclipse on Harmony, so it must be pretty close to the Sun JDK.

    Eclipse doesn't use rather large pieces of the Java API like AWT, Swing and Java2D. So you can't really base the completeness of a Java implementation by whether it can run Eclipse. Which is why there's TCK..

    Dmitri

    ReplyDelete
  8. Bible Thumper31 March 2009 05:35

    Ok dude. Will you stop whining already. Fine. Sun is EVIL and APACHE is God. Now can we all move on?

    ReplyDelete
  9. Ever tried a Java GUI application on Harmony which uses Java APIs (AWT, Swing and Java2D)? The level of compatibility is ABYSMAL! In fact I have not seen a single application running halfway correctly. I tried at least 10 Swing/AWT GUIs and a few Java2D based visualisation apps, which all ran fine on Sun, Bea, IBM and Excelsior implementations. Standard-demos such as SwingSet or Java2DDemo (which come with the JDK) do not work at all.

    Only the very simplest examples seem to work on Harmony, often with ridiculous performance and obvious glitches. I have been trying Harmony regularly since at least a year and I haven't noticed any improvement. On desktop Harmony is not even close to being compatible with Java.

    ReplyDelete
  10. Maybe I'm dense, but I don't really see all that distinct a difference between the "Harmony welcome" and the JSR 270 quote. One is made by a person, somewhat embellished, the other obviously made in a submitted document, and not embellished.

    But they both seem to say the same thing: Namely, that open licensed implementations can be created, if one wants to do so.

    Obviously that's a duplication of effort, but if one wants to expend the resources, then one can.

    As for the meeting minutes, it seems to me that the *worst* one can say about Sun on the JCP executive committees is that they have abstained on some issues others would have liked them to vote for. That's not the same as voting *AGAINST* them. And before one castigates Sun for their voting record; at least *one* of the issues where the votes were tallied in this blog showed not only Sun, but IBM, HP, and RedHat abstained as well.

    If the author (or readers) think that every vote should be unanimous, then they obviously have never been on a standards body committee. The whole *idea* of the committee is to resolve issues with the standards; and it's often likely such resolutions will not please *everyone*. Actually, I see far less contention in the minutes here than most married couples have; much less a standards organization :D

    As for compatible, either one is or one isn't. "A little bit compatible"; or "mostly compatible" is just like being "a little bit pregnant". Either you are or you aren't. Since the whole idea behind Java is to have an environment available on a wide range of hardware and software that allows an application to run *unmodified* (or uncustomized) upon it; you really *should* be 100% compatible.

    Most languages allow for 'extensions'; but, then, most languages don't attempt to promise binary compatibility over differing hardware or operating systems. That's what makes Java somewhat unique. Unfortunately, to take advantage of that uniqueness also means there are restrictions.

    Yin and Yang, if you will.

    ReplyDelete
  11. please stop, we had enough of your rants.

    ReplyDelete
  12. I guess I'm being dense, but I don't see your point about "how times change" and those two quotes. One quote says basically "we allow..." and the next one says "nothing prevents...". Is there some subtle change in tone (or policy) that I'm missing?

    Though I disagree with some of your points, you have made a compelling case that Sun is avoiding a JDK7 JSR because it won't pass without them giving in to Apache. Keep up the fun posts!

    ReplyDelete
  13. Isn't this all just about IBM trying to snatch control over the Java platform away from Sun?

    ReplyDelete
  14. Stephen Colebourne31 March 2009 22:26

    @Richard, @Andy, The "how times change" quote was something I added to compare those two quotes (as a pair) to the bigger dispute. Not to compare the two quotes to one another. Sorry for the confusion.

    On the question of voting, there are three votes documented, and none are on technical spec issues.

    On compatibility, Apache Harmony wants to be compatible. Thats why Apache wants (and is required to have) the testing kit!

    ReplyDelete
  15. Gregg Wonderly1 April 2009 22:10

    Sun's point is that why have all these compatibility issues for users of the Java platform to deal with. Java compatibility, without a common codebase, is not gonna really be possible.

    Harmony developers should focus on participating in the openjdk project if they want Java to be "better." There can never be a cleanroom implementation which is compatible with the AWT/Swing stuff in particular. There are still things happening which are implicitly part of the "Java" standard because they don't have a test yet.

    ReplyDelete
  16. Stephen Colebourne2 April 2009 11:41

    @Greg, "Sun's point is that why have all these compatibility issues for users of the Java platform to deal with. Java compatibility, without a common codebase, is not gonna really be possible"

    By that argument, there should only be one implementation of any JSR, for example, Glassfish would be the only implementation of Java EE!

    The point of having a spec and a testing kit is to ensure compatibility and allow multiple implementations - and that includes Java SE (which as the second quote in the blog shows, was clearly the intention of the Java SE 6 spec).

    ReplyDelete
  17. "The point of having a spec and a testing kit is to ensure compatibility and allow multiple implementations"

    The main point of the "spec" is to be user documentation, not requirements for compatibility. The main point of the TCK is to provide a test suite for testing Sun's JDK, not to test compatibility of other implementations.

    As Gregg notes above, alternative implementations will always be incompatible, as we learned with C, C++, Javascript, HTML, etc. and standards bodies.

    And yes, a single implementation would minimize compatibility problems.

    ReplyDelete
  18. @Andy
    "The main point of the "spec" is to be user documentation, not requirements for compatibility"

    That is quite explicitly not the case for most (if not all JSRs).
    e.g. The Servlet 2.4 spec says:

    The intended audience for this specification includes the following groups:
    • Web server and application server vendors that want to provide servlet engines that conform to this standard.
    • Authoring tool developers that want to support Web applications that conform to this specification
    • Experienced servlet authors who want to understand the underlying mechanisms of servlet technology.
    We emphasize that this specification is not a user’s guide for servlet developers and is not intended to be used as such

    Graham Hamilton's comment ("The licensing rules for J2SE 5.0 were carefully designed to allow independent, compatible open-source implementations of the J2SE specification") shows that Sun believe (or at least wanted to create the impression that they believe) that it is possible to be both independent, and compatible.

    Furthermore the download page (http://jcp.org/aboutJava/communityprocess/final/jsr270/index.html) for the JSE6 spec says:
    * If you want to read the specification for evaluation, or to build an application that uses the specification, click here.
    * If you intend to build an implementation of the specification, click here.

    They go to the trouble of separating out the 2 possible uses, and have different documents to download. The website of the JCP claims that it intends to support people building implementations of the JSE specs.

    If you and Gregg are correct in your analysis that Sun actually has no interest in allowing or supporting alternate implementations, then it simply adds more support for Stephen's argument - that Sun's public (and contractual) statements are disingenuous, and do not actually reflect their real intentions. All their comments about supporting alternate + open source implementations were simply an attempt to get the open source community (notably Apache) off their back, with no intention to follow through.

    The facts seem fairly simple - and I think we might even agree on them:
    * Sun is (or at least was) trying to create an impression that they support independent, open source and compatible implementations of Java.
    * Sun is behaving in ways that make independent, open source and compatible implementations of Java impossible.

    The only conclusion that I can draw is that Sun was trying to pull the wool over the eyes of the Java community. They wanted the PR benefits of supporting open source, without having to follow through on it.

    I personally don't care whether Harmony is actually released. But I most certainly don't like being lied and used in the way I believe Sun has lied to and used the Java community.

    ReplyDelete