More detail on Closures in JDK 7

This blog goes into a little more detail about the closures announcement at Devoxx and subsequent information that has become apparent.

Closures in JDK 7

At Devoxx 2009, Mark Reinhold from Sun announced that it was time for closures in Java. This was a big surprise to everyone, and there was a bit of a vacuum as to what was announced. More information is now available.

Firstly, Sun, via Mark, have chosen to accept the basic case for including closures in Java. By doing so, the debate now changes from whether to go ahead, to how to proceed. This is an important step.

Secondly, what did Mark consider to be in and what out? Well, he indicated that non-local returns and the control-invocation statement were out of scope. There was also some indication that access to non-final variables may be out of scope (this is mainly because it raises nasty multi-threading Java Memory Model issues with local variables).

In terms of what was included, Mark indicated that extension methods would be considered. This would be necessary to provide meaningful closure style APIs since the existing Java collection APIs cannot be altered. The result of what was in was being called "simple closures".

Finally, Mark offered up a possible syntax. The syntax he showed was very close to FCM:

  // function expressions
  #(int i, String s) {
    System.println.out(s);
    return i + str.length();
  }

  // function expressions
  #(int i, String s) (i + str.length())
  
  // function types
  #int(int, String)

As such, its easy to say that Sun has "chosen the FCM proposal". However, with all language changes, we have to look at the semantics, not the syntax!

The other session at Devoxx was the late night BOF session. I didn't attend, however according to Reinier Zwitserloot, Mark indicated that Exception transparancy might not be essential to the final proposal.

According to Reinier, Mark also said "It is not an endorsement of FCM. He was very specific about that." I'd like to consider that in a little more detail (below).

The final twist was when a new proposal by Neal Gafter was launched which I'm referring to as the CFJ proposal. After some initial confusion, it became clear that this was written 2 weeks before Devoxx, and that Neal had discussed it primarily with James Gosling on the basis of it being a "compromise proposal". Neal has also stated that he didn't speak to Mark directly before Devoxx (and nor did I).

There are a number of elements that have come together in the various proposals:

	CICE	BGGA 0.5	FCM 0.5	CFJ 0.6a	Mark's announcement (Devoxx summary)
Literals for reflection	-	-	Yes	-	No (No info)
Method references	-	-	Yes	Yes	Worth investigating (No info)
Closures assignable to single method interface	Yes	Yes	Yes	Yes	Yes (No info)
Closures independent of single method interface	-	Yes	Yes	Yes	Yes
Access non-final local variables	Yes	Yes	Yes	Yes	No (Maybe not)
Keyword 'this' binds to enclosing class	-	Yes	Yes	Yes	No info (Probably yes)
Local 'return' (binds to closure)	Yes	-	Yes	Yes	Yes
Non-local 'return' (binds to enclosing method)	-	Yes	-	-	No
Alternative 'return' binding (Last-line-no-semicolon)	-	Yes	-	-	No
Exception transparancy	-	Yes	Yes	Yes	No info (Maybe not)
Function types	-	Yes	Yes	Yes	Yes
Library based Control Structure	-	Yes	-	-	No
Proposed closure syntax	Name(args) {block}	{args => block;expr}	#(args) {block}	#(args) {block} #(args) expr	#(args) {block} #(args) (expr)

A table never captures the full detail of any proposal. However, I hope that I've demonstrated some key points.

Firstly, the table shows how the CFJ is worthy of a new name (from BGGA) as it treats the problem space differently by avoiding non-local returns and control invocation. Neal Gafter is also the only author of the CFJ proposal, unlike BGGA.

Secondly, it should be clear that at the high level, the CFJ and FCM proposals are similar. But in many respects, this is also a result of what would naturally occur when the non-local returns and control invocation is removed from BGGA.

Finally, it should be clear that it is not certain that the CFJ proposal meets the aims that Mark announced at Devoxx ("simple closures"). There simply isn't enough hard information to make that judgement.

One question I've seen on a few tweets and blog posts is whether Mark and Sun picked the BGGA or FCM proposal. Well, given what Mark said in the BOF (via Reinier), the answer is that neither was "picked" and that he'd like to see a new form of "simple closures" created. However, it should also be clear that the choices announced at Devoxx were certainly closer to the FCM proposal than any other proposal widely circulated at the time of the announcement.

At this point, it is critical to note that Neal Gafter adds a huge amount of technical detail to each proposal he is involved with - both BGGA and CFJ. The FCM proposal, while considerably more detailed than CICE, was never at the level necessary for full usage in the JDK. As such, I welcome the CFJ proposal as taking key points from FCM and applying the rigour from BGGA.

As of now, there has been relatively little debate on extension methods.

Summary

The key point now is to focus on how to implement closures within the overall scope laid out. This should allow the discussion to move forward considerably, and hopefully with less acrimony.

Why JSR-310 isn't Joda-Time

One question that has been repeatedly asked is why JSR-310 wasn't simply the same as Joda-Time. I hope to expain some reasons here.

Joda-Time as JSR-310?

At its heart, JSR-310 is an effort to add a quality date and time library to the JDK. So, since most people consider Joda-Time to be a quality library, why not include it directly in the JDK?

Well, there is one key reason - Joda-Time has design flaws.

Now before everyone panics and abuses that line as a tweet, I need to say that Joda-Time is by far the best option curently available, and that most users won't appreciate the design flaws. But, I do want to document them, so the basis for the changes in JSR-310 is clear.

1) Human/Machine timelines

One element of clarity is a better understanding of the distinction between the two principle views of the timeline - Human and Machine.

Machines have one view - a single, ever increasing number. In Java we set zero as 1970-01-01T00:00Z and count in milliseconds from there.

Humans have a totally different view of time. We have multiple calendar systems (one primary, many others), which divide the timeline into years, months, days, hours, minutes and seconds. In addition, humans have time zones which cause the values to vary around the globe, and for there to be gaps and overlaps in the human timeline as DST starts and ends.

Much of the time, a conversion between the two timeline views is possible with a time zone, however in a DST gap or overlap, things are much less clear.

Joda-Time defines two key interfaces - ReadableInstant and ReadablePartial. Both the Instant class (simple instant in time) and the DateTime class (human view of an instant in time) are implementations of ReadableInstant. This is wrong.

DateTime is a human-timeline view of the world, not a machine-timeline view. As such, DateTime is much better thought of, and designed as, a LocalDateTime and a timezone rather than the projection of the machine timeline onto the human timeline. Thus, DateTime should not implement ReadableInstant.

2) Pluggable chronology

What is the range of values returned by this method in Joda-Time?:
int month = dateTime.getMonthOfYear();
The answer is not 1 to 12, but could be 1 to 13! (yet January is still 1 and December is 12)

The answer to this puzzler is down to pluggable chronologies. Each date/time class in Joda-Time has a pluggable chronology. This defines the calendar system that is in use. But most users of the API never check to see if the chronology is the standard ISO/ chronology before calling getMonthOfYear(). Yet, the Coptic chronology has 13 months in a year, and thus can return a range of 1 to 13.

A better solution would be to keep the date/time classes restricted to a single calendar system. That way, the result from each method call is clear, and not dependent on any other state in the class, like the chronology.

3) Nulls

Joda-Time accepts null as a valid value in most of its methods. For date/times it means 1970-01-01T00:00Z. For durations it means zero. For peiods it means zero.

This approach causes random bugs if your code happens to provide a null to Joda-Time that you hadn't originally planned for. Instead of throwing an error, Joda-Time continues, and the resulting date/time is going to be different from what you want.

4) Internal implementation

Certain aspects of the internal implementation are complex, and the result of having pluggable chronologies and a misunderstanding of the machine/human divide in the timeline. Changing this is a big change to the code.

One particular area of trouble is managing DST overlaps. The behaviour in Joda-Time of these isn't that well defined.

Summary

Joda-Time isn't broken!

It does the job it was designed for, and does it much better than the JDK. And it is widely used and without too many major issues. However, after a few years, it is now clear where it could be designed better.

I took the decision that I didn't want to add an API to the JDK that had known design flaws. And the changes required weren't just minor. As a result, JSR-310 started from scratch, but with an API 'inspired by Joda-Time'.

I hope that explains the thought process behind the creation of a new API in JSR-310.

Closures in JDK 7

So, on Wednesday, Mark Reinhold from Sun announced that it was time for closures in Java. This came as a surprise to everyone, but what was announced?

Closures in JDK 7

Mark took the audience through the new features of JDK 7 in his presentation at Devoxx. As part of this he showed the Fork Join framework.

Part of this framework features a set of interfaces to support functional style predicates and transforms. But, in order to implement this in Java required 80 interfaces, all generified, and that only covered 4 of the primitive types! Basically, Mark, and others, had clearly come to the conclusion that this code was important to Java, but that the implementation with single-method interfaces, generics and inner classes was too horrible to stomach for the JDK.

Thus, 'its time to add closures to Java'.

Mark announced that Sun would investigate, and intended to commit, closures to OpenJDK for JDK 7. While there will be no JSR yet, it isn't unreasonable to expect that there will be an open aspect to discussions.

Mark noted that the work in the closures discussion, and particularly the detail in the BGGA proposal, had allowed the options to be properly explored. He emphasised how this had resulted in the decision to reject key aspects of BGGA.

JDK 7 closures will not have control-invocation statements as a goal, nor will it have non-local returns. He also indicated that access to non-final variables was unlikely. Beyond this, there wasn't much detail on semantics, nor do I believe that there has been much consideration of semantics yet.

On syntax, Mark wrote up some strawman syntaxes. These follow the FCM syntax style:

  // function expressions
  #(int i, String s) {
    System.println.out(s);
    return i + s.length();
  }

  // function expressions
  #(int i, String s) (i + s.length())
  
  // function types
  #int(int, String)

Mark also argued that a form of extension methods would be added to allow closures to be used with existing libraries like the collections API.

Although I must emphasise that everything announced was a proposal and NOT based on any specific proposal (BGGA, FCM or CICE). However, the syntax and semantics do follow the FCM proposal quite closely.

In addition, Neal Gafter has produced an initial formal writeup of what was announced derived from BGGA. Based on my initial reading, I believe that Neal's document represents a good place to start from. I also hereby propose that we refer to Neal's document as the CFJ proposal (Closures for Java), as BGGA rather implies control invocation.

So, lets hope the process for closures provides for feedback, and we get the best closures in the style most suitable for Java in JDK 7.

Joda-Money 0.5

I've just released v0.5 of Joda-Money. It is a simple money and currency package along the lines of Joda-Time.

Joda-Money 0.5

Joda-Money is a project to provide a small, focussed monetary library for Java, similar to Joda-Time. I released v0.5 this morning.

Website, Javadoc, Download, Maven 2 repo

The code is all tested and there are no known bugs. But I wouldn't suggest using it in production code quite yet. I do want lots of feedback and testing though! So, all feedback is welcomed, here, the forum or the list.

The main API is as I originally intended. There is a Money and BigMoney class, the former decorates the latter to restrict the number of decimal places to the currency scale.

There is also CurrencyUnit and a provided data file of currency data. Finally, there is printing (but not parsing yet).

The goal of the library remains just these basic classes. Financial calculations (for whatever domain) or classes storing specific concepts like gross/net/tax are out of scope for the library.

This release is part of the 'release early' approach. Please provide as much feedback as you can!

JSRs submitted over time

The JCP is the body that looks after standards in Java, and a JSR is the individual request to create a standard. But how has the volume of JSRs changed over time?

JSRs over time

In tackling this question, I decided to focus on the simple statistic of how many JSRs were submitted per quarter over time. This does not address JSRs that were never finished, or how long the JSR took to complete, or whether it is still active and in maintenance mode.

The data as to when a JSR was submitted is publicly recorded on the JSR page, for example JSR-311 was submitted for voting on 13th February 2007.

So, without any further ado, here is the main graph of JSRs submitted over time. The red line is the total of all JSRs (Java SE, EE and ME) while the green line is only SE and EE:

Hopefully, the trend is clear - the number of JSRs submitted has been falling over time. To be clear - only 5 JSRs were submitted in the whole of 2008, and only 4 have been submitted so far this year (roughly 1 JSR per quarter).

The question is whether this is a worrying trend, or just normal for a maturing language. I suspect its a combination of factors.

In part, Java is now a platform where most innovation occurs outside the JCP. This is a good thing, and shows a healthy ecosystem that doesn't need the 'blessing' of an official standard to make a technology or product useful. Examples of this are Spring and OSGi - both were widely used without a JSR (although ironically both now have some connection to the JCP).

Another factor is that a lot of the Java EE work, and possibly others, appears to be going on as maintenance revisions to an existing JSR. This doesn't show up on this graph, as the no new JSR is created. This is fine, although it may suggest an unwillingness to go through the process of starting a new JSR.

One more point worth noting is the timeline. JavaOne 2006 was when the Open Sourcing of Java was announced, and Sun developers were taken off active development to begin the very long process of legally reviewing the entire Java codebase. This was then followed by the (mis)adventure of JavaFX, which drew even more developers away from core Java and new developments. There may be some signs of this in the graph, where the steady state of 2004-2006 declines at the end of 2006.

Finally of course, there is no doubt about the absence of Java SE JSRs for Java SE 7 (language changes and API changes). I've written before about the politics behind this.

Certainly I see the JCP as being at a crossroads. The Oracle acquisition is key here. Oracle could choose to reinvigorate the JCP, with industry partners, ideally separating the JCP from Oracle itself, as IBM did with Eclipse. However, given the money Oracle is spending, I suspect that won't happen, despite their previous sentiments.

Unfortunately, continuing the status quo doesn't wash either. At present, the JCP is deadlocked on how to proceed with the Java SE 7 specification. As a result, JDK 7, Project Jigsaw and Project Coin are proceeding outside the JCP. Plus of course, JavaFX isn't related to the JCP at all.

Summary

The JCP has considerably fewer JSRs being submitted now than the 2000 to 2003 period. This doesn't quite mean that the JCP is dead or no longer has value, but its another data point as to how Java is maturing and stabilising. Or perhaps less kindly - fossilising.

Feedback welcome.

JDK 7 - Method suggestions

Joe Darcy has opened up a call for methods to add to the core JDK 7 lang/util package. The idea is to add methods that are "commonly-written utility methods".

JDK 7 utility methods

There is lots of prior art in the field of general utility method libraries for Java. Perhaps the best known is Apache Commons Lang. Lets choose some methods (other than string utilities) that might be appropriate for the JDK from ObjectUtils.

1) Null-safe equals check.
boolean Objects.equals(Object object1, Object object2)
Returns true if both are null, or both are non-null and equal.

2) Null-safe hash code.
int Objects.hashCode(Object object)
Returns the hash code of the object, or zero if the object is null.

3) Null-safe toString.
String Objects.toString(Object object)
Returns the toString of the object, or "" if the object is null.

4) Null-safe toString with specified default.
String Objects.toString(Object object, String defaultStr)
Returns the toString of the object, or defaultStr if the object is null.

5) Null-safe object defaulting.
T Objects.defaultNull(T object, T defaultValue)
Returns the object, or defaultValue if the object is null.
(Of course, the Elvis operator is a much clearer approach for this common case...)

6) Get maximum and minimum.
T Objects.max(T comparable1, T comparable2)
T Objects.min(T comparable1, T comparable2)
Returns the maximum/minimum object, returning the non-null object if either is null, or null if both are null.
Generics allow the return type to check the input is also a Comparable.

These would appear to be the obvious methods to add to an Objects utility class.

The following are missing methods on Integer/Long:

7) Compare two primitives.
int Byte.compare(byte value1, byte value2)
int Short.compare(short value1, short value2)
int Integer.compare(int value1, int value2)
int Long.compare(long value1, long value2)
Safely returns the comparison (-1/0/1) indicator for two primitives. (These methods already exist on Float/Double).

The methods from SystemUtils can be very useful in reducing arbitrary strings in code:

8) Get common environment variables.
File System.getJavaIoTempDir()
File System.getJavaHomeDir()
File System.getUserHomeDir()
File System.getUserDir()
Returns the java environment variables. There are lots more in SystemUtils that could be added.

Some more null handling methods for primitives - again, Elvis would be a better solution:

9) Convert wrapper to primitive avoiding NPE.
boolean Boolean.booleanValue(Boolean obj, boolean defaultValue)
char Character.charValue(Character obj, char defaultValue)
byte Byte.byteValue(Byte obj)
byte Byte.byteValue(Byte obj, byte defaultValue)
short Short.shortValue(Short obj)
short Short.shortValue(Short obj, short defaultValue)
int Integer.intValue(Integer obj)
int Integer.intValue(Integer obj, int defaultValue)
long Long.longValue(Long obj)
long Long.longValue(Long obj, long defaultValue)
float Float.floatValue(Float obj)
float Float.floatValue(Float obj, float defaultValue)
double Double.doubleValue(Double obj)
double Double.doubleValue(Double obj, double defaultValue)
Safe ways to convert a wrapper to a primitive.
The numeric ones return zero if the default argument isn't specified.
There is also a case for methods to convert arrays of wrappers to arrays of primitives.

These should probably be on a Booleans utility class as per BooleanUtils.

10) Boolean methods for clearer code.
boolean Booleans.isTrue(Boolean booleanObj)
boolean Booleans.isFalse(Boolean booleanObj)
Return true as per the method name, false if null.
boolean Booleans.isNotTrue(Boolean booleanObj)
boolean Booleans.isNotFalse(Boolean booleanObj)
Return true as per the method name, true if null.

11) Negate handling null.
Boolean Booleans.negate(Boolean booleanObj)
TRUE returns FALSE, FALSE returns TRUE, null returns null.

11) Boolean arithmetic.
boolean Booleans.and(boolean[] array)
boolean Booleans.or(boolean[] array)
boolean Booleans.xor(boolean[] array)
boolean Booleans.and(Boolean[] array)
boolean Booleans.or(Boolean[] array)
boolean Booleans.xor(Boolean[] array)
Performs the stated binary maths.

12) Character comparison.
boolean Character.equalsIgnoreCase(char ch1, char ch2)
Compares two characters ignoring case.

These should be constants for empty arrays on common classes:

13) Empty array constants.
Boolean[] Boolean.EMPTY_ARRAY
boolean[] Boolean.EMPTY_PRIMITIVE_ARRAY
Character[] Character.EMPTY_ARRAY
char[] Character.EMPTY_PRIMITIVE_ARRAY
Byte[] Byte.EMPTY_ARRAY
byte[] Byte.EMPTY_PRIMITIVE_ARRAY
Short[] Short.EMPTY_ARRAY
short[] Short.EMPTY_PRIMITIVE_ARRAY
Integer[] Integer.EMPTY_ARRAY
int[] Integer.EMPTY_PRIMITIVE_ARRAY
Long[] Long.EMPTY_ARRAY
long[] Long.EMPTY_PRIMITIVE_ARRAY
Float[] Float.EMPTY_ARRAY
float[] Float.EMPTY_PRIMITIVE_ARRAY
Double[] Double.EMPTY_ARRAY
double[] Double.EMPTY_PRIMITIVE_ARRAY
String[] String.EMPTY_ARRAY
Class[] Class.EMPTY_ARRAY
Object[] Objects.EMPTY_OBJECT_ARRAY
Return true as per the method name, true if null.

Or a better solution might be a method on Class:
T[] Class.emptyArray();
This allows code like:
Boolean.class.emptyArray();

14) Array size.
boolean Arrays.isEmpty(Object[] array)
int Arrays.size(Object[] array)
Where a null array is empty/size-zero.

15) Collection size.
boolean Collections.isEmpty(Collection coll)
int Collections.size(Collection coll)
boolean Collections.isEmpty(Map map)
int Collections.size(Map map)
Where a null collection is empty/size-zero.

Or, even better, add a new interface Sized with a single method to get the size. This would be retrofitted to Collection, Map, String and arrays (of course, technically, thats a language change...).

And some Class utilities for less NPEs when outputting the class name in debugging:

16) NPE safe way to get class name.
String Class.getName(Class cls)
String Class.getSimpleName(Class cls)
String Class.getPackageName(Class cls)
Returns the class name, or null if the input is null.
Again, the null-safe operators would avoid this kind of specific method.

Locale could do with some love, as per LocaleUtils:

17) Parse a locale string.
Locale parse(String localeStr)
Parses the locale string to a locale.

16) Country/Language lists.
List Locale.countriesByLanguage(String langaugeStr)
List Locale.languagesByCountry(String countryStr)
Extracts just countries or languages.

A better solution would be two new classes Country and Language. These would be useful in other places in the JDK, where a locale is used and it should be a country or language.

And the big one that I've missed - safe maths, as per Joda-Time FieldUtils:

18) Add/subtract/multiply/cast safely.
int Math.safeToInt(long value)
int Math.safeNegate(int value)
long Math.safeNegate(long value)
int Math.safeAdd(int value1, int value2)
long Math.safeAdd(long value1, int value2)
long Math.safeAdd(long value1, long value2)
int Math.safeSubtract(int value1, int value2)
long Math.safeSubtract(long value1, int value2)
long Math.safeSubtract(long value1, long value2)
int Math.safeMultiply(int value1, int value2)
long Math.safeMultiply(long value1, int value2)
long Math.safeMultiply(long value1, long value2)
These perform the specified mathematical operation, but throw ArithmeticException, rather than failing quietly, if overflow occurs.

I should also note that there are a whole heap of BigDecimal, BigInteger and String utilities that could be added.

Finally, of course, there are lots of other utility class libraries apart from Commons-Lang. Its simply that Commons-Lang provides a good point to start the discussion.

Feel free to (a) complain about my choices, and (b) suggest your own ideas. Remember to focus on non-string core classes for now.

Joda-Money

I've been busy piloting a new open source project - Joda-Money. The idea is a simple money and currency package along the lines of Joda-Time.

Joda-Money

Joda-Money is a new project to try and create a small, focussed monetary library for Java. While there are various libraries that tackle this area, I wanted one that fitted with the Joda-Time style.

So far, I've created a Money and CurrencyUnit class and started on formatting. The goal of the library is only these basic classes. Financial calculations (for whatever domain) or classes storing specific concepts like gross/net/tax are out of scope for the library.

The design is as follows:

Money is the main class, and supports an amount of money with the decimal places determined by the currency. Thus, GBP always has 2 decimal places, while JPY always has 0.

CurrencyUnit is the class representing currency. I debated whether to just use the JDK Currency class, but it doesn't handle currency changes, or provide the numeric code (both being fixed in JDK 7).

MoneyFormatter will provide formatting of Money instances.

BigMoney will be the final class, which will allow any amount of decimal places to be specified for any currency.

This blog is a 'release early' notification. If you like the Javadoc, or think that this is an area that should be tackled, then let me know. If not, then also let me know (I don't want to waste my time...). Lets hear the feedback!

No Java SE 7 - US DOJ investigation

There are signs that the US Department of Justice is interested in the Java licensing issues I've reported on recently.

Department of Justice

As part of the Oracle takeover of Sun, the US Department of Justice (DOJ) has been investigating the deal for antitrust/monopoly issues. There is normally a fixed time limit of 30 days for these investigations. On Friday, however, the DOJ decided to extend the period.

Wall Street Journal report (cached):
The extension by the Justice Department is unusual. The department typically only extends investigations "in a small percentage of mergers," said Samuel Miller, an antitrust attorney at Sidley Austin LLP, adding that the department likely "determined that there were issues of competitive concern that required seeking additional information."
Still, the extension doesn't mean the Justice Department will move to block the deal, said Mr. Miller. Often in these cases, the acquirer agrees to certain conditions or to divest some assets, he said.

However, what was most interesting was Oracle's press release in response to this:

Oracle press release:
Oracle issued the following statement, attributable to Dan Wall, Latham & Watkins counsel to Oracle:
"We've had a very good dialogue with the Department of Justice and we were almost able to resolve everything before the Second Request deadline. All that's left is one narrow issue about the way rights to Java are licensed that is never going to get in the way of the deal. I fully expect that the investigation will end soon and not delay the closing of the deal this summer."

The second sentence is quite a teaser. It clearly indicates that the DOJ is concerned about the licensing of Java, something that I've hopefully shown is critical to every Java developer.

Of course we have no knowledge of the detail here beyond this bare statement. But if I were to speculate, I'd be thinking that the DOJ has noticed the ASF-Sun dispute and the amount of power the JCP gives the owner of Java, and are thinking about how to handle them.

Summary

The US Department of Justice is concerned about the licensing of Java. Lets hope that they get some good advice on the issues (oh, and if you're from the DOJ and reading this, feel free to contact me!!!)

No Java SE 7 - The Oracle perspective

So, I've been writing this series of blogs about how Sun's decision to block Apache Harmony has caused a lack of a Java SE 7 specification. Does the Oracle takeover change anything?

Obviously, at this point, everyone is just digesting the news that Oracle will soon own Sun (subject to various details). I don't have any special or inside information on this, but I can draw out some pointers from the past and my reading of licenses.

Oracle JCP voting

Firstly, lets look at the key vote that is recorded in the JCP minutes:

JCP EC meeting summary - December 7th 2007
Resolution 1 (proposed by Oracle, seconded by BEA)
"It is the sense of the Executive Committee that the JCP become an open independent vendor-neutral Standards Organization where all members participate on a level playing field with the following characteristics:
* members fund development and management expenses
* a legal entity with by-laws, governing body, membership, etc.
* a new, simplified IPR Policy that permits the broadest number of implementations
* stringent compatibility requirements
* dedicated to promoting the Java programming model
Furthermore, the EC shall put a plan in place to make such transition as soon as practical with minimal disruption to the Java Community."

Vote tally from SE/EE Executive Committee:
Voting "yes": 13 (Apache, BEA, Fujitsu, Google, HP, IBM, Intel, Doug Lea, Nortel, Oracle, Red Hat, SAP, Hani Suleiman)
Voting "abstain": 1 (Sun)
Vote tally from ME Executive Committee:
Voting "yes": 13 (BenQ, Jean-Marie Dautelle, Ericsson, IBM, Intel, Nokia, Orange, RIM, Samsung, Siemens, Sony-Ericsson, Time Warner, Vodafone)
Voting "abstain": 1 (Sun)

Resolution 2 (proposed by BEA, seconded by Intel)
"The SEEE EC requests Sun to submit a Java SE7 JSR with the following characteristics:
* EG members will share costs by contributing to the TCK and RI submitted under common licensing terms that will allow any implementer to use them;
* accompanied by public spec, tck, ri licenses at JSR initiation;
* licenses do not contain Field of Use restrictions on spec implementations.
* the JSR is operated by the Spec Lead under democratic principles (e.g. Java SE6)
Furthermore, from the time of submission, TCK license(s) for Java SE5 and later will be offered without field of use restrictions on spec implementations enabling the TCK to be used by organizations including Apache."

Vote tally from SE/EE Executive Committee:
Voting "yes": 10 (Apache, BEA, Fujitsu, Google, Intel, Doug Lea, Nortel, Oracle, SAP, Hani Suleiman)
Voting "abstain": 4 (Sun, HP, IBM, Red Hat)

(My highlights)

So, we have a clear indication from December 2007 that Oracle were actively supporting the ASF in the dispute with Sun.

Of course, that may have been a competitive move against Sun. Now that Oracle holds the power, it may choose to use its power very differently.

JCP

I've read the JSPA and JCP documents today to see what they say about Sun being taken over. I couldn't find anything in those documents (Feel free to correct me).

As such, we should go with the simple model that I drew up, however now all contracts previously signed between companies/individuals and Sun (about the JCP), will automatically become contracts with respect to Oracle. Thus, Oracle now sits in the centre of the JCP ring, and all power accrues to it. How it chooses to use that power will be a determining factor of Java's future.

OpenJDK, Glassfish, Netbeans, etc

In addition, any contracts signed with respect to any Sun Open Source project (the SCA) would I'm assuming naturally become contracts with Oracle. There is a FAQ on this though:

SCA FAQ
Q: What if Sun is acquired, or the rights to a particular code base are transferred? Do I have assurances that the party receiving these rights will continue to honor the SCA?
A: The SCA does not include any relicensing terms or obligations in the event of transfer of rights. But because contributors retain all their rights, there is no danger that contributions can be made exclusively proprietary. Contributors retain the ability to make sure their contributed material is always freely available.

So, the SCA itself provides no guarantees that the SCA will continue to be honoured. There are some nice platitudes here about your own personal contributions, however the reality is that your single patch to a large project like Glassfish, Netbeans or OpenJDK is meaningless without the rest of the codebase.

The Java Trap

The 'Java Trap' is a reference to a campaign by Richard Stallman and the FSF to ensure that the Java platform was usable by Free software (ie. the FSF definition of free, not free as in beer).

A key aim of the campaign was to ensure that were Sun to ever change its mind about the licensing of Java, or be taken over by a Free/Open Source Hostile company, that existing Java programs could continue to be run. The 'Java Trap' battle was apparently won in May 2006 when Sun announced that Java would become Open Source, and later that the GPL license would be used.

Interestingly, this wasn't that long after Apache Harmony was launched in May 2005. Various commentators have suggested that Harmony's faster than expected progress was in fact a key factor in Sun's decision to make Java Open-Source.

If that is the case, then its interesting to consider just how important that pressure that Harmony (a project that many like to criticise) put on Sun to cause Java SE to become Open Source. We will probably never know for sure, but whatever the reasons, we should be thankful that Java SE is Open Source now and that Oracle can't remove that right.

Except....

The Java Trap still exists

I claim that there is still a Java Trap (although I'll accept that I'm talking about something slightly different to the original issue). So, what do I mean?

Well, as I've been banging on in this blog series, it's all about the IP.

OpenJDK 6 is a GPL licensed implementation of the Java SE 6 specification. As such, it can be tested using the official testing kit. And as I've shown, once you pass the tests, you get granted the IP.

But what about the IP added to Java since version 6?

Well, there is no specification for Java SE 7, we've established that. And Oracle as new masters of the JCP could choose to keep it that way. And without that specification and JSR, there is no testing kit, and there is no way to pass the IP from all the contributors to the differences between Java SE 6 and Java SE 7 to the completed Open JDK 7. Ironically, this is exactly the same situation that the ASF has been highlighting in the Apache Harmony dispute.

The problem isn't that the code of Open JDK 7 isn't Free and Open under the GPL. The problem is that there is no specification and JSR to grant the necessary IP rights.

Thus, the new 'Java Trap' is that an owner of Java could still choose to take the next version of Java back into a closed world again, simply through its control of the IP. Sure, existing programs would still run on Open JDK 6, but that could be the one and Free and Open version of Java SE.

Now, perhaps some will choose to ignore IP questions, or to consider them meaningless. Unfortunately lawyers don't think that way.

Some of you will be shouting 'but we'll fork Open JDK 6' in that scenario. But I'll say it again - you can't get the IP! You can't get certified as being compatible if there is no specification! The Trap is Back.

Of course, this is probably worst case scenario. After all, we have no idea what Oracle plans to do with Java. But there is one less major player now to keep the market honest.

Anyway, I do think its worthwhile considering and being aware that Bad Things can still happen. And that Open JDK and the GPLv2 isn't necessarily the big saviour that some might think it is. (But lets be very thankful for Open JDK 6, and perhaps Apache Harmony's role in making it happen).

Summary

Well, thats my slightly legalistic take on how the Java SE 7 and ASF/Sun disputes might play out now Oracle is in the frame. And I know some may disagree with my take on this.

I'm still hopeful that Oracle will do the right thing - the signs from two years ago are good. But sometimes power corrupts. So lets stay vigilant.

Stephen Colebourne
Apache Software Foundation member, speaking personally
Sun Java Champion, speaking personally
Not a committer on Harmony or OpenJDK

The JCP doesn't exist!

The Java Community Process doesn't exist. At least not in the way you probably think it does. This surprised even me (although it shouldn't have), and I'll explain here just exactly what I mean.

JCP overview

The Java Community Process (JCP) is the standards process for Java. The FAQ gives the best summary:

Q: What is the JCP?
A: Since its introduction in 1998 as the open, participative process to develop and revise the Java™ technology specifications, reference implementations, and test suites, the Java Community Process (JCP) program has fostered the evolution of the Java platform in cooperation with the international Java developer community.
JCP FAQ

So, the JCP is a process to define standards around Java.

Each standard, known as a Java Specification Request(JSR), follows a clearly defined process. Each JSR is developed by an Expert Group (EG) who are hand selected by the Spec Lead. Most newer JSRs are now run in a more open and transparent manner.

JCP members

The JCP has many members, all of whom are listed online. Many of these members are individuals, who pay no annual fee. Not-for-profits pay $2000 per year, while companies pay $5000 per year. There are currently over 1200 members.

JCP executive committee

While the JCP has many members, only a select few have real power. These are the members of the Executive Committees (EC). There is one EC for Java ME and one for Java SE/EE - each has 16 members.

Each EC member serves a three year term. Ten of the seats are 'ratified', which means that Sun picks who they are (based on a balanced community and regional representation) and the membership votes to approve or disapprove the selection. Five seats are available for a free vote of the JCP membership. One seat on each EC is the 'permanent' seat held by Sun. The current Java SE/EE members are:

• Apache Software Foundation
• Eclipse Foundation Inc.
• Ericsson
• Fujitsu Limited
• Google
• Hewlett-Packard
• IBM
• Intel
• Werner Keil (individual)
• Doug Lea (individual)
• Nortel Networks
• Oracle
• Red Hat middleware LLC
• SAP
• Springsource
• Sun Microsystems

JCP structure

This is the basic structure as it relates to JCP members:

The JCP is fundamentally a series of legal agreements (black lines) between Sun and each member of the JCP (orange sqaure). This has been described as a hub and spoke layout, and that is the obvious way to draw it.

The JCP is supported by a small Program Management Office which, amongst other tasks, assists JSR spec leads, handles the website and arranges Executive Committee meetings. The PMO is a department of Sun and funded by Sun (apart from the fees listed above):

Q: What is the Program Management Office (PMO)?
A: The Program Management Office is the group within Sun designated to oversee the Java Community Process and manage the daily running of the program. The actual development of the specification occurs within the Expert Groups.
JCP FAQ

Does the JCP really exist?

This sounds like a daft question!

But if you've been reading carefully you will have noticed that it really isn't.

Firstly, the PMO is named very carefully - its a "program" management office. Thats because the PMO is managing a program, not an organisation. The program is the JCP program of Sun.

Secondly, consider the name 'Java Community Process'. The clue is in the word "process". The JCP is just a process for defining standards. The JCP is not an organisation in its own right - it's a group within Sun.

Thirdly, consider the legal situation. All legal agreements (the JSPA) are signed between the member and Sun. There are no legal agreements between the member and the JCP, because the JCP is not a real organisation.

Not convinced? Look at the legal definitions for the JCP and its membership:

Java Community Process (JCP): The formal process described in this document for developing or revising Java technology specifications.
Java Community Process Member (Member): A company, organization, or individual that has signed the JSPA and is abiding by its terms.
JSPA legal document defining the JCP

It can't really be clearer - the JCP is a "formal process", not an organisation.

And note how clever the definition of "membership" is. You are a member simply because you signed a legal agreement (the JSPA). And remember that the legal agreement is actually with Sun, not the JCP (which you might naturally think).

Now, I don't know about you, but that isn't the kind of membership I normally think of, such as a sports club membership. Put simply, as a JCP "member" I'm not really a member of the JCP, because there is actually no such organisation.

By comparison, here is the definition of Eclipse:

The Eclipse Foundation was created in January 2004 as an independent not-for-profit corporation to act as the steward of the Eclipse community.
About Eclipse

As can be seen, "The Eclipse Foundation" is a real organisation - something entirely different to the JCP. Being a "member" of The Eclipse Foundation is something that would be very meaningful.

Summary

There is no JCP organisation. The JCP is simply a "process" to produce standards, not an actual standards organisation itself. (Even the Executive Committees are simply members that guide Java technology)

As it isn't an independent organisation with power, the "JCP" has been unable to effectively take action in the ASF vs Sun Apache Harmony dispute. The picture clearly shows how all legal agreements feed back to Sun, so no other party can influence the issue directly.

I have to admit that I've been a bit dense in not really grasping the lack of any JCP organisation until today. But I do wonder how many other developers have grasped it either?

Put simply, the JCP is NOT a standards organisation - because it isn't an organisation at all. In fact, there is only one organisation here - Sun - and they still control all the cards.

Stephen Colebourne
Apache Software Foundation member, speaking personally
Sun Java Champion, speaking personally
Not a committer on Harmony or OpenJDK

PS. I'd just like to note that the individuals that staff the JCP PMO have always been very helpful, in exactly the same way as the engineers developing JDK 7. This debate is all about Sun's management and how we really don't have an open standards process organisation.

No Java SE 7 - The ASF perspective

I've allowed my recent blog posts on the lack of a Java SE 7 platform JSR and the ASF-Sun dispute to simmer for a couple of weeks. Now, in this post, I'm going to draw on the public information available from the ASF on this topic.

Since all the quotes in this blog are taken from ASF sources (ie. one side of the dispute) I would encourage all readers to consider carefully whether they state fact or opinion. As always, I state that I'm an ASF member, but I'm speaking personally here. My aim is to bring this debate to a wider audience.

ASF board minutes

The ASF is an open and transparent organisation whenever possible. As such, the board meeting minutes for the ASF board are publicly available. As usual, certain items are kept private when sensitive, but most information is public, and the ASF-Sun dispute around Harmony can be traced.

VP of JCP [Geir]
Geir reported that involvement in the JCP regarding JSR's has been active and healthy. He also noted Harmony's intent to request and receive the TCK.
Meeting minutes, July 19th 2006

Here is the first reference to the ASF deciding to ask for the testing kit for Apache Harmony.

VP of JCP [Geir]
Further, we are currently working on the Java SE 5 TCK (for the eventual Apache Harmony project), which is on course for having identical terms as our existing Java EE license. We have been granted the scholarship for the support for Java SE 5 and I have had one draft of the TCK license, so there is measurable progress.
Meeting minutes, October 25th 2006

So, back in 2006 there is evidence that Sun was offering a license with "identical terms" to the other 25 JSR testing kits that the ASF uses.

VP of JCP [Geir]
In other areas, we are still negotiating with Sun regarding the Java SE TCK license (also known as the "JCK"). Discussions on appropriate terms seem to be nearing an impasse, with the current terms unacceptable to the ASF. There still is one more avenue of exploration, and if unsuccessful, will need to escalate inside Sun, or beyond.
Meeting minutes, January 17th 2007

Now, just a few months later it is clear that a problem has arisen.

VP of JCP [Geir]
Last month I attended the quarterly JCP EC f2f [face to face] meeting and was able to give a presentation on our JCK [testing kit] issue. I first described a hypothetical case that was "structurally" similar to ours - where an imple- mentor was being deliberately denied a usable TCK license due to the business strategy of the spec lead. I then argued why such behavior was prohibited by referring to the relevant sections of the JSPA. Once I felt that the basics were clearly explained, I conducted a "straw poll" (informal, unrecorded) and IMO demonstrated that the EC generally agreed with our assertion that the JSPA prohibits field of use limitations in a TCK license. Once this was estabilshed, I informed the EC of the basic details of our dipute with Sun, after which followed a vigorous and somewhat heated general discussion. I'm satisfied that our issue is clearly understood by the EC.
Meeting minutes, March 28th 2007
(Abbreviations expanded in square brackets)

The face to face meeting referred to is described in the JCP meeting minutes of February 27th 2007. It was at this meeting that we now have a record of the vote (in the ASFs favour).

The interesting aspect is how Geir indicates that he described the issue first without any reference to the specifics of the ASF case. Only then was the ASF issue raised. This could be interpreted such that the 'Field-of-use' clause could be used against any JCP member, not just the ASF.

VP of JCP [Geir]
In other related activities, we've asked the SFLC for help in this area, specifically to validate our interpretation that the offered TCK license is in violation of the JSPA from the perspective that the terms prevent us from distributing under an open source license, and also act as a liaison between us and Sun, as Eben is prominently positioned by Sun as a supporter of their recent OpenJDK project. They did agree with our interpretation of terms, and engaged in discussion with Sun (to no avail).

I've also asked some other industry notables that are independent from the ASF and the various commercial actors in the Java ecosystem to appeal to Sun on our behalf.
Meeting minutes, March 28th 2007

In this continuation, Geir clearly indicates that the ASF made efforts to use other groups to mediate before writing an open letter. Specifically mentioned are the Software Freedom Law Center and Eben Moglen.

VP of JCP [Geir]
The ASF has adopted the policy of voting "no" on any JSR for which Sun is the spec lead, starting with the Java EE 6 JSR. This is the only logical position for the ASF to take regarding this matter - we believe that Sun is in breach of the JSPA and public promises, and therefore shouldn't be allowed to start new JSRs in the JCP until the matter is resolved.
Meeting minutes, July 18th 2007

The ASF have now adopted their stance of voting 'no' to Sun spec lead JSRs.

VP of JCP [Geir]
While we have not received an official response from Sun, we do believe that the open letter has alerted the community to the problem. The most visible manifestation of this is how EC members are using a commitment for a FOU-free TCK license as a gating factor in voting decisions. See the recent Java EE 6 vote (http://www.jcp.org/en/jsr/results?id=4306)
Meeting minutes, July 18th 2007

It is noted how other JCP members (companies) are starting to push for FOU-free licenses.

VP of JCP [Geir]
We initiated a motion for vote by both Executive Committees of the JCP...
The vote was held by electronic ballot, and finished last week on Sept 13.
... The vote went as we expected, and I am satisfied with the result. For the record, the motion was as follows, bracketed by "-=-" :
-=-
This is a very important issue and the proposers of this motion recommend that each EC member use the ballot period to review this question within their respective companies and get whatever business or legal opinions they feel that they need to get in order to come back with a firm position.

Motion: We move that the ECs adopt the following statement:

"TCK licenses must not be used to discriminate against or restrict compatible implementations of Java specifications by including field of use restrictions on the tested implementations or otherwise. Licenses containing such limitations do not meet the requirements of the JSPA, the agreement under which the JCP operates, and violate the expectations of the Java community that JCP specs can be openly implemented."
-=-

Question was asked of Geir as to whether he expected Sun to change as a result of this vote. Geir seemed less hopeful than before.
Meeting minutes, September 19th 2007

As far as I can tell, the results of this vote are not present in the JCP minutes. It seems reasonable to assume that if Geir was "satisfied" with the result that the vote was in favour of the ASFs position.

VP of JCP [Geir]
The highlight of this month was the quarterly JCP EC F2F meeting held in Marlborough, a frigid suburb of Boston. Held in a barn heated by a woodstove in the corner, the closest seats to which where appropriated and viciously defended by the repectable gentlemen from IBM and the ASF, the meeting was very well attended (probably the best F2F attendence in literally years) and has been noted publicly elsewhere, the meeting concentrated on the "future of the JCP". Led by Doug Lea, it really was a refreshing and open discussion.

Due to the letter and spirit of the confidentiality restrictions of the event, I cannot report publicly any details other than what I did, but will file a note to the members list later on this week.

As part of the discussion, I noted that for whatever reason, Sun has not offered a JSR for Java SE 7, despite publicly talking about such a version, and creating a project at OpenJDK to work on the RI for the same. Given that it has been 12 months since the release of Java SE 6, it clearly was time to move with Java SE 7 in order to prevent Java from falling even further behind other computing platforms. I asked that Sun, as the spec lead for Java SE 6, to please bring forth a JSR for Java SE 7 for vote by the EC, and be sure that such JSR made it clear that there would be no FOU or other restrictions that could prevent any independent implementation from being distributed under the terms of choice by the implementor. Failing that, I noted that the ASF was ready to lead or co-lead such a JSR, and if vetoed by Sun due to their special rights under the JSPA, we'd lead JSRs to add new features and extensions to Java SE 6 as non-platform JSRs.
Meeting minutes, December 19th 2007
(My highlight)

As far as I know, this constitutes information not previously reported on. The ASF, via Geir, asked Sun to submit the Javs SE 7 platform JSR in December 2007. As an alternative, the ASF offered to lead the JSR themselves. Again, it highlights the connection between the Apache Harmony dispute and the Java SE 7 JSR.

VP of JCP [Geir]
It was made clear that all parties, including Sun, understand the seriousness of the situation and the firmness of resolve among many of the EC to resolve in a satisfactory manner
Meeting minutes, January 16th 2008

The point to note here is the "resolve" of other EC members, by which I'd interpret that the dispute is broader than just ASF vs Sun.

VP of JCP [Geir]
Over the last month, Sun offered the ASF a TCK license for Java SE that while a step in the right direction, failed to comply with the JSPA on a number of fronts and that nothwithstanding, placed requirements upon the ASF that weren't acceptable.
Meeting minutes, April 16th 2008

Sun did try to resolve the dispute here, but they clearly failed to address the root cause of the problem.

VP of JCP [Geir]
Still in the rabbit hole, we continue our discussions with Sun regarding the Harmony TCK license. I had a long meeting with Jeet Kaul at Sun during JavaOne, and I continue to appreciate his candor and directness, but am concerned about the ultimate outcome.

We currently have a proposed license that contains a mix of objectionable elements, including a kind of FOU limitation, a notice requirement and some discussion on the version of the TCK that would be licensed. Honestly, I'm getting tired of backing up.

The notice still doesn't conform to the "3 principles" that have been articulated on internal (and soon to be external) member and legal lists, so I haven't been formally wasting the board's or member's time with the minutea. ...
Meeting minutes, May 21th 2008

This provides more details on Sun's attempts to 'fix' the problem. The minutes indicate that the proposal from Sun wasn't a lot better than previous offerings.

VP of JCP [Geir]
On June 12, I formally told Sun that we are rejecting the then- current TCK license for Java SE as it contained requirements that were unacceptable to us, and in our opinion, incompatible with the spirit and letter of the JSPA as well as open standards.
Meeting minutes, June 25th 2008

The ASF formally rejects the latest attempt by Sun to 'fix' the problem.

VP of JCP [Geir]
We currently remain in a state of deadlock. The effect of this on those who know the details as well as the overall progress of Java itself has been fairly profound, and I'm looking for ways to get the information of the history and status out to the community.

I've thought about drastic action that we could take - such as exiting the JCP EC - but I think that given the reasons for Sun's lack of compliance with the JSPA in this matter, I don't think it would help.
Meeting minutes, August 20th 2008

Deadlock remains. But the effect is becoming more explicit.

VP of JCP [Geir]
The Incomprehensible Battle Against Stupidity and Evil (iBASE) (a.k.a. WOFTAM - "Waste of ... Time and Money") continues, with Sun standing firm with their continued refusal to supply a JSPA-compliant and open-source compatible TCK license for Apache Harmony. This was the only subject of any interest at the recent JCP Quarterly F2F. There is a growing recocnition that Sun's recalcitrance is causing growing harm to the Java platform as a whole, and EC members are eager for something to happen.
Meeting minutes, October 15th 2008

Are there really acronyms for the dispute now?! Again, the message is that Java itself is being damaged here.

VP of JCP [Geir]
Finally - after the last EC f2f meeting, the terms of the Java SE TCK license were made available to all EC members at my request. Many could not understand the problem with the NOTICE as it is deceptively similar to the language found in the JSPA. I've been challenging EC members to take a harder look with their legal team and decide if they would post such an item with their products and services at the request of Sun. So far, one of the major EC members who initially questioned our resistance to post the NOTICE has publicly reversed his position and is arguing on our behalf.
Meeting minutes, December 17th 2008

This seems interesting. Its clear that the exact wording of the terms in dispute appear to be fine at a first read. Yet, at least one corporate (and probably their legal department) has found an issue on closer examination.

VP of JCP [Geir]
0) At request of the JCP EC from the last quarterly meeting, Onno Kluyt of Sun and myself met in NYC to discuss the TCK dispute and see if any forward progress could be found. While we had a nice lunch, and it's always nice to see Onno, nothing was achieved. Sun still believes that it's critical for their business to force us to post the NOTICE (implying, to me, that Sun certainly believes that the NOTICE will materially affect our users - logically, there could be no other reason for them to persist with this destructive dispute.) I iterated our position that we only distribute software under the Apache License, and because the NOTICE would be adding additional terms to that license, it was unacceptable.

1) Quarterly JCP EC Meeting : I attended by phone (there were several that did). Amazingly, the Apache-Sun TCK dispute is still a hot topic. I reported the lack of progress from my meeting with Onno, and participate in a general, sometimes heated, discussion. I iterated the following points :
* This is an active dispute - it is not something of only historical nature, and thus any proposal attempts to fix Java SE licensing in the JCP must address current, active, ongoing problems
* Apache's position is that implementors should be able to distribute implementations under terms that they choose, and Sun trying to limit the license under which the ASF distributes it's software is no different from Sun telling a competitor how much to charge or what markets are acceptable, neither of which would be tolerated.
* Apache isn't looking for an open source TCK or the right to distribute the TCK to anyone else - it would be only for our own use in the Harmony project. We have always worked with Sun's closed source TCKs under confidentiality constraints as specified by Sun.
* Apache would have a tough time trusting any statements from Sun about future licensing as our experience with the Java SE TCK shows that while such statements may be made with the best of intentions, Sun has demonstrated that commitments may not be met.
Meeting minutes, January 21st 2009

This is a fairly detailed restatement of the issues involved. Motivation is indicated for Sun to continue in this battle despite the negative impact. Specifically it is mentioned that the limitations being applied are fundamentally no different to Sun trying to tell a competitor "how much to charge or what markets are acceptable".

It is also interesting to note that after over two years, the EC are still interested in this topic. And that the ASF have, perhaps understandable, deep distrust of any future promises made by Sun.

ASF JCP mailing list

Another source of ASF info is the open mailing list - jcp-open.

> What I don't quite yet understand is why the terms of the TCK were not
> disclosed when we first announced our intentions to implement JSR-176.

Because we understand the JSPA and after N TCK licenses (I forget what N is) I never suspected this would happen. As a matter of fact, the first draft of the TCK license we received was just fine - it was like every other TCK license we received. THe only thing it was missing was the "list price" for the JCK in the event that the ASF converted to a for-profit entity. :/

At the time, I suggested that they put any number they wanted (I suggested $1B) given the likelihood of the ASF being a for-profit. But someone wanted to be a sticker for process, and my assessment is that once the sales people caught wind of this, that's when this problem started.
Geir on jcp-open, May 13th 2007
(My highlights)

So, according to Geir (speaking personally on this occasion), the original version of the testing kit license was fine. It was normal. There was no dispute.

And then something changed which has caused this ongoing dispute. It seems fair to say that the dispute is quite deliberate on the part of Sun.

Summary

So, this is another action packed tour into the world of the lack of a Java SE 7 JSR and the ASF-Sun dispute. I know that there is lots of detail here (and more that I have omitted). And in some ways it doesn't tell us that much more than the previous blogs. But it does help to strengthen the overall view once you find the same information from different sources.

As always, please read and check my sources as this is a controversial issue, and especially as the quotes in this blog (but hopefully not the analysis) is solely from the perspective of the ASF.

Finally, I noticed that Joe Darcy's talk at JavaOne now has a title:

JavaOne 2009: Small Language Changes in JDK™ Release 7

Yep - no 'Java SE 7' in sight...

Stephen Colebourne
Apache Software Foundation member, speaking personally
Sun Java Champion, speaking personally
Not a committer on Harmony or OpenJDK

Shedding new light on No Java SE 7 JSR

There is a direct connection between the 'Sun vs Apache Harmony' dispute and the lack of a Java SE 7 platform JSR. Using newly available evidence I hope to shed new light on what that link is.

Back in the day

Its interesting to read again how Harmony was welcomed when it first started:

Part of the reason these Apache projects are possible is that the JCP has been working over the last few years to clarify the role of open source projects and also to improve overall transparency. For example there were changes to the intellectual property rules in JCP 2.5 specifically designed to allow open source implementations. There have also been a number of transparency improvements in JCP 2.6 to allow earlier access to specification information. In addition to those JCP updates, Sun created a scholarship program under which we have provided Apache (and other not-for-profit groups) with free access and free support for Sun's Java compatibility test suites.
...
The licensing rules for J2SE 5.0 were carefully designed to allow independent, compatible open-source implementations of the J2SE specification. Personally, I am not entirely sure if the world really needs a second J2SE implementation, but at the same time I am also glad to see that all the effort we put into getting the rules and the licensing issues straightened out is actually proving useful!
Graham Hamilton's blog VP and Fellow at Sun responsible for Java at the time, written May 7, 2005.
(My emphasis)

And lets look at how that was reiterated in the business terms for Java SE 6:

JSR 270: JavaTM SE 6 Release Contents
...
2006.10.24:
2.18 Please provide a description of the business terms for the Specification, RI and TCK that will apply when this JSR is final.
...
7. Nothing in the licensing terms will prevent open source projects from creating and distributing their own compatible open source implementations of Java SE 6, using standard open source licenses. (Yes, you can create your own open source implementation of Java SE 6 if you really want to. But we're also doing everything we can to make it easy for you to use the original RI sources! See http://jdk6.dev.java.net.)

JSR-270
(My emphasis)

How times change. Update: The 'how times change' reference was a comparison of both of the above quotes to the dispute that followed, not a comparison of the two quotes. Sorry for the confusion.

Trying to resolve the dispute

As is clear, Apache claim that Sun is in violation of the JSPA contract for both Java SE 5 and Java SE 6 platform specifications. The comments above also indicate how matters changed over the course of time.

Given the dispute, it is clear that Apache would object to there being a Java SE 7 platform JSR without changes being made to resolve the issue. However, Apache is just one voice in the industry. The power in the JCP is held in two 16 member Executive Committes - one for Java SE/EE and one for Java ME. Clearly, therefore, Apache could be outvoted 15-1 on the SE/EE committee, and it has no vote on the ME committee.

So, why hasn't Sun just submitted the Java SE 7 platform JSR and accepted/ignored Apache's No vote?

Well, until recently, I thought that the reasons were all private information. However, it turns out that since September 2008, the Executive Committee meeting minutes have been openly published. As such, we can see how the Executive Committees have tried to resolve the issue. As far as I know, no journalist or blog has yet picked up on this.

Given the controversial nature of this, I strongly encourage anyone following this to read the minutes themselves. Its a good insight into the the way in which the JCP operates (a style probably typical of any commercial based standards body). In this section I will extract some key elements, and make some minimal comment, but please read the full documents to make up your own mind.

Meeting summary - February 27th 2007
Geir Magnusson presented on a hypothetical situation regarding a company pursuing a certain Java-related business opportunity and whether the JSPA allows licenses that could affect that opportunity. During the meeting Geir asked for a straw poll on the following statement:
"The Executive Committee of the JCP supports the Apache Software Foundation's assertion that placing limitations on field of use for compatible implementations of Java specifications is contrary to the letter of the JSPA and the spirit of the JCP as a creator of open specifications."
Voting "yes": 17
Voting "no": 2
Voting "abstain": 3

(Geir Magnusson is the Apache representative on the Executive Committee)
(Note that the official document status is draft and unapproved, however it is publicly linked from the JCP website)

Note the 'straw poll' explicitly mentions the field of use restriction that Sun used against Harmony. This vote represents the collective wisdom of the two Executive Committees - Java ME and Java SE/EE - 32 votes in total, however there were 10 absentees.

Note that there were two 'no' votes.
Also note that Sun sits on both executive committees and thus has two votes.
But there is no other information to tie those two facts together.

Meeting summary - December 7th 2007
Resolution 1 (proposed by Oracle, seconded by BEA)
"It is the sense of the Executive Committee that the JCP become an open independent vendor-neutral Standards Organization where all members participate on a level playing field with the following characteristics:
* members fund development and management expenses
* a legal entity with by-laws, governing body, membership, etc.
* a new, simplified IPR Policy that permits the broadest number of implementations
* stringent compatibility requirements
* dedicated to promoting the Java programming model
Furthermore, the EC shall put a plan in place to make such transition as soon as practical with minimal disruption to the Java Community."

Vote tally from SE/EE Executive Committee:
Voting "yes": 13 (Apache, BEA, Fujitsu, Google, HP, IBM, Intel, Doug Lea, Nortel, Oracle, Red Hat, SAP, Hani Suleiman)
Voting "abstain": 1 (Sun)
Vote tally from ME Executive Committee:
Voting "yes": 13 (BenQ, Jean-Marie Dautelle, Ericsson, IBM, Intel, Nokia, Orange, RIM, Samsung, Siemens, Sony-Ericsson, Time Warner, Vodafone)
Voting "abstain": 1 (Sun)

Note how the proposal is explicit about the JCP becoming 'independent' and 'vendor-neutral'. Note also how the proposal accepts that the costs of such a body would have to be shared by its members. Again, the two Executive Committees - Java ME and Java SE/EE - clearly voted in favour of a resolution to further open the JCP. The sole exception was Sun, who abstained.

Meeting summary - December 7th 2007
Resolution 2 (proposed by BEA, seconded by Intel)
"The SEEE EC requests Sun to submit a Java SE7 JSR with the following characteristics:
* EG members will share costs by contributing to the TCK and RI submitted under common licensing terms that will allow any implementer to use them;
* accompanied by public spec, tck, ri licenses at JSR initiation;
* licenses do not contain Field of Use restrictions on spec implementations.
* the JSR is operated by the Spec Lead under democratic principles (e.g. Java SE6)
Furthermore, from the time of submission, TCK license(s) for Java SE5 and later will be offered without field of use restrictions on spec implementations enabling the TCK to be used by organizations including Apache."

Vote tally from SE/EE Executive Committee:
Voting "yes": 10 (Apache, BEA, Fujitsu, Google, Intel, Doug Lea, Nortel, Oracle, SAP, Hani Suleiman)
Voting "abstain": 4 (Sun, HP, IBM, Red Hat)

So, the SE/EE Executive Committee clearly voted in favour (10-4) of a resolution in favour of an open Java SE 7 specification, and for Apache to receive the testing kit for Java SE 5 and 6. There is no information available as to the details of the discussion, nor or the reasons why individual participants voted for or against.

Meeting summary - April 8th 2008
Sun's commitments on JCP reform
Onno Kluyt gave a presentation in which he outlined a series of JCP reforms that Sun is willing to commit to. The presentation was followed by an extensive discussion.
Java SE 7 JSR
Roberto Chinnici summarized Sun's plans for the Java SE 7 JSR, expressing the hope that this would be submitted and approved soon.

We can note that back in April 2008 Sun were making efforts to resolve the issues. Also note that Sun expressed the desire to submit the Java SE 7 JSR 'soon'. I'd also note that April 2008 is just before JavaOne 2008, which was in May - a time when Sun frequently makes big announcements.

Meeting summary - June 24th 2008
Geir Magnusson and Onno Kluyt reported that Sun and Apache have failed to reach agreement on the terms under which the JCK would be licensed to Apache for use in the Harmony project. They also reported that no further discussions are planned. Onno stated that Sun has no immediate plans to submit the JSR for Java SE 7. Rather, Sun plans to continue SE 7 development work through the OpenJDK project and through component JSRs that are already in progress. He expressed the hope that when the JSR is submitted at some point in the future the EC will approve it.

EC members expressed their disappointment at this outcome, and discussed the possible implications without reaching any firm conclusions.

Sun now have 'no immediate plans' to submit the Java SE 7 platform JSR. It is fair to ask what happened between April and June to cause the submission of the JSR to go from 'soon' to 'no immediate plans'. Sadly, there is no published information to answer that question, so I'll leave that open to debate (I can't comment further).

Further, it can be seen that Sun are beginning to focus on working towards version 7 using Open JDK instead.

Meeting minutes - September 25th 2008
Patrick gave an update on the upcoming elections, and informed the ECs of Sun's nominations:
...
Several members expressed concern that they weren't consulted about Sun's nominations. Patrick said that Sun was unwilling to make concessions on "control points" (such as the right to nominate for EC elections) without something in return. When members asked what concessions Sun wanted, Patrick responded that Sun expressed the desire to feel confident that the Java SE7 JSR would be considered by EC members without allowing the dispute about TCK licensing between Apache and Sun to obstruct it.

This passage shows Sun as being concerned about whether the Java SE 7 JSR would be considered without the TCK dispute being resolved. Further, they link certain aspects of Sun control in the JCP (ie. who gets a guaranteed seat on the JCP) to the Java SE 7 debate. I'm not going to speculate on the details of what is clearly a politically-charged passage.

Meeting minutes - September 25th 2008
Private Session
The ECs then went into private session for a discussion on the negotiations between Apache and Sun.
After these discussions it was agreed that Apache and Sun would explore the possibility of arbitration to resolve their differences.

Arbitration is mentioned as one possible solution.

Compatibility Rules discussed at the meeting on September 25th 2008
Slide 13:
Compatibility is a contractual obligation.
- You must not ship incompatible products.
-- Compatibility is binary.
- You can't be "almost compatible" or "a little bit incompatible".
- The JCK contains 120,000 test cases.
-- If you pass 119,999 you are not compatible.
-- If you pass 120,000 and don't meet all the other requirements you are not compatible.

This hammers home the point that any implementation of any JSR must be fully compatible. And that it is a contractual obligation. It also shows the size of the Java SE platform JSR testing kit.

Meeting minutes - January 13th 2009
Private session
The ECs then went into private session to discuss the status of the Sun-Apache negotations. During this session members had a lively discussion, and made some proposals that will be brought to the attention of Sun's management

Again, discussions to resolve the issue by the Executive Committee are still ongoing in January 2009.

There are no further minutes publicly available - January 2009 is the last published date. However, the next meeting is on April 9th 2009, so we should expect to see the minutes from February 25th approved at the April meeting very soon.

Any clearer?

I hope that these quotes, and the original minutes, provide a deeper insight into the dispute, and into how the Java SE 7 is affected. I'll draw out three points:

Firstly, that this isn't just Apache vs Sun. Other members of the Executive Committee have been willing to side with Apache and vote against Sun (February and December 2007). Bear in mind that those members have access to even more information than is public here in order to judge which way to vote. Remember though that those votes are not-binding. Due to the structure of the JCP, the other Executive Committee members cannot actually intercede on behalf of Apache.

Secondly, from the September 2008 meeting, there is distinct evidence (but not proof) that Sun feels unable to get a Java SE 7 JSR passed until the Harmony dispute is resolved. We can also note the difference in tone between April 2008 and June 2008, where a Java SE 7 platform JSR went from 'soon' to 'no immediate plans', and Sun's focus became clearer around using Open JDK instead of a JSR.

Finally, it is clear that discussions have continued in one form or another for the entire time of the dispute. It isn't the case that the Apache open letter was published and then forgotten or ignored.

Despite all the efforts, there has been no solution. And as such there is still no Java SE 7 platform specification. Thus, I hope its a little clearer why I'm arguing that, in all likelihood, we will get a JDK 7 implementation, but not a Java SE 7 platform open specification.

Summary

Personally, I'm glad these meeting minutes are now public. Everyone can now see the behind-the-scenes discussion for themselves and make up their own minds as to the true state of the debate. In addition, you can freely judge if my blogging has been factual or not.

Finally, for anyone still wondering why Apache Harmony might be of any value at all, consider this:

Apache Harmony: Thanks for the TreeMap Work!
I'd like to thank Apache Harmony for their JDK library performance efforts. We were given a tip that the Harmony folks were doing some interesting work with the TreeMap collection class, and low and behold they were.
...
Controlled measurements showed the "fat" TreeMap significantly faster with in order traversals, and improved our SPECjbb2005 score by a solid 3-5% depending on the platform.
...
Considering the potential performance gain we started the effort of porting the Apache Harmony TreeMap code to JDK 6.
...
The new TreeMap is included in JDK 6 Update 6 Performance Release
David Dagastine

So, Sun's JDK runs faster today as a direct result of code written in Apache Harmony. If you use Sun's JDK 6 update 6 or later, Apache Harmony has already benefited you. And that just one of the benefits of open specifications and competing implementations.

Stephen Colebourne
Apache Software Foundation member, speaking personally
Sun Java Champion, speaking personally
Not a committer on Harmony or OpenJDK