Monday, 23 November 2009

More detail on Closures in JDK 7

This blog goes into a little more detail about the closures announcement at Devoxx and subsequent information that has become apparent.

Closures in JDK 7

At Devoxx 2009, Mark Reinhold from Sun announced that it was time for closures in Java. This was a big surprise to everyone, and there was a bit of a vacuum as to what was announced. More information is now available.

Firstly, Sun, via Mark, have chosen to accept the basic case for including closures in Java. By doing so, the debate now changes from whether to go ahead, to how to proceed. This is an important step.

Secondly, what did Mark consider to be in and what out? Well, he indicated that non-local returns and the control-invocation statement were out of scope. There was also some indication that access to non-final variables may be out of scope (this is mainly because it raises nasty multi-threading Java Memory Model issues with local variables).

In terms of what was included, Mark indicated that extension methods would be considered. This would be necessary to provide meaningful closure style APIs since the existing Java collection APIs cannot be altered. The result of what was in was being called "simple closures".

Finally, Mark offered up a possible syntax. The syntax he showed was very close to FCM:

  // function expressions
  #(int i, String s) {
    System.println.out(s);
    return i + str.length();
  }

  // function expressions
  #(int i, String s) (i + str.length())
  
  // function types
  #int(int, String)

As such, its easy to say that Sun has "chosen the FCM proposal". However, with all language changes, we have to look at the semantics, not the syntax!

The other session at Devoxx was the late night BOF session. I didn't attend, however according to Reinier Zwitserloot, Mark indicated that Exception transparancy might not be essential to the final proposal.

According to Reinier, Mark also said "It is not an endorsement of FCM. He was very specific about that." I'd like to consider that in a little more detail (below).

The final twist was when a new proposal by Neal Gafter was launched which I'm referring to as the CFJ proposal. After some initial confusion, it became clear that this was written 2 weeks before Devoxx, and that Neal had discussed it primarily with James Gosling on the basis of it being a "compromise proposal". Neal has also stated that he didn't speak to Mark directly before Devoxx (and nor did I).

There are a number of elements that have come together in the various proposals:

  CICE BGGA 0.5 FCM 0.5 CFJ 0.6a Mark's announcement
(Devoxx summary)
Literals for reflection - - Yes - No
(No info)
Method references - - Yes Yes Worth investigating
(No info)
Closures assignable to
single method interface
Yes Yes Yes Yes Yes
(No info)
Closures independent of
single method interface
- Yes Yes Yes Yes
Access non-final local variables Yes Yes Yes Yes No
(Maybe not)
Keyword 'this' binds to
enclosing class
- Yes Yes Yes No info
(Probably yes)
Local 'return'
(binds to closure)
Yes - Yes Yes Yes
Non-local 'return'
(binds to enclosing method)
- Yes - - No
Alternative 'return' binding
(Last-line-no-semicolon)
- Yes - - No
Exception transparancy - Yes Yes Yes No info
(Maybe not)
Function types - Yes Yes Yes Yes
Library based Control Structure - Yes - - No
Proposed closure syntax Name(args) {block} {args => block;expr} #(args) {block} #(args) {block}
#(args) expr
#(args) {block}
#(args) (expr)

A table never captures the full detail of any proposal. However, I hope that I've demonstrated some key points.

Firstly, the table shows how the CFJ is worthy of a new name (from BGGA) as it treats the problem space differently by avoiding non-local returns and control invocation. Neal Gafter is also the only author of the CFJ proposal, unlike BGGA.

Secondly, it should be clear that at the high level, the CFJ and FCM proposals are similar. But in many respects, this is also a result of what would naturally occur when the non-local returns and control invocation is removed from BGGA.

Finally, it should be clear that it is not certain that the CFJ proposal meets the aims that Mark announced at Devoxx ("simple closures"). There simply isn't enough hard information to make that judgement.

One question I've seen on a few tweets and blog posts is whether Mark and Sun picked the BGGA or FCM proposal. Well, given what Mark said in the BOF (via Reinier), the answer is that neither was "picked" and that he'd like to see a new form of "simple closures" created. However, it should also be clear that the choices announced at Devoxx were certainly closer to the FCM proposal than any other proposal widely circulated at the time of the announcement.

At this point, it is critical to note that Neal Gafter adds a huge amount of technical detail to each proposal he is involved with - both BGGA and CFJ. The FCM proposal, while considerably more detailed than CICE, was never at the level necessary for full usage in the JDK. As such, I welcome the CFJ proposal as taking key points from FCM and applying the rigour from BGGA.

As of now, there has been relatively little debate on extension methods.

Summary

The key point now is to focus on how to implement closures within the overall scope laid out. This should allow the discussion to move forward considerably, and hopefully with less acrimony.

8 comments:

  1. Good summary, Stephen.

    Although at this point I guess we're really all waiting for Mark to publish a little more detail on his plans.

    ReplyDelete
  2. Very good writeup - thanks for putting this together.

    "There was also some indication that access to non-final variables may be out of scope"

    Not really closures then, is it? After all, the ability to capture and manipulate free variables is where the term "closure" comes from. That's not to say I wouldn't like code-blocks in Java - just sayin'.

    This whole thing is an unfortunate mess - don't know why Sun continues to pull these two-minute-warning drills; transparency is not their strong suit.

    ReplyDelete
  3. About 90% of our local variables are final anyway. Our company has a policy on that. We've identified countless bugs in old code that didn't have that restriction.

    ReplyDelete
  4. I agree with R.J. Lorimer. Let's be clear: that's first class functions for Java, and not "may be not" closures for Java! ;-)

    ReplyDelete
  5. This proposal seems realistic and elegant. But I really can't understand the need to introduce a second notation (expressions) only to avoid a single word (return) as I couldn't understand the need for the "several returns" mess in BGGA !
    I thought the point was to keep it simple.
    And why not use a new "function" keyword ? This is what it is, isn't it ? Has the enum keyword caused so much trouble in Java/JDK 6 ? Are you afraid of the "Java becomes JavaScript" comments ?

    But, first of all: remove expressions.

    ReplyDelete
  6. Stephen Colebourne25 November 2009 at 01:59

    I agree that without capture of free local variables that the resulting construct isn't a closure as generally defined. Of course, in the broader debate in Java-land, the actual definition of closure has on occasion been ignored. I suspect that the omission of access to free local variables won't hugely damage the proposal.

    The expression form was something new, and aims to handle the common case where the 'closure' contains just a single expression. I think its a neat solution, although everything would work fine were it to be dropped, and only blocks supported.

    I oppose the use of a keyword, because one key aspect of 'closures' is their brevity. If implemented correctly, then they will be used all over the place, and keywords would be intrusive. I should also note that Sun have a policy of no new keywords (only context sensitive psuedo keywords).

    ReplyDelete
  7. The key phrase from Mark Reinhold's blog (http://blogs.sun.com/mr/entry/closures) is:

    "the principal use case of parallel programming"

    previously it was merely an important use case and not the one getting the most comments. In this use case the capture of free local variables is not merely useless but positively dangerous.

    ReplyDelete
  8. Shouldn't the JCP, and not Sun, decide on the future of Java (including closures support)?

    ReplyDelete