Tuesday, 19 February 2008

Evaluating BGGA closures

The current vote forces us all to ask what proposal is best for the future of Java. Personally, I don't find the BGGA proposal persuasive (in its current form). But exactly why is that?

Touchstone debate

The closures debate is really a touchstone for two other, much broader, debates.

The first "what is the vision for Java". There has been no single guiding vision for Java through its life, unlike other languages:

In the C# space, we have Anders. He clearly "owns" language, and acts as the benevolent dictator. Nothing goes into "his" language without his explicit and expressed OK. Other languages have similar personages in similar roles. Python has Guido. Perl has Larry. C++ has Bjarne. Ruby has Matz.

The impact of this lack of guiding vision is a sense that Java could be changed in any way. We really need some rules and a guiding Java architect.

The second touchstone debate is "can any language changes now be successfully implemented in Java". This is generally a reference to generics, where erasure and wildcards have often produced havoc. In the votes at Javapolis, 'improving generics' got the highest vote by far.

The result of implementing generics with erasure and wildcards has been a loss of confidence in Java language change. Many now oppose any and all change, however simple and isolated. This is unfortunate, and we must ensure that the next batch of language changes work without issues.

Despite this broader debates that surround closures, we must focus on the merits of the individual proposals.

Evaluating BGGA

BGGA would be a very powerful addition to Java. It contains features that I, as a senior developer, could make great use of should I choose to. It is also a well written and thought out proposal, and is the only proposal tackling some key areas, such as exception transparancy, in detail.

However, my basic issue with BGGA is that the resulting proposal doesn't feel like Java to me. Unfortunately, 'feel' is highly subjective, so perhaps we can rationalise this a little more.

Evaluating BGGA - Statements and Expressions

Firstly, BGGA introduces a completely new block exit strategy to Java - last-line-no-semicolon expressions. BGGA uses them for good reasons, but I believe that these are very alien to Java. So where do they come from? Well consider this piece of Java code, and its equivalent in Scala:

 // Java
 String str = null;
 if (someBooleanMethod()) {
  str = "TRUE";
 } else {
  str = "FALSE";
 }
 
 // Scala
 val str = if (someBooleanMethod()) {
  "TRUE"
 } else {
  "FALSE"
 }

There are two points to note. The first is that Scala does not use semicolons at the end of line. The second, and more important point is that the last-line-expression from either the if or the else clause is assigned to the variable str.

The fundamental language level difference going on here is that if in Java is a statement with no return value. In Scala, if is an expression, and the result from the blocks can be assigned to a variable if desired. More generally, we can say that Java is formed from statements and expressions, while in Scala everything can be an expression.

The result of this difference, is that in Scala it is perfectly natural for a closure to use the concept of last-line-no-semicolon to return from a closure block because that is the standard, basic language idiom. All blocks in Scala have the concept of last-line-no-semicolon expression return.

This is not the idiom in Java.

Java has statements and expressions as two different program elements. In my opinion, BGGA tries to force an expression only style into the middle of Java. The result is in a horrible mixture of styles.

Evaluating BGGA - Non local returns

A key reason for BGGA using an alternate block exit mechanism is to meet Tennant's Correspondance Principle. The choices made allow BGGA to continue using return to mean 'return from the enclosing method' whilst within a closure.

There is a problem with using return in this way however. Like all the proposals, you can take a BGGA closure, assign it to a variable and store it for later use. But, if that closure contains a return statement, it can only be successfully invoked while the enclosing method still exists on the stack.

 public class Example {
  ActionListener buttonPressed = null;
  public void init() {
   buttonPressed = {ActionEvent ev => 
     callDatabase();
     if (isWeekend() {
      queueRequest();
      return;
     }
     processRequest();
   };
  }
 }

In this example, the init() method will be called at program startup and create the listener. The listener will then be called later when a button is pressed. The processing will call the database, check if it is weekend and then queue the request. It will then try to return from the init() method.

This obviously can't happen, as the init() method is long since completed. The result is a NonLocalTransferException - an unusual exception which tries to indicate to the developer that they made a coding error.

But this is a runtime exception.

It is entirely possible that this code could get into production like this. We really need to ask ourselves if we want to change the Java programming language such that we introduce new runtime exceptions that can be easily coded by accident.

As it happens, the BGGA specification includes a mechanism to avoid this, allowing the problem to be caught at compile time. Their solution is to add a marker interface RestrictedFunction to those interfaces where this might be a problem. ActionListener would be one such interface (hence the example above would not compile - other examples would though).

This is a horrible solution however, and doesn't really work. Firstly, the name RestrictedFunction and it being a marker interface are two design smells. Secondly, this actually prevents the caller from using ActionListener with return if they actually want to do so (within the same enclosing method).

The one final clue about non local returns is the difficulty in implementing them. They have to be implemented using the underlying exception mechanism. While this won't have any performance implications, and won't be visible to developers, it is another measure of the complexity of the problem.

In my opinion, allowing non local returns in this way will cause nothing but trouble in Java. Developers are human, and will easily make the mistake of using return when they shouldn't. The compiler will catch some cases, with RestrictedFunction, but not others, which will act as a further level of confusion.

Evaluating BGGA - Functional programming

Java has always laid claim to be an OO language. In many ways, this has often been a dubious claim, especially with static methods and fields. Java has never been thought of as a functional programming langauge however.

BGGA introduces function types as a key component. These introduce a considerable extra level of complexity in the type system.

A key point is that at the lowest compiler level, function types are no different to ordinary, one method, interfaces. However, this similarity is misleading, as at the developer level they are completely different. They look nothing like normal interfaces, which have a name and javadoc, and they are dynamically generated.

Moreover, function types can be used to build up complex higher-order function methods. These might take in a function type parameter or two, often using generics, and perhaps returning another function type. This can lead to some very complicated method signatures and code.

 public <T, U> {T => U} converter({=> T} a, {=> U} b, {T => U} c) {
   return {T t => a.invoke().equals(t) ? b.invoke() : c.invoke(t)};
 }

It shoud be noted that FCM also includes function types, which are named method types. However, both Stefan and myself struggle with the usability of them, and they may be removed from a future version of FCM. It is possible to have FCM inner methods, method references and method literals without the need for method types.

Function types have their uses, however I am yet to be entirely convinced that the complexity is justified in Java. Certainly, they look very alien to Java today, and they enable code which is decidedly hard to read.

Evaluating BGGA - Syntax

The syntax is the least important reason for disliking BGGA, as it can be altered. Nevertheless, I should show an example of hard to read syntax.

 int y = 6;
 {int => boolean} a = {int x => x <= y};
 {int => boolean} b = {int x => x >= y};
 boolean c = a.invoke(3) && b.invoke(7);

Following =>, <= and >= can get very confusing. It is also a syntax structure for parameters that is alien to Java.

Summary

As co-author of the FCM proposal it is only natural that I should want to take issue with aspects of competitor proposals. However, I do so for one reason, and that is to ensure that the changes included in Java 7 really are the right ones. Personally, I believe that BGGA attempts something - full closures - that simply isn't valid or appropriate for Java.

It remains my view that the FCM proposal, with the optional JCA extension, provides a similar feature set but in a more natural, safe, Java style. If you've read the proposals and are convinced, then please vote for FCM at java.net.

Opinions welcome as always :-)

References

For reference, here are the links to all the proposals if you want to read more:

  • BGGA - full closures for Java
  • CICE - simplified inner classes (with the related ARM proposal)
  • FCM - first class methods (with the related JCA proposal)

27 comments:

  1. My only objection to BGGA is that the syntax sucks. Really. And - obviously - no one is going to change that because of amazingly stupid design at the start of java (separate namespaces for functions and variables anyone?)

    Its easier in C to have a callback defined as a method. Also cleaner.

    BGGA examples seem to have an unhealthy obsession with local closure creation instead of separation of implementation and instantion too.

    ReplyDelete
  2. Would it be so awful if everything was turned into a expression? There are no backward code incompatibilities right?

    ReplyDelete
  3. How's introducing closures a "simple and isolated" change? Instead of ensuring "that the next batch of language changes work without issues," current issues should be addressed first. The way I see it, if 40% of the community are against closures, then none of the competing proposals should go through; at least not in their current form. If there's an overwhelming demand to address current problems (e.g. generics), then these should be given first priority. What we have instead, is various parties pushing their own agenda.

    ReplyDelete
  4. WOW!
    What a nice and clear presentation of BGGA issues.
    Bravo!

    ReplyDelete
  5. I've enjoyed each of your posts comparing closure proposals and have already voted for FCM. :-) A question arose today about how FCM closures would interact with generics. Could do something like ask for a constructor for some implementation of an interface?

    E.g.
    public void doSomething(#(T()) ref){...}

    ReplyDelete
  6. Hi stephen, nice post.

    About the benelovent dictator, Java has Gosling
    see http://blogs.sun.com/jag/entry/closures,
    its last blog about closure.

    In my opinion, Java is currently in a transition phase from a cathedral organization to a bazaar one, it takes time and the way closure will or will not be integrated in jdk7 can be seen as a test of the new organisation.

    You are right about statement & expression, Java is neither Scala nor JavaFX.

    About non-local returns, one can decide to add a super class, non accessible to Java but accessible a bytecode level, to Throwable and makes NonLocalTransferException inherits from that class.
    Furthermore, ActionListener is a listener, a callback that can be called by another thread that the one that creates the listener.
    Having an interface that marks this kind of objects is a good idea.
    I just think that RestrictedFunction can be renamed but the concept behind is interresting because it leads to better code.

    Functional programming: BGGA function types are translated to generics so it doesn't add complexity to the type system.
    But it clearly adds a level of indirection directly into the programmer mind :)

    If you find a clean way to specify such function types using names, I will cast my vote for a mix between BGGA and FCM.

    Rémi

    ReplyDelete
  7. Excellent. A fair evaluation of the BGGA proposal. Thanks in particular for highlighting its ugly syntax alien to Java. It isn't a trivial issue, mind you. This paragraph should have been in font size 32, bold:

    int y = 6;
    {int => boolean} a = {int x => x <= y};
    {int => boolean} b = {int x => x >= y};
    boolean c = a.invoke(3) && b.invoke(7);

    "Following =>, <= and >= can get very confusing. It is also a syntax structure for parameters that is alien to Java. "

    ReplyDelete
  8. Great writeup Stephen. I believe you and Josh have more in common than you might think. I also believe that you should cooperate with Josh to create a single strong proposal to counter BGGA.
    Understanding that complexity will only benefit the really interested developers is the key piece of information here. Both you and Josh have that same view and it would be a shame if BGGA became the implemenation just because there were two individually competing, but better, proposals.

    ReplyDelete
  9. Stephen Colebourne20 February 2008 at 15:21

    whatever: It would be interesting to pursue the idea of converting all Java statements to allow them to be expressions. I suspect that it isn't actually pratical - and it would definitely change the feel of Java.

    Arman: You misunderstood the "simple and isolated" comment, which was a separate point to closures (ie. people are now objecting to any change, even something like a new string literal form.

    Frederic, Bharath, Mikael: Thanks for the kind words.

    ReplyDelete
  10. Converter is a nice example. It both demonstrates the peculiarity of the syntax compared to plain old Java, and shows off how to handle poles when mapping sets (e.g., how to handle "blowing up" when approaching divide-by-0 case).

    This is a very elegant, enlightening post!

    ReplyDelete
  11. I heartily recommend to the other commenters Stephen's note that whining about BGGA syntax isn't very productive, because it can be changed.

    Closures are easily complex enough to need crucial issues sorted out first, before we dig down into syntax considerations.

    Personally I consider the two real issues at this point:

    1. function types. (The notion that {int, int => void} is just as legal a type as 'String'). Introducing them turns java into a structural/nominal typing hybrid, and that's an enormous change. Right now -any- type in java has a 'home' with javadocs and a place where you can easily run 'come from' queries in your editor. function types don't have that, and they have no proper namespace either. It's big, in other words. Do we really want to go there? As Stephen said, FGM has them but can live without them, and CICE doesn't have them but it can be faked a little by adding a new package with a bunch of Function interfaces that you can choose from.

    2. 'long' returns. They are part of BGGA. They can be added without any problem to the CICE proposal, using the same exception-based mechanism BGGA uses, and e.g. the same syntax used by long breaks/continues (return someLabel: theReturnValue;). This is normally a minor consideration, but as Stephen correctly identified in this post, the fallout of doing it right is considerable, and really requires some big changes to get it right (such as turning every statement into an expression).

    Lastly, there are two things to keep in mind with this closure stuff:

    1. these proposals are not written in stone. If function types are out and long returns are in, you can take any of the proposals and add or remove features as needed until you have just that. Don't get hung up on syntax, that really isn't the issue (yet).

    2. Don't consider closures as its own little world. Some of the java5 features didn't make a whole lot of sense on their own but the package of all java5 features supported each other and made each individual change better. autoboxing is so-so, but together with generics, it makes a lot more sense. The foreach loop is a lot less magical when Iterators have a generics type. Think about it - without generics, foreach would be virtually useless.

    ReplyDelete
  12. Very nice summary. My reaction: If that's all that's wrong with BGGA, then let's get on with it and add BGGA closures to Java.

    - Strange syntax? What was strange yesterday is commonplace tomorrow. Ask the folks who use Ruby, XPath, JPQL, etc.

    - Functions aren't object-oriented? So what? They are useful, and it is a pain to fake them with objects of single-method anonymous classes.

    - Non-local return? I agree that part needs a bit more work. Sometimes you want it, sometimes you don't, and errors should be found at compile time. I played with the current BGGA prototype, and it does have a mechanism for compile-time checking. I would like to see more blogs that come up with creative solutions for this issue, rather than the usual handwringing about >= and =>.

    ReplyDelete
  13. I like FCM as it stands now, what exactly would you be removing? Method types? Could you give an example of how inner methods, method references and method literals would look without method types?

    thanks,

    Collin

    ReplyDelete
  14. Good overview of the current debate. I agree that syntax should probably be re-examined. According to a Google video of Neal Gafter presenting the BGGA proposal, the control invocation syntax arose as a result of collaboration with James Gosling. I really would hope to see a consensus formed around this, so that Java can acquire some means of making anonymous inner classes easier to use, and I personally like the notion of abstracting over control statements offered by BGGA. I can see that you're making an honest effort to improve Java, but lately I've been getting the feeling that some of the anti-change folks have basically migrated to some other language and perhaps feel that Java should just stagnate and die.

    I sincerely hope that those folks would not have a voice in the debate.

    Thanks for your efforts.

    ReplyDelete
  15. Just to clarify, the control invocation syntax allows one to elide the superfluous declaration of the function type when there is only one of them, making things like

    withLock(l) {
    ...
    }

    indeed appear very natural and Java-like. I do believe that the participation of James Gosling will ultimately provide a direction that is closer to what we Java-folks like so much about the language. In short, one can say that James is the so called benevolent dictator and so far has served his post very well, as is evidenced by the success of Java as a language, and as a platform.

    ReplyDelete
  16. Stephen Colebourne20 February 2008 at 19:37

    Gregory: If method types (or similar) remain in FCM, then yes you could write code as you show using generics.

    Remi: James Gosling seems to be a Sun marketing person these days. His opinions carry too much weight for his day to day Java efforts.

    With ActionListener, why should all ActionListeners be restricted? I might want to use one in a non-resticted way. The marker interface approach is broken - a use-site approach is what is needed.

    On function types, I do believe that named function types ought to be possible, and I hope to blog on that soon.

    ReplyDelete
  17. Stephen Colebourne20 February 2008 at 20:17

    Cay: Even BGGA will create closures as one method interfaces. So, maybe we shouldn't be trying to hide that fact from developers. I would say that the majority of Java developers are very comfortable with interfaces. Strengthening their usage makes perfect sense - whereas trying to create a new fundamental program element is far from obvious.

    Aberrant: The possibility is to remove the whole section of the FCM spec about method types. The result would be that method references and inner methods would always have to be converted on construction to a SMI (single method interface).

    ReplyDelete
  18. The main benefit of closures is not in the language itself, but their usage in libraries... There are a huge amount of libraries that would immediately become 'legacies' once the closures come in. In the end we have collections with closures, collections with generics (if we use reified generics), collections with the closures and generics, collections without them (backward compatibility)... I mean is it the way that we want to push Java? And for what purpose, to save a few lines of code writing anonymous inner classes?

    I personally do NOT want closures in Java.. I like them in Scala and Ruby but NOT in Java. Java has far too much baggage of conventions, idioms and "feel" to add more changes. Just freeze the changes in Java. Move on to better languages.

    ReplyDelete
  19. Stephen: knowing if a method like actionPerformed can be called by another thread later is in my opinion inherent to all instances of ActionListenener. So it's not compatible with a use-site approach but more with a declaration-site approach.
    Futhermore, declaration site marker is more easy to grasp than use-site one, i think that wildcard is a good example.

    About single method interface, am i wrong or will you loose the type safety ?

    Rémi

    ReplyDelete
  20. > The result of implementing generics with erasure
    > and wildcards has been a loss of confidence in
    > Java language change. Many now oppose any and
    > all change, however simple and isolated. This
    > is unfortunate, and we must ensure that the next
    > batch of language changes work without issues.

    The next batch? No, "you" first of all must ensure that the LAST batch of changes (e.g. generics) works without issues. Only then "you" should be allowed to work on new features.

    "You" should be treated like schoolchildren. No play in the park and no fun before the homework is done, and done right. Actually, "you" should be grounded for month because of the generics misconduct alone, without closure-TV and closure-telephone. And "you" should get an extra month for not being able to get along with the other closure children in the Java park.

    ReplyDelete
  21. Capattar, are someone else who wasn't payiong attention when generics was developer (in public over many years)? What should your penalty be? Do you have any positive proposals?

    ReplyDelete
  22. I think the syntax can be improved to look more familiar and less verbose. Why not use a simplified C/C++ function pointer syntax given that Java got most of its syntax from C++.

    Something like this maybe:

    boolean* (int) a = (int x) {x <= y};

    And this:

    public U* (T) converter(T* a, U* b, U* (T) c) {
    return (T t) {a.invoke().equals(t) ? b.invoke() : c.invoke(t)};
    }

    ReplyDelete
  23. You say that, "Despite this broader debates that surround closures, we must focus on the merits of the individual proposals." but we still can't judge this in isolation. One must consider the overall size and complexity of the language, especially for people who haven't been living and breathing Java for 10+ years.

    The problem with generics is not only that the syntax and implementation was individually bad. It was that they pushed the language over the threshold where the basics could comfortably be learned and understood in a reasonable amount of time and effort.

    Every new feature we add to the core language now takes us further down that road, unless we start removing things from Java to offset what we're adding. No closure proposal is acceptable at this point unless we also remove inner classes and other obsoleted constructs. Otherwise Java will sink under its own weight. Frankly, it's been going under since Java 5, and none of the existing proposals address this.

    ReplyDelete
  24. Stephen Colebourne24 February 2008 at 03:27

    Indukumar: Your point is a key distinguishing factor between the different proposals. FCM is really just about filling in the gaps in Java - it doesn't much that is conceptually new (eg. we already have Method in reflection - FCM just makes it more typesafe). The inner methods (closures-like) part of FCM is just a way to write the common one method inner class more neatly.

    Remi: The prototype is out now - I haven't lost type safety anywhere that I can think of.

    Elliotte: I actually agree with the sentiment of your reply, but unfortunately I think Java 5 is a bad place to stop and freeze. There are relatively few things we could actually remove (String var[] being the most obvious). Inner classes would need to stay, as they have a different role to closures.

    For me, the real problem with Java 5 is wildcards - erasure is probably fixable, but we are stuck with the undesirable, complex, wildcard part.

    ReplyDelete
  25. I think you could just drop wildcards:

    http://www.artima.com/weblogs/viewpost.jsp?thread=222021

    You could also make closures inner classes, the changed behaviour takes away power and causes problems:

    http://www.artima.com/weblogs/viewpost.jsp?thread=220920

    ReplyDelete
  26. What I dislike is how the examples I see are contrived. the first example could be written much more simply as

    String str =
    someBooleanMethod() ? "TRUE" : "FALSE";

    or even

    String str = upperCase(someBooleanMethod());

    for some reasonable implementation of upperCase().

    I also think the odd look of the operators could be solved if we go to Unicode source files.

    Replace

    int y = 6;
    {int => boolean} a = {int x => x <= y};
    {int => boolean} b = {int x => x >= y};
    boolean c = a.invoke(3) && b.invoke(7);

    with

    int y = 6;
    {int ? boolean} a = {int x ? x <= y};
    {int ? boolean} b = {int x ? x >= y};
    boolean c = a.invoke(3) && b.invoke(7);

    I'm sorry, but the U+21A6 RIGHTWARD ARROW FROM BAR character is missing in this rss reader. That's what those empty boxes are trying to represent.

    ReplyDelete
  27. Come along to JAVAWUG BOF where Stephen will be discussing the closure proposal

    http://jroller.com/javawug

    Be there in!!!

    ReplyDelete