More detail on Closures in JDK 7

This blog goes into a little more detail about the closures announcement at Devoxx and subsequent information that has become apparent.

Closures in JDK 7

At Devoxx 2009, Mark Reinhold from Sun announced that it was time for closures in Java. This was a big surprise to everyone, and there was a bit of a vacuum as to what was announced. More information is now available.

Firstly, Sun, via Mark, have chosen to accept the basic case for including closures in Java. By doing so, the debate now changes from whether to go ahead, to how to proceed. This is an important step.

Secondly, what did Mark consider to be in and what out? Well, he indicated that non-local returns and the control-invocation statement were out of scope. There was also some indication that access to non-final variables may be out of scope (this is mainly because it raises nasty multi-threading Java Memory Model issues with local variables).

In terms of what was included, Mark indicated that extension methods would be considered. This would be necessary to provide meaningful closure style APIs since the existing Java collection APIs cannot be altered. The result of what was in was being called "simple closures".

Finally, Mark offered up a possible syntax. The syntax he showed was very close to FCM:

  // function expressions
  #(int i, String s) {
    System.println.out(s);
    return i + str.length();
  }

  // function expressions
  #(int i, String s) (i + str.length())
  
  // function types
  #int(int, String)

As such, its easy to say that Sun has "chosen the FCM proposal". However, with all language changes, we have to look at the semantics, not the syntax!

The other session at Devoxx was the late night BOF session. I didn't attend, however according to Reinier Zwitserloot, Mark indicated that Exception transparancy might not be essential to the final proposal.

According to Reinier, Mark also said "It is not an endorsement of FCM. He was very specific about that." I'd like to consider that in a little more detail (below).

The final twist was when a new proposal by Neal Gafter was launched which I'm referring to as the CFJ proposal. After some initial confusion, it became clear that this was written 2 weeks before Devoxx, and that Neal had discussed it primarily with James Gosling on the basis of it being a "compromise proposal". Neal has also stated that he didn't speak to Mark directly before Devoxx (and nor did I).

There are a number of elements that have come together in the various proposals:

	CICE	BGGA 0.5	FCM 0.5	CFJ 0.6a	Mark's announcement (Devoxx summary)
Literals for reflection	-	-	Yes	-	No (No info)
Method references	-	-	Yes	Yes	Worth investigating (No info)
Closures assignable to single method interface	Yes	Yes	Yes	Yes	Yes (No info)
Closures independent of single method interface	-	Yes	Yes	Yes	Yes
Access non-final local variables	Yes	Yes	Yes	Yes	No (Maybe not)
Keyword 'this' binds to enclosing class	-	Yes	Yes	Yes	No info (Probably yes)
Local 'return' (binds to closure)	Yes	-	Yes	Yes	Yes
Non-local 'return' (binds to enclosing method)	-	Yes	-	-	No
Alternative 'return' binding (Last-line-no-semicolon)	-	Yes	-	-	No
Exception transparancy	-	Yes	Yes	Yes	No info (Maybe not)
Function types	-	Yes	Yes	Yes	Yes
Library based Control Structure	-	Yes	-	-	No
Proposed closure syntax	Name(args) {block}	{args => block;expr}	#(args) {block}	#(args) {block} #(args) expr	#(args) {block} #(args) (expr)

A table never captures the full detail of any proposal. However, I hope that I've demonstrated some key points.

Firstly, the table shows how the CFJ is worthy of a new name (from BGGA) as it treats the problem space differently by avoiding non-local returns and control invocation. Neal Gafter is also the only author of the CFJ proposal, unlike BGGA.

Secondly, it should be clear that at the high level, the CFJ and FCM proposals are similar. But in many respects, this is also a result of what would naturally occur when the non-local returns and control invocation is removed from BGGA.

Finally, it should be clear that it is not certain that the CFJ proposal meets the aims that Mark announced at Devoxx ("simple closures"). There simply isn't enough hard information to make that judgement.

One question I've seen on a few tweets and blog posts is whether Mark and Sun picked the BGGA or FCM proposal. Well, given what Mark said in the BOF (via Reinier), the answer is that neither was "picked" and that he'd like to see a new form of "simple closures" created. However, it should also be clear that the choices announced at Devoxx were certainly closer to the FCM proposal than any other proposal widely circulated at the time of the announcement.

At this point, it is critical to note that Neal Gafter adds a huge amount of technical detail to each proposal he is involved with - both BGGA and CFJ. The FCM proposal, while considerably more detailed than CICE, was never at the level necessary for full usage in the JDK. As such, I welcome the CFJ proposal as taking key points from FCM and applying the rigour from BGGA.

As of now, there has been relatively little debate on extension methods.

Summary

The key point now is to focus on how to implement closures within the overall scope laid out. This should allow the discussion to move forward considerably, and hopefully with less acrimony.

Closures in JDK 7

So, on Wednesday, Mark Reinhold from Sun announced that it was time for closures in Java. This came as a surprise to everyone, but what was announced?

Closures in JDK 7

Mark took the audience through the new features of JDK 7 in his presentation at Devoxx. As part of this he showed the Fork Join framework.

Part of this framework features a set of interfaces to support functional style predicates and transforms. But, in order to implement this in Java required 80 interfaces, all generified, and that only covered 4 of the primitive types! Basically, Mark, and others, had clearly come to the conclusion that this code was important to Java, but that the implementation with single-method interfaces, generics and inner classes was too horrible to stomach for the JDK.

Thus, 'its time to add closures to Java'.

Mark announced that Sun would investigate, and intended to commit, closures to OpenJDK for JDK 7. While there will be no JSR yet, it isn't unreasonable to expect that there will be an open aspect to discussions.

Mark noted that the work in the closures discussion, and particularly the detail in the BGGA proposal, had allowed the options to be properly explored. He emphasised how this had resulted in the decision to reject key aspects of BGGA.

JDK 7 closures will not have control-invocation statements as a goal, nor will it have non-local returns. He also indicated that access to non-final variables was unlikely. Beyond this, there wasn't much detail on semantics, nor do I believe that there has been much consideration of semantics yet.

On syntax, Mark wrote up some strawman syntaxes. These follow the FCM syntax style:

  // function expressions
  #(int i, String s) {
    System.println.out(s);
    return i + s.length();
  }

  // function expressions
  #(int i, String s) (i + s.length())
  
  // function types
  #int(int, String)

Mark also argued that a form of extension methods would be added to allow closures to be used with existing libraries like the collections API.

Although I must emphasise that everything announced was a proposal and NOT based on any specific proposal (BGGA, FCM or CICE). However, the syntax and semantics do follow the FCM proposal quite closely.

In addition, Neal Gafter has produced an initial formal writeup of what was announced derived from BGGA. Based on my initial reading, I believe that Neal's document represents a good place to start from. I also hereby propose that we refer to Neal's document as the CFJ proposal (Closures for Java), as BGGA rather implies control invocation.

So, lets hope the process for closures provides for feedback, and we get the best closures in the style most suitable for Java in JDK 7.

FCM closures - options within

The FCM closures proposal, with the JCA extension, consists of multiple parts. This blog outlines how those parts fit together.

FCM+JCA spec parts

The FCM+JCA spec, contains the following elements:

• Method literals, also constructor and field literals
• Method references, also constructor references
• Inner methods
• Method types, aka function types
• Control invocation, in the JCA extension

These five parts all fulfil different roles in the proposal. But what is often not understood is how feasible it would be to implement less than the whole specification.

Method references and literals

Reviewing response to the entire debate, it is clear to me that method references and/or method literals have generally widespread appeal. The ability to reference a method, constructor or field in a compile-time safe and refactorable way is a huge gain for a statically typed language like Java.

It should be noted that although Method References appear only in the FCM spec, they could be added to the BGGA or CICE spec without difficulty. Also, they are included in the closures JSR proposal.

There are three areas of contention with method references and literals.

Firstly, should both references and literals be supported, or just references. Or, looking at the question differently, should literals have a different syntax.

The problem here is that if a method reference and literal have the same syntax then it is unclear as to what the type of the expression is. The FCM prototype demonstrates, I believe, that this can be solved using the same syntax. The approach taken is to say that the default type is a method literal, but that it can be converted (boxed) to a method reference at construction time. Any ambiguity is an error.

 Method m = Integer#valueOf(int);
 IntToInteger i = Integer#valueOf(int);

In this example, IntToInteger is a single method interface. Because the expression Integer#valueOf(int) is assigned to a single method interface, the conversion occurs (generating a wrapping inner class).

The second issue is what a method reference can be boxed into. This is essentially a question of whether function types should be supported, and I'll cover that below.

The third issue is syntax, specifically the use of the #. Personally, I find this syntax natural and obvious to read, but I know others differ. I think it is important to get the syntax right, but final decisions on that can come later.

So, are method literals and reference required when implementing FCM? I would say 'yes'. These are simple, popular, constructs that naturally extend Java in a non-threatening way. Some have suggested omitting the literals as reflection is not type-safe, however this misses the point of the large number of existing frameworks and APIs that accept reflection Method as an input parameter.

Inner methods / Closures

 ActionListener lnr = #(ActionEvent ev) {
   ...
 };

This is where the key difference with BGGA lies, notably over the meaning of return and the value, and safety, of non-local returns.

Opinions on this appear to me to be impacted by the generics implementation, where the decision was made to do what feels like 'half a job'. As a result, there is a meme that runs 'we must implement closures fully or not at all'. This meme is extremely unfortunate, as it is not allowing a rational analysis of the semantics of the proposals. Anyone supporting BGGA really needs to consider the mistakes that developers will make again and again with the non-local return/last-line-no-semicolon approach.

So, are inner methods required when implementing FCM? I would say 'effectively, yes'. Although you could just implement method literals and references alone, there are even bigger gains to be had from adding inner methods. They greatly simplify the declaration of single method inner classes, and allow much of the impact of closures in the style of Java.

Function types / Method types

 #(int(String) throws IOException)

These allow a new powerful form of programming where common pieces of code can be easily abstracted. They simply act as types, but they have two different properties from other types.

Firstly, they have no name. This means an absence of Javadoc, including any semantic requirements of the API, such as thread-safety or null/not-null.

Secondly, they only describe the input and output types. This is a higher abstraction than Java has previously used, and will require a mindset shift for those using them.

So, are function types required when implementing FCM? I would say 'no, not required'. It is perfectly possibly and reasonable to implement FCM without method types. In fact, that is what the prototype does. In practice, this just means that all conversions from method references and inner methods must be to single method interfaces rather than method types.

Omitting method types greatly simplifies the conceptual weight of the change. The downside is that true higher order functional programming becomes near impossible. That may be no bad thing. Java is not, and never has been, a functional programming language. It seems very odd to try and push it in that direction at this point in its life.

A better alternative would be to pursue supporting primitive types in generics. This would greatly reduce the overhead of single method interfaces required by something like the fork-join framework.

Similarly, making single method interfaces easier to write (lightweight interfaces) would be a direction to take in the absence of method types.

Control invocation

 withLock(lock) {
   ...
 }
 public void withLock(#(void()) block : Lock lock) {
   ...
 }

Control invocation forms are perhaps the only way forward in Java longer term because they allow us to escape from many of these language change debates. They allow anyone to write methods that can be used in the style of control statements. It is vital to remember that they are just methods however.

BGGA appears to build much of its spec around control invocation, and the non-local returns make perfect sense in this area.

The JCA spec defines that the calling code should be identical to BGGA, but the method invoked should be written differently. The aim of JCA is to provide an element of discouragement from using control invocation. This is because of the additional complexities in getting the code right (exception transparancy, completion transparancy, non-local returns etc). A different, special, syntax encourages this feature to be restricted to senior developers, or heavily code reviewed.

So, is control invocation required when implementing FCM+JCA? I would say 'no'. It is perfectly possibly and reasonable to implement FCM+JCA without control invocation (although of course that means it would just be FCM!).

The inclusion or omission of method types is also linked to control invocation, as method types are a pre-requisite for control invocation in the JCA spec.

Summary of possible implementations

Thus, here are the possible FCM+JCA implementation combinations that make sense to me:

1. Literals and References
2. Literals, References and Inner methods
3. Literals, References, Inner methods and Method types
4. Literals, References, Inner methods, Method types and Control invocation

My preferred options are number 2 and number 4.

Why? Because, I believe inner methods are too useful to omit, and I believe method types are generally too complex unless you really need them. (Also Java isn't a functional programming language.)

The key point of this blog is to emphasise that FCM is not a take it or leave it proposal. There are different options and levels within it that could be adopted.

This extends to versions of Java. For example, it would be feasible to implement option 1, literals and references in Java 7, whilst adding inner methods and maybe more in Java 8.

Summary

I've shown how FCM has parts which can be considered separately to a degree. I've also indicated which combinations make sense to me.

Which combinations make sense to you?

FCM prototype available

I'm happy to announce the first release of the First Class Methods (FCM) java prototype. This prototype anyone who is interested to find out what FCM really feels like.

Standard disclaimer: This software is released on a best-efforts basis. It is not Java. It has not passed the Java Compatibility Testing Kit. Having said that, in theory no existing programs should be broken. Just don't go relying on code compiled with this prototype for anything other than experimentation!

FCM javac implementation

The FCM javac implementation is hosted at Kijaro. The javac version used as a baseline is OpenJDK (the last version before the Mercurial cutover).

The prototype includes the following features:

• Method literals
• Constructor literals
• Field literals
• Static method references
• Bound method references
• Constructor references
• Anonymous inner methods

The following are not implemented:

• Method types
• Instance method references
• Named inner methods
• Inner method non-final local variable access
• Inner method exception inference
• Inner method exception/completion transparancy
• Conversion to single abstract method classes

The download includes a README with many FAQs answered, including more information on how the types work.

The biggest outstanding area is getting generics working properly. This is a complex task however, and I took the view that it was better to release early than spend any more time trying to get generics working properly.

Nevertheless, even without full generics, you can get a really good feel for how Java would look and feel with FCM. In addition, the FCM enabled code just falls off my fingers very nicely. Now if only we could get Eclipse or Netbeans support...

Summary

One of the key requests for considering FCM as a viable proposal has been having a prototype to play with. Now the prototype is out there, it would be really great to hear some feedback, including any bugs. Comments welcome here, at kijaro-dev mailing list or scolebourne-joda-org.

Implementing FCM - part 1

Its been a long coding weekend, but after much head scratching, I might finally be getting the hang of changing the java compiler. My goal? to implement First class methods (FCM).

Implementing FCM

Currently, on my home laptop I have an altered version of javac, originally sourced from the KSL project which supports Method literals (section 2.1 of the spec). What this means in practice is that you can write:

  Method m = String#substring(int,int);

This will now compile and return the correct Method object. The left hand side can be a reference to an inner class such as Map.Entry without problem. It will also validate the standard access rules, preventing you from referencing a private/package/protected method when you shouldn't.

In fact, the implementation is currently more generous than this, as the left hand side can also be a variable. This is needed for later phases of FCM, but isn't really appropriate for a method literal.

Making a language change like this is definitely intimidating though, and mostly because the compiler feels like such a big scary beast. The first two parts - Scanner and Parser are actually pretty easy. Once those are done you have an AST (Abstract Syntax Tree) representing your code. In this case that involves a new node, currently called 'MethodReference'.

After that it gets more confusing. The main piece of work should be in Lower, which is where syntax sugar is converted to 'normal' Java. For example the new foreach loop is converted to the old one here. Since method literals are just syntax sugar, Lower is where the change goes.

Unfortunately, to get to Lower, you also have to fight with Attr which is where the types get allocated, checked and validated. That was a real head scratcher, and I'm sure a compiler expert would laugh at my final code. It does work though, and after all thats the main thing.

The next step will be to convert 'MethodReference' to 'MemberReference' and enable Field and Constructor literals. After that, the real FCM work begins!

Publishing

So, by now you probably want to have a download link. But I'm not going to provide one. As the code is GPL, I can't just publish a binary, the source has to come too. And personally, I'd rather actually have it in a SVN repo somewhere before I let it escape at all (because that can probably count as the GPL source publication). So, I'm afraid you'll have to be patient :-)

This probably should go in the KSL, but right now I don't know whether it will. The trouble with the KSL is that it still has a few guards and protections around it. And to date, no-one other than Sun has committed there, even in a branch. I'm feeling more comfortable in creating a separate, 'fewer rules', project at sourceforge. In fact, it would be great if other javac hackers like Remi wanted to join too ;-)

One final thing... if anyone wants to help out with this work I'd much appreciate it - scolebourne|joda|org. Compiler hacking isn't easy, but its a great feeling when you get it working!

Closures options - FCM+JCA is my choice

In my last post I drew a diagram showing how I viewed the main proposals in terms of complexity. It has since been pointed out to me that I didn't clearly identify whether the JCA part of FCM was included or not. So I'll redraw the diagram and express my opinon.

The three key proposals in the 'closure' space are - BGGA from Neal Gafter et al, CICE from Josh Bloch et al, and FCM from Stefan and myself. FCM has an optional extension called JCA which adds control abstraction [1].

Control abstraction [1] is that part of the closures language change which allows a user to write an API method which acts, via a static import, as though it is a built in keyword. This is an incredibly powerful facility in a programming language. However, reading many forums, this is a facility that many Java developers also find concerning.

Given this, I'll redraw the diagram from yesterday:

 --------------------------------------------------------------------------
  Simpler                                                     More complex
 --------------------------------------------------------------------------
   No change   Better inner   Method-based   Method-based   Full-featured
                 classes        closures     closures and   closures with
                                             control abst.   control abst.
 --------------------------------------------------------------------------
 |        No control abstraction [1]      |    Control abstraction [1]   |
 --------------------------------------------------------------------------
   No change      CICE*           FCM           FCM+JCA         BGGA
 --------------------------------------------------------------------------

* It should also be noted that Josh Bloch has written the ARM proposal for resource management that tackles a specific control abstraction [1] use case.

So, what is my preference?

Firstly, I believe that Java should be enhanced with FCM-style closures (where the 'return' keyword returns to the invoker of the closure).

Secondly, I believe that control abstraction [1] would be a very powerful and expressive language enhancement. Personally, I would like to see Java include control abstraction, but I am very aware that it is a real concern for others in the community. I am also not yet fully confident of the risks involved by rushing into control abstraction [1].

Thus, my preferred option is FCM+JCA for Java 7. However I would also accept a potentially lower risk option of including just the FCM-style solution in Java 7 and adding JCA-style control abstraction [1] in Java 8.

Feel free to add your preference to the comments.

[1] Update: Neal Gafter has indicated that I am using conflicting terminology wrt 'control abstraction'. Neal uses the term 'control abstraction' to refer to any closure which can abstract over an arbitrary block of code including interacting with break, continue and return. Whereas, in this blog post I have been referring to control abstraction as adding API based keywords (described in paragraph 3), which Neal refers to as the 'control invocation syntax'. Apologies for any confusion

Java Control Abstraction for First-Class Methods (Closures)

Stefan Schulz, Ricky Clarkson and I are pleased to announce the release of Java Control Abstraction (JCA). This is a position paper explaining how we envisage the First-Class Methods (FCM) closures proposal being extended to cover control abstraction.

Java Control Abstraction

So, what is control abstraction? And how does it relate to FCM? Well its all about being able to add methods in an API that can appear as though they are part of the language. The classic example is iteration over a map. Here is the code we write today:

  Map<Long,Person> map = ...;
  for (Map.Entry<Long, Person> entry : map) {
    Long id = entry.getKey();
    Person p = entry.getValue();
    // some operations
  }

and here is what the code looks like with control abstraction:

  Map<Long,Person> map = ...;
  for eachEntry(Long id, Person p : map) {
    // some operations
  }

The identfier eachEntry is a special method implemented elsewhere (and statically imported):

  public static <K, V> void eachEntry(for #(void(K, V)) block : Map<K, V> map) {
    for (Map.Entry<K, V> entry : map.entrySet()) {
      block.invoke(entry.getKey(), entry.getValue());
    }
  }

As can be seen, the API method above has a few unique features. Firstly, it has two halves separated by a colon. The first part consists of a method type which represents the code to be executed (the closure). The second part consists of any other parameters. As shown, the closure block is invoked in the same way as FCM.

The allowed syntax that the developer may enter in the block is not governed by the rules of FCM. Instead, the developer may use return, continue, break and exceptions and they will 'just work'. Thus in JCA, return will return from the enclosing method, not back into the closure. This is the opposite to FCM. This behaviour is required as the JCA block has to act like a built-in keyword.

One downside of the approach is that things can go wrong because the API writer has access to a variable that represents the closure. The API writer could store this in a variable and invoke it at a later time after the enclosing method is complete. However, if this occurred, then any return/continue/break statements would no longer operate correctly as the original enclosing method would no longer be on the call stack and a weird and unexpected exception will be thrown.

The semantics of a pure FCM method invocation are always safe, and there is no way to get one of these unexpected exceptions. But, for JCA control abstraction we could find no viable way to stop the weird exceptions. Instead, we have chosen to specifically separate the syntax of FCM from the syntax of control abstraction in JCA.

Our approach is to accompany the integration of control abstraction into Java by a strong set of messages. Developers will be encouraged to use both FCM callbacks and JCA control abstractions. However, developers would only be encouraged to write FCM style APIs, and not JCA.

Writing the API part of any control abstraction (including JCA) is difficult to get right (or more accurately easy to get wrong). As a result, some coding shops may choose to ban the writing of control abstraction APIs, but by having a separate syntax this will be easy to do for the tools. It is expected, of course, that the majority of the key control abstractions will be provided by the JDK, where experts will ensure that the control abstraction APIs work correctly.

Summary

This document has taken a while to produce, especially by comparison with FCM. In the end this indicated to us that writing a control abstraction is probably going to be a little tricky irrespective of what choices the language designer makes. By separating the syntax and semantics from FCM we have clearly identified the control abstraction issue in isolation, which can only be a good thing.

Feedback always welcome!

First-Class Methods: Java-style closures - v0.5

Stefan and I are pleased to announce the release of v0.5 of the First-class Methods: Java-style closures proposal.

Changes

Since v0.4, we have focussed on understanding the relationship between the various use cases for closures and FCM. We identified the following three use case groups:

• Asynchronous - callbacks, like swing event listeners - where the block is invoked by a method other than the higher order method to which it was passed
• Synchronous - inline callbacks, like list sorting, filtering and searching - where the block is invoked by the higher order method and completes before it returns
• Control abstraction - keywords, like looping around a map, or resource acquisition

Having identified the three groups, we found that we could extend FCM from asynchronous to synchronous with a couple of simple changes. But we became convinced that the control abstraction use case is completely different. This is because the meaning of return, continue and break needs to be entirely different in the control abstraction use case.

As a result, of our deliberations, these are the main changes from v0.4:

1) Local variables can now be accessed read/write by all FCM inner methods. This enables the synchronous closure use case, and does not greatly compromise the asynchronous use case. We were concerned about the possibility of race conditions, but we identified that this is caused by concurrent access to the local variable, and could be addressed by a warning annotation - @ConcurrentLocalVariableAccess:

  public void addUpdater(@ConcurrentLocalVariableAccess #(void()) updater) {
    // invoke updater on a new thread
    // marked with annotation as local variable could be accessed
    // by new thread and original at the same time
  }

  public void process() {
    int val = 6;
    addUpdater(#{
      int calc = val;    // OK, can read local variables
      val = 7;           // OK, can update local variables
    });
    System.out.println(val);  // race condition - warning from annotation
  }

2) Added brief section on transparency. When considering synchronous closures it is desirable to not have to be concerned with every checked exception. Although we discussed alternatives, we simply reference the BGGA work here.

3) Removed the creation of stub methods for named inner methods. This prevents FCM from creating MouseListener instances, but is safer. As a result, we also removed the method compound concept.

Summary

This version tidies up the princpal loose ends of FCM and provides rationale for our decisions. We think it represents a simple and safe extension to Java which would be of great value to developers. It should also be noted that we are not intending to extend FCM further to cover the control abstraction use case.

If you've any more feedback, please let us know!

First-Class Methods: Java-style closures - v0.4

Stefan and I are pleased to announce the release of v0.4 of the First-class Methods: Java-style closures proposal.

Changes

Since v0.3, we have tried to incorporate some of the feedback received on the various forums. The main changes are as follows:

1) Constructor and Field literals. It is now possible to create type-safe, compile-time changed instances of java.lang.reflect.Constructor and Field using FCM syntax:

  // method literal:
  Method m = Integer#valueOf(int);

  // constructor literal:
  Constructor<Integer> c = Integer#(int);

  // field literal:
  Field f = Integer#MAX_VALUE;

2) Extended method references. Invocable method references have been renamed to method references. There are now four types, static, instance, bound and constructor:

  // static method reference:
  #(Integer(int)) ref = Integer#valueOf(int);

  // constructor reference:
  #(Integer(int)) ref = Integer#(int);

  // bound method reference:
  Integer i = ...
  #(int()) ref = i#intValue();

  // instance method reference:
  #(int(Integer)) ref = Integer#intValue();

3) Defined mechanism for accessing local variables. We have chosen a simple mechanism - copy-on-construction. This mechanism copies the values of any local variables accessed by the inner method to the compiler-generated inner class at the point of instantiation (via the constructor).

The effect is that changes made to the local variable are not visible to the inner method. It also means that you do not have write access to the local variables from within the inner method. The benefits are simplicity and thread-safety:

  int val = 6;
  StringBuffer buf = new StringBuffer();
  #(void()) im = #{
    int calc = val;    // OK, val always equals 6
    val = 7;           // compile error!
    buf.append("Hi");  // OK
    buf = null;        // compile error!
  };

4) Stub methods are only generated for methods of return type void. This addresses a concern that it was too easy to create invalid implementations using named inner methods.

5) Added method compounds. This addresses the inability to override more than one method at a time. Unlike inner classes, each inner method in a method compound remains independent of one another, thus in this example, the mouseClicked method cannot call the mouseExited method.

  MouseListener lnr = #[
    #mouseClicked(MouseEvent ev) {
      // handle mouse click
    }
    #mouseExited(MouseEvent ev) {
      // handle mouse exit
    }
  ];

6) Added more detail on implementation throughout the document, with specific examples.

We believe that we have addressed the key concerns of the community with this version of FCM. But if you've any more feedback, please let us know!

Comparing closures (2 more examples) - CICE, BGGA and FCM

As my last post comparing the three closure proposals - FCM, CICE and BGGA - seemed to be useful, I thought I'd post another. Again, I'll try not to be biased!

Example 3: Sorting

The Java Collections Framework has a callback to support sorting using the Comparator interface. This is used via the two argument sort method on Collections, taking the List and the Comparator. This example will sort a list of strings by length placing the "selected" string first.

The following example is from Java 6. This uses an inner class, which can access the parameter 'selected' so long as it is final:

  public void sort(List<String> list, final String selected) {
    Collections.sort(list, new Comparator<String>() {
      public int compare(String str1, String str2) {
        if (str1.equals(selected)) {
          return -1;
        } else if (str2.equals(selected)) {
          return 1;
        } else {
          return str1.length - str2.length();
        }
      }
    });
  }

The following example is from the CICE proposal. This is shorthand for an inner class, so the parameter 'selected' must be final:

  public void sort(List<String> list, final String selected) {
    Collections.sort(list, Comparator<String>(String str1, String str2) {
      if (str1.equals(selected)) {
        return -1;
      } else if (str2.equals(selected)) {
        return 1;
      } else {
        return str1.length - str2.length();
      }
    });
  }

The following example is from the BGGA proposal, and uses the standard-invocation syntax and closure conversion. The closure can access the parameter 'selected' without it needing to be final. The closure can only return a value from its last line, and does so by not specifying the final semicolon, hence the need for the result local variable:

  public void sort(List<String> list, String selected) {
    Collections.sort(list, {String str1, String str2 =>
      int result = 0
      if (str1.equals(selected)) {
        result = -1;
      } else if (str2.equals(selected)) {
        result = 1;
      } else {
        result = str1.length - str2.length();
      }
      result
    });
  }

The BGGA proposal control-invocation syntax may not be used for this example. This is because the control-invocation syntax is not permitted to yield a result as required here.

The following example is from the FCM proposal, and uses an inner method. The inner method has access to the parameter 'selected' without it needing to be final:

  public void sort(List<String> list, String selected) {
    Collections.sort(list, #(String str1, String str2) {
      if (str1.equals(selected)) {
        return -1;
      } else if (str2.equals(selected)) {
        return 1;
      } else {
        return str1.length - str2.length();
      }
    });
  }

The example cannot be directly translated to an FCM invocable method reference. This is because the functionality being performed relies on the parameter 'selected', which is not being passed into the compareTo method on the Comparator. If the 'selected' variable was an instance variable then an invocable method reference could be used.

Example 4: Looping over a map

This example aims to print each item in a map.

The following example is from Java 6, and uses an enhanced foreach loop:

  Map<String, Integer> map = ...
  for (Map.Entry<String, Integer> entry : map.entrySet()) {
    String key = entry.getKey();
    Integer value = entry.getValue();
    System.out.println(key + "=" + value);
  }

The CICE proposal contains no new syntax to address this issue.

The following example is from the BGGA proposal, and uses the control-invocation syntax.

  Map<String, Integer> map = ...
  for eachEntry(String key, Integer value : map) {
    System.out.println(key + "=" + value);
  }

The FCM proposal contains no new syntax to address this issue. However the authors hope to publish a separate proposal to cover this use case at some point.

Summary

This blog contains two more comparisons of CICE, BGGA and FCM with examples. For the previous two, see the previous post.

Comparing closures - CICE, BGGA and FCM

I've been asked by a comment to compare the new FCM closure proposal from Stefan and myself with the other two proposals, CICE and BGGA. Obviously, this is difficult without being too biased, but I'll to do my best!

Its all about this

The key difference, when considering closures, between CICE and BGGA or FCM is the handling of this.

CICE is just a simplified syntax for creating an inner class. As such, the meaning of this is the same as in an inner class. As a reminder, in an inner class, this refers to the inner class instance, but there is implicit support for calling methods from the outer class. If you need to refer to this of the outer class you have to use the OuterClass.this syntax.

BGGA closures and FCM inner methods make this refer directly to the nearest surrounding class in the source code (the outer class). This results in much simpler code within the closure/inner method.

Ignoring this, the closure part of the three proposals vary principally by the detail of syntax. Thats why you have to focus on the semantics, not the syntax, when comparing them.

In addition, BGGA offers the control-invocation syntax for closures, while FCM offers method literals and invocable method references.

Example 1: Adding an ActionListener to a swing button

This is a fairly common example when using swing. The standard solution today is to use an inner class.

The following example is from Java 6. The method within the inner class can access handleButtonPress() via the special inner class rules:

  public void init(final JButton button) {
    button.addActionListener(new ActionListener() {
      public void actionPerformed(ActionEvent ev) {
        handleButtonPress(ev);
      }
    });
  }
  public void handleButtonPress(ActionEvent ev) {
    // actually handle the button press
  }

The following example is from the CICE proposal. This is shorthand for an inner class, so the method within the inner class can still access handleButtonPress() via the special inner class rules:

  public void init(JButton button) {
    button.addActionListener(ActionListener(ActionEvent ev) {
      handleButtonPress(ev);
    });
  }

The following example is from the BGGA proposal, and uses the standard-invocation syntax and closure conversion. The closure has full access to this (the instance of the init() method) and hence handleButtonPress():

  public void init(JButton button) {
    button.addActionListener({ActionEvent ev =>
      handleButtonPress(ev);
    });
  }

The following example is also from the BGGA proposal, but uses the control-invocation syntax. I believe that the authors of BGGA would not recommend that this syntax should be used for an ActionListener:

  public void init(JButton button) {
    button.addActionListener(ActionEvent ev : ) {
      handleButtonPress(ev);
    }
  }

The following example is from the FCM proposal, and uses an inner method. The inner method has full access to this (the instance of the init() method) and hence handleButtonPress():

  public void init(JButton button) {
    button.addActionListener(#(ActionEvent ev) {
      handleButtonPress(ev);
    });
  }

The following example is also from the FCM proposal, and uses an invocable method reference. This syntax merely references handleButtonPress() and FCM creates the bridging ActionListener. Note however, that the signature of handleButtonPress() must match that of the listener.

  public void init(JButton button) {
    button.addActionListener(this#handleButtonPress(ActionEvent));
  }

Example 2: A closure that multiplies a number

This example is using closures in the style of functional programming. This is rarely used in Java today, but can be simulated to a degree using an inner class.

The following example is from Java 6, and uses an inner class. Note how the button variable must be declared final:

  final int multiplier = 3;
  IntMultiplier mult = new IntMultiplier() {
    public int multiply(int value) {
      return value * multiplier;
    }
  };
  int result = mult.multiply(6);  // result is 18 (ie. 3 * 6)

The following example is from the CICE proposal. The only change is that the variable is now declared to be public (it could also be declared as final, or have no modifier which also means final):

  public int multiplier = 3;
  IntMultiplier mult = new IntMultiplier() {
    public int multiply(int value) {
      return value * multiplier;
    }
  };
  int result = mult.multiply(6);

The following example is from the BGGA proposal, and uses a closure. The {int => int} is a function-type that defines the parameters and return from the closure. The closure definition names the parameter and uses the value of multiplier from its environment. The result of the closure is returned by omitting the semicolon from the last line of the closure. The closure is called using the implied invoke() method:

  int multiplier = 3;
  {int => int} mult = {int value =>
    value * multiplier
  };
  int result = mult.invoke(6);

The following example is from the FCM proposal, and uses an inner method. The #(int(int)) is a method-type that defines the parameters and return from the inner method. The method definition names the parameter and uses the value of multiplier from its environment. The result of the method is returned using the return keyword. The closure is called using the implied invoke() method:

  int multiplier = 3;
  #(int(int)) mult = #(int value) {
    return value * multiplier;
  };
  int result = mult.invoke(6);

Summary

I've presented a quick comparison of CICE, BGGA and FCM with examples. Let me know if it was useful, and if you'd like more comparisons.

First-Class Methods: Java-style closures

The debate on Closures in Java 7 has been limited to two proposals so far. Today Stefan Schulz and myself would like to offer a third option - First-Class Methods: Java-style closures (FCM) - to the Java community.

First-Class Methods: Java-style closures

Our aim when writing this proposal was to provide developers with much of the power of full-blown closures, but without the complexity. The syntax had to fit neatly within Java, but more importantly so did the semantics. The result of our work is a closure proposal that focuses on methods rather than closures, hence the name 'First-class methods'.

For those currently programming in Java, the method is the principal construct where application logic resides. Our proposal simply takes methods and allows them to be used as first-class objects within the application. But the new constructs are still recognisably methods. This results in a rapid learning curve - since all developers are already familiar with methods.

The proposal adds four new syntax elements:

• Method literals - a type-safe way to refer to a method
• Method types - a way to define the parameters and result of a method
• Invocable method references - a reference to a method-object combination that together can be invoked
• Inner methods - a way to write a method inline within another method, similar to inner classes

Full details of the syntax and semantics can be found in the First-Class Methods proposal.

Example

To whet your appetite, here is an example of creating a comparator using an inner class in Java 5:

  List<String> list = ...
  Collections.sort(list, new Comparator<String>() {
    public int compare(String str1, String str2) {
      return str1.length() - str2.length();
    }
  });

And here is the code rewritten using an inner method from the proposal:

  List<String> list = ...
  Collections.sort(list, #(String str1, String str2) {
    return str1.length() - str2.length();
  });

The inner method version will compile to almost exactly the same bytecode as the inner class version. But the syntax is much shorter and the semantics clearer.

Relationship to other proposals

There are two other closure proposals being debated. The CICE proposal aims to simplify inner class creation, while the BGGA proposal aims to introduce full-blown closures and API driven control flow statements.

The First-Class Methods (FCM) proposal is located between CICE and BGGA. FCM is considerably more powerful than CICE, which is still lumbered with the dreaded OuterClass.this syntax of the inner class. Equally, FCM doesn't get caught up in the obsessive BGGA requirement of Tennent's Correspondence Principles with its myriad of nasty implications (such as last-line-no-semicolon-return, exceptions for control-flow and RestrictedFunction). Instead FCM focuses on empowering the humble, but familiar, method.

Summary

The First-Class Methods proposal (FCM) proposal provides an alternative to the two existing closure proposals. It focusses on the familiar concept of methods, providing all the syntax and semantics necessary to use them to their full potential. The result is a proposal that gives the developer power without complexity or surprise and would make a great Java language change.

As always, Stefan and I would love to hear your feedback on this proposal.