Closures use cases - updating

More 'use cases' for the closures proposal (see yesterday). This time, the more thorny subject of methods that alter the collection.

The standard solution to altering a collection using a closure in other languages seems to be 'don't do it'. Instead, functional programming prefers returning a new collection with the contents altered. I would argue that while that might sometimes be what you want, it probably doesn't fit well with the Java paradigms. ie. in Java, we tend to alter the state of our existing objects. Lets see how that works:

filter(Collection)

  Collection[String] coll = ...
  filter (String str : coll) {
    boolean remove = false;
    if (str == null) {
      remove = true;
    }
    remove
  }

This operation would remove the element (calling Iterator.remove) if the result of the closure is true. But, the syntax is very clumsy. The result can only be specified as the last expression in the block, and it has to be without a semicolon. This looks like a guaranteed cause of confusion.

transform(List)

  List[String] list = ...
  transform (String str : list) {
    str.toUpperCase()
  }

This operation would change each value in the list to a new value, based on the result of the closure. Note again that the result is defined by a statement with no semi-colon at the end. In this case, because there is no other statement in the block, this doesn't look too bad, but I'm sure I'd put the semi-colon in as a reflex action.

Alternate approach 1 - yield parameter

  List[String] list = ...
  update (String str : list : boolean removeItem) {
    if (str == null) {
      removeItem = true;
    }
  }

The idea here is that instead of yielding a result back to the closure by some arbitrary last statement without a semi-colon, a variable is provided which must be updated with the resut of the code-block. In other words, removeItem is the result of the closure, and you have to assign a value to it

There may be ways to play with the syntax, but this does feel a little 'odd'. It would work as a general purpose replacement for the 'last-line-no-semi-colon' concept.

Alternate approach 2 - an updater

  List[String] list = ...
  update (String str, Updater[String] it : list) {
    if (str == null) {
      it.remove();
    } else {
      it.set(str.toUpperCase());
    }
  }

The idea is to provide the closure with an object suitable for updating the list. Now perhaps this could just be a ListIterator, or it might be a separate class with a different set of methods (ie. 'Updater') - thats detail.

The nice part about this solution is that it is closer to traditional Java syntax and traditional Java mentality. Of course you might then ask exactly what this loop is gaining over the current code. One downside is that the 'String' parameter has to be repeated. This could potentially be solved, but with its own issues:

  List[String] list = ...
  update (Updater[String] it : list) {
    if (it.get() == null) {
      it.remove();
    } else {
      it.set(it.get().toUpperCase());
    }
  }

Alternate approach 3 - improve the syntax of the closures proposal

  List[String] list = ...
  removeLoop:update (String str : list) {
    if (str == null) {
      goto removeLoop => true;
    }
    goto removeLoop => false;
  }

The idea here is to provide a mechanism to return the result of the closure clearly and safely. The 'removeLoop' is a label, as currently exists in Java. Here I am using it to clearly identify which closure to return the result to. (Neal Gafter has indicated that yielding a result without identifying the closure to yield to causes issues.) The goto in this proposal also doesn't break Java, as goto is a reserved keyword.

It should be noted that there are many, many possible syntax variations to achieve the effect above. I'm certainly not hung up on this one, although I think it reads pretty clearly, and is safe.

Summary

Well, that outlines where I see a hole in the closures proposal at present, and three or four possible solutions. Next time, I'll try and cover some non-collection use-cases. All feedback welcomed!!

Closures use cases - looping

The closures proposal for Java 7 continues to progress. But there doesn't seem to be a large list of potential use cases lying about, just a few random examples. I'm going to try and write up some example 'use cases' hopefully to help the debate.

each(Collection)

  Collection[String] coll = ...
  each (String str : coll) {
    // do something with str
  }

This would loop around each entry in the collection exactly as per the enhanced foreach loop.

eachIndexed(Collection)

  Collection[String] coll = ...
  eachIndexed (String str, int index : coll) {
    // do something with str
  }

This would provide a loop (0-based, incrementing by 1) as well as the entry each time around the loop.

eachReversed(List)

  List[String] list = ...
  eachReversed (String str : list) {
    // do something with str
  }

This operation would loop around the list in reverse order.

each(Map)

  Map[String, Integer] map = ...
  each (String key, Integer value : map) {
    // do something with key/value
  }

This loops around each entry in the map, giving the loop the key and value each time around. An indexed version of this method would probably also be needed.

Javapolis whiteboards on Java language changes

At Javapolis I 'looked after' a set of whiteboards on which all 2800+ participants were free to write up their ideas for improving their Java lives.

These were photographed at the end, and I've now extracted the information into textual format. So, if you were at Javapolis, or even if you weren't, you can see what was on the collective mind here.

As you look through them, you'll note that there were a few indicative votes taken. It appeared that developers liked the idea of closures and solving some issues with NPEs, but weren't so fussed about improving the foreach loop. There was also a vote about native XML, which showed a majority against the feature at present.

enum + switch == NPE

Enums hide most of their implementation successfully from the developer., You just use them as constants, assign them to fields, call methods on them, no real issues and no exposed implementation details. However, they do fail on at least one count - a NullPointerException in switch statements.

When enums were introduced, the compiler was extended to support enums in switch statements. Consider an enum of traffic lights (RED, YELLOW, GREEN):

public enum TrafficLight {RED, YELLOW, GREEN}

TrafficLight trafficLights = ...
switch (trafficLights) {
  case RED: {/* do stuff */}
  case YELLOW: {/* do stuff */}
  case GREEN: {/* do stuff */}
}

This code can throw a NullPointerException on the first line. How? Well, take a look at how the compiler sees it:

switch (trafficLights.ordinal()) {
  case 0: {/* do stuff */}
  case 1: {/* do stuff */}
  case 2: {/* do stuff */}
}

Now its clear whats going on. The enum switch statement is just syntax sugar for a regular switch statement. But in order to get the ordinal for the switch, a method is called, and that method call will throw an NPE if trafficLights is null.

Can we solve this issue? Well, yes! Since this is just syntax sugar already, why not extend the sugar to allow us to avoid the NPE? This can be achieved by allowing a "case null" within the sugared enum switch statement:

switch (trafficLights) {
  case RED: {/* do stuff */}
  case YELLOW: {/* do stuff */}
  case GREEN: {/* do stuff */}
  case null: {/* do stuff */}  // the null case
}

which compiles to:

switch (trafficLights != null) ? trafficLights.ordinal() : -1) {
  case 0: {/* do stuff */}
  case 1: {/* do stuff */}
  case 2: {/* do stuff */}
  case -1 {/* do stuff */}  // the null case
}

Its a simple extension, but it closes a gap in the enum syntax sugar. Perhaps one for Java SE 7?

Days.daysBetween(today, christmas);

Using Java, how many days are there between now and Christmas? Its actually quite a tricky calculation to get reliably right (Did you forget to handle daylight savings?).

Now you no longer need to write it yourself if you use version 1.4 of Joda-Time:

LocalDate today = new LocalDate();
LocalDate christmas = new LocalDate(2006, 12, 25);
Days daysToChristmas = Days.daysBetween(today, christmas);

So what's going on here? Well its pretty readable really, but we are (a) creating two date objects (no time, no timezone) and (b) calculating the days between the dates. The calculation effectively includes the start date and excludes the end date. Issues with daylight savings time are avoided by using the no timezone LocalDate objects. But what if you want to calculate the number of days considering time, lets say 08:00 on christmas morning?

DateTime now = new DateTime();
DateTime christmas = new DateTime(2006, 12, 25, 8, 0, 0, 0);
Days daysToChristmas = Days.daysBetween(today, christmas);

Well, that's pretty similar! Internally, the code will count the number of whole days between the datetimes. Or put another way, how many 24 hours units are there between the two datetimes (except that it handles 23/25 hours DST dates too!). So, 08:01 on christmas eve returns zero days, but 07:59 returns one day.

And what's the other noticeable feature about the code above? Well, the result is being returned as an object, Days, not as an int. This allows your application to store a value in days, and actually know it is days and not some random int.

Up until now in Joda-Time, we have used the Period class for this. But that allowed you flexibility in storing a period such as "3 days, 12 hours and 32 minutes" - fine if you want that flexibility, but not if you don't. Days on the other hand can only store a number of days.

Now, you may be asking what is so special about days? Why can't I calculate the difference between months or hours. Well, of course you can! The new classes are Years, Months, Weeks, Days, Hours, Minutes and Seconds. They all operate in a similar manor, and they all implement the relevant interface, ReadablePeriod so they can interoperate.

Oh, and one last point. the factory method have a naming style that allows them to be statically imported on Java SE 5. Still not sure if I really like static imports, but at least the API supports the coding style!

As always, any feedback on Joda-Time is welcomed, here, mailing list or forum. With your help, the library can improve still further!

Generics, Iterables and Arrays - more banging of the head

Yes, I'm back again, with another Java 5 example that caused me to bang my head against the wall (see yesterday).

Lets recap the scenario - a CompositeCollection wraps a list of collections and makes them look like a single collection. Here's the class outline again: (I've removed the simple one and two argument methods from yesterdays example as they're not relevant to today):

public class CompositeCollection<E> implements Collection<E> {

  private Collection<Collection<E>> all;

  public void addComposited(Collection<E>[] arr) {
    all.addAll(Arrays.asList(arr));
  }
}

Well we know varargs doesn't work here from yesterday. What if we try to improve the functionality in a different way. What if we make it so that you can pass in any Iterable, wouldn't that make it simpler? After all, Iterable is the new super-interface to all things in the JDK that can be iterated around:

public class CompositeCollection<E> implements Collection<E> {

  private Collection<Collection<E>> all;

  public void addComposited(Iterable<Collection<E>> it);
    for (Collection<E> c : it) {
      all.add(c);
    }
  }
}

This is all fine right? I can pass in a List, a Collection or an array right? Wrong.

Why? Well arrays don't implement Iterable. Did you know that? Well, if you did then three gold stars to you! But it was against my expectations.

The reason I expected an array to implement Iterable is because the new foreach loop works on Iterable objects. Thus I naturally expected all arrays to implement Iterable. But it isn't so. And that made me bang my head against the wall. Again.

Now one reason for this may be that an Iterator defines the remove method, but that could easily have thrown UnsupportedOperationException as it is defined to do.

The second possible reason is that it clashes with generics. Consider the case where arrays implement Iterable. Of course an Integer[] would have to implement Iterable<Integer>. But that generification causes problems:

Integer[] arrayInt = new Integer[10];
Iterable<Integer> iterInt = arrayInt;  // assume array implements Iterable
Iterable<Number> iterNum1 = iterInt;   // illegal due to generics

Number[] arrayNum = arrayInt;          // always been legal in Java
Iterable<Number> iterNum2 = arrayNum;  // legal or illegal???????????

Thus, an Iterable<Number> can be obtained via one sequence of code (array assign) but not another (generic assign). So, the Java5 expert groups chose to not make an array implement Iterable - it would have clashed with generics.

And yet, the irony of all this is that there is in fact no risk here, and the final line could have been legal. Iterable is a read-only interface (wrt generics), so there wasn't actually anything that could go wrong!!!

To be fair, I'm only speculating on what the thinking was. All we know is that arrays don't implement Iterable, and the foreach loop is coded differently for arrays as opposed to Iterable instances.

And hence API designers will have to choose to make their API only accept Collection, and force users to use Arrays.asList. Or alternatively have two versions of the API method, one for collections and one for arrays.

Generics == bashing your head against a wall

Well, I've been spending a few hours generifying commons-collections recently. The jobs only just started, and my head hurts from all the times I've banged it against the wall. Is there anybody listening who might actually be able to communicate to the powers that be in Sun that generics are just way messed up?

This time, my problem is in CompositeCollection. It holds a collection of collections, and makes it look like a single collection. Simple enough use case. So here is a simple generification (showing the methods that add a composited collection to the collection of composited collections):

public class CompositeCollection<E> implements Collection<E> {

  private Collection<Collection<E>> all;

  public void addComposited(Collection<E>);
  public void addComposited(Collection<E>, Collection<E>);
  public void addComposited(Collection<E>[]);

}

Thats great! It works and we're all happy, right? Well, not really, because those three addComposited() methods can be simplified using varargs of course:

public class CompositeCollection<E> implements Collection<E> {

  private Collection<Collection<E>> all;

  public void addComposited(Collection<E>...);

}

Now isn't that a much simpler API. We can pass in an array, one, two, or any number of collections we like, right? Wrong.

The problem is that calling this method is totally screwed due to the messed up generics implementation:

public class MyClass {

  private void process() {
    Collection<String> a = new ArrayList<String>()
    Collection<String> b = new ArrayList<String>()
    CompositeCollection<String> c = new CompositeCollection<String>;

    c.addComposited(a, b);    // compiler warning!!!!!!!!
  }

}

Why is there a compiler warning? Because in order to pass a and b to the method the varargs implementation needs to create an array - Collection<String>[]. But the generics implementation can't create a generified array, yet thats what the method is demanding. Arrrgh!!

So, instead of being helpful, Java creates an ungenerified Collection[], and then the generics system warns us that it is an unchecked cast to make it a Collection<String>[]. But this conversion is all internal to the compiler! As far as the developer is concerned, there is nothing even vaguely type unsafe about what s/he is doing.

Now this is just headbangingly stupid. There is absolutely no risk of breaking the generics model here. No risk of a ClassCastException or ArrayStoreException. No, this is really just another case of generics being nearly completely broken with regards to arrays. And the varargs group didn't manage to realise/coordinate and get a decent implementation that works.

The problem for me is serious though. I am writing an API (commons-collections) that will be widely used, and will have to retain backwards binary compatibility. If I go with the varargs option (the logically correct choice), then every client of this method will get a useless java warning - not very friendly. The alternative is to go back to the original design and have three methods, an array, one-arg and two arg. And really thats just crap.