Tuesday 6 November 2007

Implementing FCM - part 1

Its been a long coding weekend, but after much head scratching, I might finally be getting the hang of changing the java compiler. My goal? to implement First class methods (FCM).

Implementing FCM

Currently, on my home laptop I have an altered version of javac, originally sourced from the KSL project which supports Method literals (section 2.1 of the spec). What this means in practice is that you can write:

  Method m = String#substring(int,int);

This will now compile and return the correct Method object. The left hand side can be a reference to an inner class such as Map.Entry without problem. It will also validate the standard access rules, preventing you from referencing a private/package/protected method when you shouldn't.

In fact, the implementation is currently more generous than this, as the left hand side can also be a variable. This is needed for later phases of FCM, but isn't really appropriate for a method literal.

Making a language change like this is definitely intimidating though, and mostly because the compiler feels like such a big scary beast. The first two parts - Scanner and Parser are actually pretty easy. Once those are done you have an AST (Abstract Syntax Tree) representing your code. In this case that involves a new node, currently called 'MethodReference'.

After that it gets more confusing. The main piece of work should be in Lower, which is where syntax sugar is converted to 'normal' Java. For example the new foreach loop is converted to the old one here. Since method literals are just syntax sugar, Lower is where the change goes.

Unfortunately, to get to Lower, you also have to fight with Attr which is where the types get allocated, checked and validated. That was a real head scratcher, and I'm sure a compiler expert would laugh at my final code. It does work though, and after all thats the main thing.

The next step will be to convert 'MethodReference' to 'MemberReference' and enable Field and Constructor literals. After that, the real FCM work begins!


So, by now you probably want to have a download link. But I'm not going to provide one. As the code is GPL, I can't just publish a binary, the source has to come too. And personally, I'd rather actually have it in a SVN repo somewhere before I let it escape at all (because that can probably count as the GPL source publication). So, I'm afraid you'll have to be patient :-)

This probably should go in the KSL, but right now I don't know whether it will. The trouble with the KSL is that it still has a few guards and protections around it. And to date, no-one other than Sun has committed there, even in a branch. I'm feeling more comfortable in creating a separate, 'fewer rules', project at sourceforge. In fact, it would be great if other javac hackers like Remi wanted to join too ;-)

One final thing... if anyone wants to help out with this work I'd much appreciate it - scolebourne|joda|org. Compiler hacking isn't easy, but its a great feeling when you get it working!


  1. How do you intend to disambiguate static method references from bound method references, given that they share an identical syntax?

  2. Hi Stephen

    Nice work, keep us posted if you post the code to some shared site. I think some sort of playground for javac would be nice to have for people learning their way around the compiler/compilers.

    One question--do you know if Method and Member count as references to an object instance? e.g. if I have a reference to a Method on an instance, will that prevent the instance from being garbage collected?

    Am also interested, as Neil is, about how you distinguish between static and instance-specific method references.


  3. Awesome... what else is there to say. Will be following closely and hoping we can get this power into Java before the end of the decade.

  4. Stephen Colebourne6 November 2007 at 12:59

    @Neal, My intention is that it is context sensitive. If assigned to a Method you get a Method, if assigned to another type it attempts inner class wrapping. Obviously, when passed as a method parameter that gets more complex. We'll have to see if I can work that one out.

    @Patrick, AFAIK, a Method is only Class based, and has no reference to any instances of the class.

    Oh, and any suggested names for a new project?

  5. Good work.
    I had the same experience with abstract enum "Dive in KSL", and getting out with a working compiler that passes the tests, What a feeling.
    I did not investigate the license issue, and I published on my personnal mercurial repo on www.jfrog.org/hg.
    Really like to see a open javac...

  6. It's really a breath of fresh air to see all the independent efforts going on in Java now that the source is GPL. Except for individuals at Sun or IBM, I couldn't imagine this kind of experimentation being possible before.

    Bravo to Sun for embracing the open source philosophy and making this possible. I think they'll be pleasantly surprised at the speed of innovation in Java because of this decision.

    Hopefully we'll all behave ourselves and keep the new features clean and well-integrated with existing Java. I'm confident that FCM+JCA will be recognized as a better way of enhancing Java's expressiveness, without uglifying the syntax.

    Looking forward to experimenting with FCM, keep up the great work!

  7. Hi Stephen

    My bad--I read too quickly and thought you were deriving executable method references (references to a method for a given instance). Silly of me. I had also forgotten that Method's invoke takes an instance as an argument.


    As for a project name--I would just keep away from anything with "java" in it to avoid running afoul of trademark.

    (comes after coffee)



  8. Name? Hmmm how about Cream. It goes into the coffee and improves the user experience. :)

  9. Stephen Colebourne9 November 2007 at 13:35

    I've registered a project at java.net. Still pending approval...

    (anyone who wants to hack the javac can contact me to discuss ;-)


Please be aware that by commenting you provide consent to associate your selected profile with your comment. Long comments or those with excessive links may be deleted by Blogger (not me!). All spam will be deleted.