Scala feels like EJB 2, and other thoughts

At Devoxx last week I used the phrase "Scala feels like EJB 2 to me". What was on my mind?

Scala

For a number of years on this blog I've been mentioning a desire to write a post about Scala. Writing such a post is not easy, because anyone who has been paying attention to anti-Scala blog posts will know that writing one is a sure fire way of getting flamed. The Scala community is not tolerant of dissent.

But ultimately, I felt that it was important for me to speak out and express my opinions. As I said in my talk, if it was just me that had a poor opinion of Scala I would probably keep quiet (or try to figure out why I was out of step). But I perceive considerable uneasiness amongst many that have tried or looked at the language - something that reinforces my concerns.

Before I start I should mention that although I like the Fantom programming language, I also see merit in aspects of other languages - Groovy, Kotlin, Ceylon, Gosu, Xtend and many others. I also respect Clojure. By comparison, I really struggle to find positive feelings for Scala, and what positive feelings I have had have reduced over time. But why is that?

(For those not at my Devoxx talk, I tried to do two things - firstly to show Fantom off and explain how most simple comparisons to Scala rather miss the point, and secondly to point out some of the difficulties I have with Scala. To be clear, I'm not bashing Scala to promote Fantom. I'm bashing Scala because I think its entirely the wrong direction for the future.)

For this blog post Ive picked out a few key areas for discussion. I probably could have written a post 3 times as long as this one, and this one is very long as it is. There is lots to say about Scala, and very little is good.

Modules

Scala does not have a module system. By that I mean a deeply integrated system of modules that treats the basic program unit as something larger than a class, with versioning and dependencies. A key test is whether the new language compile modules or classes.

One of the greatest issues with Java is the lack of a module system. This absence has over time cause the platform to gain cruft (like CORBA) and struggle to shed it. The multi-year effort of modularising the JDK is evidence of how complicated this work is to do if not done from the start. And of course the Java platform will always have to support code not written in modules. Beyond the core JDK, most experienced Java developers have encountered the "Jar hell" scenario, where different versions of Jar files are required by different libraries and the ease with which it becomes impossible to assemble the whole application.

So, I have a clear sense that proper modularistion is a Good Thing, with all the versioned goodness that comes with it. (Managing change of a large application over time remains one of the largest problems faced by most large development shops, and one I don't see Scala tackling.) Over time, Java has evolved the Maven, Ivy and OSGi approaches to modules. Each has some benefits, but none are integrated into the platform itself, which is a significant disadvantage.

Yet, in a recent thread on modules, Scala aficionados claimed that Scala does have modules. In fact the opinion was clear - "see the object keyword", "Scala objects and path dependent types encode ML-style modules", "Also see http://www.mpi-sws.org/~rossberg/" (an academic paper). On further prompting, the ML view (standard source code can be used to express modules) was expanded on, before eventually the Scala approved way of using Maven/Ivy and the sbt tool was finally explained. There wasn't any real sense that this was a problem for Scala - so long as it integrated with the Java solutions that was fine.

I claim that integrating with Maven/Ivy is not fine. It misses huge opportunities to make life better on a topic where developers face real productivity issues in the field. Hence I commented that "Scala focuses on the wrong issues".

I also noted that backwards compatibility has been a constant problem of the Scala libraries. Modules are a tool for managing versioning and compatibility issues and would almost certainly have helped Scala evolve.

Finally, I noted that modules allow an application to find all the classes in the classpath/modulepath. This allows an application to find all the classes that implement an interface or annotation easily, which allows applications to be easily assembled from their parts. Java and Scala can achieve this, but only via complex and slow classpath scanning tools, like scannotation.

Concurrency

Scala makes a big deal about concurrency. About how the functional approach will aid the creation of safe multi-threaded code.

Except its really a bit of smoke and mirrors.

The big problem in concurrency is shared mutable state. It turns out that us developers are pretty bad at reasoning about it and using the tools at our disposal (synchronized, locks, java.util.concurrent) to manage that state. You'd expect that Scala would have tackled the concurrency problem at source - the shared mutable state - but it doesn't.

Scala (the language) does not know whether a class is immutable or not, nor does it provide a way to check is an object is immutable (Scala's libraries might help, but the language doesn't). As a result, it is perfectly possible to have a "static" (shared-thread) variable, or a "static" value of a mutable object. Its also possible to pass a mutable object to an actor and shared mutable state that way.

object Foo {
  var bar = "Hello"        // this is shared mutable state
  val baz = new Mutable()  // so is this
}

Tackling shared mutable state is not easy in language design. It involves designing the language to know about immutability, to track it, and to only allow immutable objects to be passed by reference to another thread/actor (mutable objects can be passed by copy). Done right, it eliminates the potential for concurrency issues from shared mutable state.

Scala relies on library design and discplined behaviour from developers to get this right (whereas Fantom builds this into the language, such that code equivalent to the example above will not compile). This is of course part of Scala's design approach - to give developers the power and trust that they will not abuse it. For me, this is simply another case of Scala failing to tackle the root cause of a big developer productivity issue.

Community

Scala has a loud and vocal community, especially amongst those on forums and twitter. Some of these developers have gone on to create libraries based on Scala, in all manner of areas, from web frameworks to database access. This can have the effect of making Scala appear to be the "upcoming destination" where other developers think they should invest their time.

Unfortunately there are some aspects of the community that are much more negative. Scala appears to have attracted developers who are very comfortable with type theory, hard-core functional programming and the mathematical end of programming. Frequently, there is the sense of a brainiac competition, and an awfully large amount of argument of whether solution A, B, C or D is the right one when in reality they all do exactly the same thing (Scala typically offers many ways to achieve the same end result, something that Java sought to avoid, and something that tends to create more heat than light in debates).

There is also a sense that many in the Scala community struggle to understand how other developers cannot grasp Scala/Type/FP concepts which seem simple to them. This sometimes leads Scala aficionados to castigate those that don't understand as lazy or poor quality developers, as being second class citizens. This can easily lead into derogatory comments about "Java Joe" or worse.

My experience is that most developers are perfectly clever people, and perfectly capable of understanding many things if they are explained correctly. The classic example is variance in Java generics, where ? extends is needed. I find that it is perfectly possible to explain the issues to a developer, who will pretty quickly grasp why a List of Integer cannot just be assigned to a List of Number. However, what I find is that once the discussion is complete, and the developer solves their immediate problem, the explanation will tend to slip away. The problem is not that the developer isn't smart enough, its that the complexities of the type system isn't important enough to care about. Understanding the issue at hand, management priorities, the problem domain and the architecture/design of the large system they are working on are much more significant issues.

The Scala community is also infected with modern societies desire for more, more, more without considering the consequences (more gadgets, faster car, bigger TV, more money, more spending, yet bankrupt people and countries). Scala goes all the way with its language features - everything is about maximum power. And the community revels in that power, finding and exploiting every corner case that the power grants, without truly considering the harm it does.

Type system

Every time I look at Scala it feels rather like the type system fits the phrase "if all you have is a hammer, everything looks like a nail". Whatever the problem, the type system is bound to be part of the solution.

The trouble is that a big type system is inevitably a complex type system. The concepts added to support the type system have their own terminology which is instantly inaccessible without significant learning, from high kinds to type constructors to dependent types... Its all just a baffling mess of type theory that provides no meaningful connection to actual work that needs doing.

The trouble is that despite the pleading of aficionados, method signatures like this abound:

 def ++ [B >: A, That] (that: TraversableOnce[B])(implicit bf: CanBuildFrom[List[A], B, That]) : That

If you don't know Scala, you wouldn't have a hope of attempting to understand the code. In fact this is the equivalent to Java's addAll() on a list. (Its so complex that Scala tries to hide it from you in the documentation by giving you a simpler form instead.)

Oh, and by the way, I do get the idea that the humongous type system is there to prevent compile errors and pre-check your code. But applying that logic in the opposite direction would imply that no-one gets any real work done in languages with dynamic type systems. For me, Scala's type system is way way beyond the point of sensible returns for a language feature.

Steve Yegge's analysis was perhaps the most fun:

The... the the the... the language spec... oh, my god. I've gotta blog about this. It's, like, ninety percent [about the type system]. It's the biggest type system you've ever seen in your life, by 5x. Not by an order of magnitude, but man! There are type types, and type type types; there's complexity...

They have this concept called complexity complexity<T> Meaning it's not just complexity; it's not just complexity-complexity: it's parameterized complexity-complexity. ... I mean, this thing has types on its types on its types. It's gnarly

Steve uses Scala to argue for dynamic type systems. I disagree, and consider a static type system to be useful for documentation, communicating intent in a team and for basic error checking. But I don't need the world's most complicated type system to do that - I just need something simple and effective.

In essence, Scala's type system is giving static typing in general a bad name.

Syntax

Scala's syntax is very wide open. It is the case that given a small piece of code (the kind that developers look at all day long) it is frequently difficult to reason about what that code does.

Scala has a big focus on flexible syntax with the aim of allowing a user to create a DSL in almost any form without having to write any parsing code. Instead, the developer just has to write a "normal" API, and use the language's syntactic flexibility to enable the ultimate end user to write in the desired style.

Take implicits, a technique that seems perfectly sensible to allow type conversions and object enhancements in a type safe way. Well, it may be type safe, but its also silent and very deadly. You can look a piece of code and not have any idea what is being converted. Unless you understand every import, every active implicit, their scope, their priorties and much more, you really don't have a clue what your code is doing. But thats OK, you didn't want to be able to understand Scala code did you???

Or take the fold operators and placeholder _, which produce delightful code like this:

 (0/:l)(_+_)

Thats practically the very definition of line noise. (And I've not even shown any scalaz examples, or similar unicode weirdness)

By the way, if you're looking at Scala, you may come across conference presentations, blog posts and forums that show small snippets of code in Java and Scala and show how much less code you have to write in Scala, and how much simpler it appears to be. This is a clever trick. The real complexity occurs when you start mixing each of the seemingly simple features together in a large codebase, on a big team, over a period of time, where code is read far more than it is written. That is when you start to appreciate that some of Java's restrictions are there for a reason.

Quality

When evaluating a language, its important to get a sense of the quality of the implementation. This is useful for determining how easy it will be to maintain the current language and extend it in the future.

For this, I turn to the analysis by the core Scala committer, Paul Phillips, in a Scala podcast in June 2011 (selected elements):

The compiler is, and the libraries and language as a whole, its awesome but the number of places where features interact is astronomical. In order to really get the lid on that many feature intersections we need a massively comprehensive test suite, that we simply don't have.
...
[Question:] Its been suggested that you are skeptical of community involvement because there is no test suite? You're afraid that if anyone touches it but you the whole world will break.
[Answer:] I, unfortunately, continue to be bitten by extremely subtle bugs that come out because of the inadequecy of our test suite.
...
[Question:] Where do you want the test suites?
[Answer:] Collections
[Question:] Anywhere in particular?
[Answer:] All of them. There's no reason that many many many of the bugs we've seen in the collections over the last couple of years should ever have happened because they should be exhaustively shown not to exist by virtue of the tests that we have, but don't have yet.
...
[Question:] And what about the compiler?
[Answer:] An exhaustive test suite for the pattern matcher would certainly aid me in the process of finally really fixing it. It would be very very helpful actually.

An incredibly complicated language with very few tests? Sounds like a poor foundation to build real world applications on to me.

Specifically, note this line - "the number of places where features interact is astronomical". This is a key aspect of Scala. That each language feature is orthogonal and flexible. Implicits mean that code can be inserted almost anywhere (which slows the compiler, listen to the podcast). The ability to drop method invocation dots and brackets for parameters (to achieve DSLs) makes the meaning of code non-obvious, and leaves no spare syntax space for future enhancements. And these things combine to make a good IDE a very difficult challenge.

(If you're evaluating Scala for adoption by your team, I strongly recommend listening to the whole 40 minutes of the podcast. It will help you understand just what the real issues are with Scala, the quality of the implementation, and how difficult the language is to evolve.)

EJB 2

The EJB 2 spec was in many ways the nadir of Java EE, where huge amounts of boilerplate, XML and general complexity were foisted onto the Java industry. The spec was widely adopted, but adoption was followed by criticism. Developers found that while EJB 2 sought to reduce complexity in building an enterprise application through an abstracted higher level API, in reality it added more complexity without providing the expected gains. Documentation, best practices and tooling failed to solve the basic design issue. Spring was launched as a greatly simplified alternative, and eventually the much simpler EJB 3 was launched, a spec that had little to do with EJB 2.

As a data point, I attended a formal weeks training course in EJB. At the end of the course I knew that this was a very bad technology and that I would recommend against its use at every opportunity. Scala has exactly that same feel to me.

So, at Devoxx I said that "Scala feels like EJB 2 to me". The language is a well-meaning attempt to create something with a higher abstraction level. But what got created is a language that has huge inherent complexity and doesn't really address the true issues that developers face today, yet is being pitched as being a suitable replacement for Java, something which I find to be bonkers.

At the moment, Scala is at the stage of thinking that better documentation, best practices and tooling will make a huge difference. None of these helped EJB 2.

In fact one might argue that Java's biggest flaw down the years has been the architectural over-engineering of solutions, when something simpler would have done the job. Again, Scala feels very much in the mold of that strand of the Java community, over-engineered rather than YAGNI or 80/20.

Of course, neither Spring nor EJB 3 are perfect, but the core concept of injection appears to be easy to grasp and the basic mechanism of linking them simple. In particular, having easily cut and pasted sections of documentation proved very valuable. Having code that is easy to grasp where problems can be tracked down without needing a PhD in type theory is a Good Thing, not a bad one. Having code where you can work out what it does without needing to know every last detail of the "astronomical" number of language feature intersections is a Good Thing, not a bad one.

Of course the upside for Scala of my EJB 2 comparison is that EJB 3 is a lot better. Thus, it is conceivable that Scala could reinvent itself. But I would point out that EJB 2 and 3 are essentially utterly different approaches, happening to share a common name. I would say that Scala would need a similar reinvention from scratch to solve its problems.

Summary

I don't like Scala. And that dislike is increasing. Specifically, I do not want to spend any of my future working life having to write code in it.

Had Scala stayed as a remote language, for highly specialist use cases (like Haskell or Erlang) then I would have far less of an issue. But it is being sold as the solution for mainstream development, and for that it is as utterly unsuited as EJB 2 was.

Update 2011-11-24: Rather than respond to all the comments inline, I penned a response blog.