Jean-Marie Dautelle has posted a question on Javalobby asking about the future of the JCP.
Java Community Process
This question leads into broader questions of governance in Java. Java is now opens source, yet we still have no real clarification on how that works or affects the JCP.
In particular, who is the 'architect' overseeing the development of Java, and its future direction? Is it a named person in Sun? A hidden magical process? Or hit and hope?
Some examples:
1) Language changes. I, and many others, have suggested numerous language changes for Java 7. But who decides which, if any, go forward? How do people influence the process? How do people register opposition to changes?
At present, the only approach seems to be to work for Sun in a senior position, or work in silicon valley. What I'm looking for is a direction, for example 'Java will add new language features, but only to fill in gaps in the current language', or 'Java will not add any new language features'.
Personally, I've argued strongly for language change. I believe that there are quite a lot of changes, many quite small, that would help 'complete' the language. But what I really need is that architectural statement that says whether there will be any language change. If not, then I can move on, perhaps to a different language.
2) Core library changes. What if I want to add a method to one of the core libraries, such as String, or Integer. How do I lobby for this to occur? Despite requests, it seems that this isn't worthy of a JSR. Yet, Google can propose a list of methods to add, and stand a big chance of succeeding.
Here, I'm arguing for an open JSR which examines all the core library utilities out there - Commons Lang/Collections/IO, Google Collections, Spring, Jodd, etc. and works out the commonality. Plus, anyone could propose their own ideas. But it needs organising, and it needs funding.
3) Modules. The shameless behaviour of Sun wrt OSGi shows the JCP in a very poor light. Anyone who believes that the JCP functions well should be examine this tragedy.
When JSR-277 started, the OSGi team were told that they were not to be invited onto the 14 person 'Expert Group' despite being probably the most experienced people on the planet in managing modules. Apparently, the expert group was full at 14. Today, it has 20 people. Pure politics.
So, instead of adopting a widely proven, top-notch module system suitable for everything from mobile phones to cars to enterprise, Sun has gone 'Not Invented Here'. It has desperately trying to reinvent OSGi and failed. The JCP now has 6 JSRs that could be solved simply by adopting OSGi. This is absurd.
Peter Kriens from OSGi has been repeatedly blogging about this, and how the weak requirements process at the JCP is a big issue.
And modules are the biggest change currently planned for Java 7, yet one that very few people are following or appreciating. Messing this one up will be a real blow to Java. Its like java.util.logging. Only far, far worse.
Issues
The issue is that JCP, and Sun's approach to Java, seems pretty broken at present.
1) Sun ignores the JCP legal agreements it signs (ie. wrt Apache Harmony). This destroys all trust.
2) The JCP still exists to rubber stamp proposals from its big company members. Basically, a company submits code they already have, and the 'Expert Group' really exists to peer review it.
3) Neither the JCP or Sun are giving any sense of direction to Java. There should be a clear sense of where Java is headed and what it is trying to be - a five to ten year vision.
4) The JCP is not eliminating duplicate JSRs (see OSGi). Worse, it is allowing the worse option to succeed.
5) Individuals have too small a say. They tend to provide the innovation and disruption that keeps the big companies honest.
Remedies
1) Sun must regain the trust of the community, and stop fracturing the Java landscape. It must offer Apache Harmony the correct testing kit.
2) I would suggest a slightly larger executive committee. This should guarantee 20%, or maybe 25%, of seats to individuals, yet still provide enough room for companies such as Spring Source to get in (who just missed out at the last ballot.
3) Sun needs to either lose its special status on the JCP, or have it properly acknowledged. The former would mean it having to face a ballot every 3 years. Given what Sun is to Java, this probably isn't realistic. So, I'd argue that the EC should be honest about Sun, and that it isn't counted in the members of the EC (ie. it is the chair). Thus, I'd suggest an EC of 1 chair (Sun), 15 other companies, and 5 individuals.
4) The EC needs to stand up to Sun, and force a few issues - notably licensing and OSGi.
5) The JCP needs to setup an architecture group that takes 3 months to create and publish a five to ten year vision for Java.
A real alternative
The real remedy may however be to kill the JCP. We may not need it anymore. Consider these words:
Modularization creates a market for solutions. It is no longer required to come up with a catch-all API in the JCP, which is too often designed under political pressure. For example, J2EE would never have been a success in a free market. It received its life because it was the “ordained” API for enterprise applications by virtue of its JCP origin. It was not until an external party like Interface21 showed with Spring that Enterprise applications can actually be lean and mean as well.
So, the real solution would be to redefine Java as a core kernel, and an OSGi module system, with a central repository of modules that could be downloaded on demand. Vendors could then group together ad hoc to build new modules for everything from date-time to JSF. The market would then decide the winners. Thus the JCP would simply be the guardian of the language syntax and core libraries, an actually manageable task!
Finally...
Finally, as co-spec lead of JSR-310, I have to say that money is a problem. Despite all of the above, I have given freely of my time to run a JSR (Date and Time) which the vast majority of developers appear to really want, but which Sun had no desire whatsoever to run. To achieve this I have taken a 40% pay cut for 6 months by working 3 days a week instead of 5.
So, my advice to anyone considering running a JSR is don't. If the JCP wants to encourage individuals to continue running JSRs, then perhaps it needs to start paying them to do so.
Couldn't agree more... especially regarding the process for new language features and changes to core classes. Like you, I don't know if one simply downloads OpenJDK, hacks a new feature to it and posts a link to the code somewhere... or if they have to go through the admin burden of a JSR.
ReplyDeleteFrom the look of Google's proposed changes (which carry the weight of Josh Bloch) and Neal Gafter's implementation of closures... I'd be tempted to think that JSRs are on the way out.
Doug Lea has a fantastic new fork-join library... but that's being flown under the (existing) JSR-166 banner. One has to ask if it is really getting more attention because it's a JSR, or because he has beta jars for download on http://gee.cs.oswego.edu/dl/concurrency-interest/index.html
I'd be glad to see the back of JSRs... I've been stung by the licensing of the documentation and even the API in the past. And isn't it a pain that most JSRs now come only as a PDF download... no zip javadoc. Implementations are discouraged. And if that's the case, then why on Earth make it a standard in the first place?
i agree almost everything, On the Osgi part i am not sure. The model seems to be more complex comparing with the proposed jsr (i am not an expert on that tough).
ReplyDeleteAnother idea would be, setting the Java platform free as a separate organization (like Mozilla foundation)supported by the big guns (Sun, Google, IBM, Red Hat etc.) this way Sun would release form the burden of Java development expenses. The projects (like current jsr's) accepted projects also would be properly funded.
A side note: despite the problems, you are doing a smashing work on JSR 310..
Great post Stephen. I'm frustrated with the process as well, and I prefer a 'free market' over the JCP.
ReplyDeleteAgree with everything you wrote, except the idea of redefining Java as the "kernel". IMHO Developers need some broad, stable, properly versioned/managed environment to depend on. I really don't like the way JBoss manages their AS components and releases (it contains something like 20 components and each of them is separately upgradable and has different qualities).
ReplyDeleteFrom a practical 'five years' perspective, OSGi is as bad as 277.
ReplyDeleteNeither supports source code as the primary artifact for a project, so from the perspective of a world where most of the software is Free Software (and that's where we are going in the next five years, in case people haven't noticed yet), both are pretty useless, as they only let you work with binary blobs. Forcing OSGi, Maven, or something similar into Java just means picking a different brand of binary-only legacy technology. Binary only tech is pointless in an open source world.
The Java world would have a lot to learn from the Free Software C, C++, and Linux worlds, that have largely dealt with those issues 5-10 years ago. It's both sad and funny that it keeps reinventing those wheels badly over and over again.
Considering the state of Java and all the redundant re implementations Sun is doing which are basically copies of open source projects, its clear they have no plan to truly open up... in spite of Java technically being open source. It's still very top-down driven, and without a strong leader with a vision its just on its way into legacy land. How I wish Sun would've gotten a hold of A. Hejlsberg before Microsoft did.
ReplyDeleteA much easier fix for the JCP would be to remove corporate members altogether, and only have individuals in there.
ReplyDelete@David, KSL has managed to accept no changes since it was launched. Kijaro has gained 4 branches in 2 weeks. More importantly though, is that creating a demo implementation doesn't help us know who will choose language changes, or how they will be evaluated. Where is the architect?
ReplyDeleteOn small language changes, my link was to a thread where the discussion didn't really have anywhere to go. It was closed. A JSR that owned this area would provide a named person to evaluate and own these changes, ie an architect.
@Sam, as I understand it, the fork-join library and closures are very closely related. Yet there isn't an overall architect ensuring that the two things go into Java at the same time (ie. its just Doug Lea holding back the library until closures happen)
@afsina, Part of the problem with 277 is that it will only address a subset of the problems people will have with modules. OSGi has the experience to tackle that broader range. What offends me most is that OSGi haven't been invited to the EG. This clearly demonstrates that JCPs are often about politics, not community/experts.
Should Java be set free like Mozilla? I think thats a perfect solution.
@Genady, I understand the concern about a kernel, but I believe that it can be done well. Its all about getting the versioning and compatability right.
@Dalibor, I don't have a problem with binary blobs. In fact I don't really follow the point. I certainly don't want a module system where everyone has to download source and compile it themselves.
I also don't believe that you can remove corporate members altogether. They are the only ones who have the cash to fund the coding effort required.
@Michael G, Agreed. Java is still clearly owned and controlled by Sun. And without a leader/architect.
I agree with most of the analysis but I do not agree with any of the proposed solutions.
ReplyDeleteHacking the JDK with features is the best guarantee to bloat. Many want new features but few features are need by many. In the OSGi we have the experience with Eclipse. They often add useful features to get things to work but then later when those features are standardized in a process they change because they are often more things involved than originally thought.
The purpose of Java is to have a single environment crossing many platforms. Both embedded, desktop, and server. To achieve this, some process that takes a certain amount of time to let thinks sink in is crucial. Also, as I argued often, we need to fight about the requirements, not the implementations. I have not seen an open source projects work this way. On the contrary, the Apache Avalon project was an example how things can go wrong. I think open source works great for creating implementations. However, if we want Java to cross many different platforms we need multiple implementations of a specification. The specification must consider the issues of multiple implementations. This is harder in detail but is much cleaner in the long run.
The problem is that for Java we need a standards organization where no single actor can control the overall outcome. However, this group would need a strong architecture board that also dares to say no because bloat is the biggest threat.
As I argued before, the modularity is key. If Java was a core VM with a powerful module system (class loading and collaboration model) we could have several standards in the industry and the industry would sort out which ones are the best. The only one we should agree upon is the core VM with its modularity layer.
Just to correct some misunderstandings in the replies. OSGi's API is significantly smaller than JSR 277 counting classes. Despite OSGi being an almost ten year old specification. I agree that we have much more text describing the specification (280 pages) but I dare the complainer to go through this text and describe why the functionality is not needed. OSGi is still very lean after all those years because the key architects dared to say no (well, most of the time :-)
About the use of source code or binary code ... Sigh. Java bytecodes contain as much information as source code. What C and C++ do is impressive with their compile time configurations but the customers I talk to nowadays are buried in an avalanche of compile time flags. One customer told me they did not dare to change certain flags anymore. The power of Java (especially when combined with OSGi) is that you deploy a "binary".
I have spent the last ten years trying to have one Java because I still believe that the holy grail is write-once-run-anywhere. If I look at Eclipse, then I think we are moving in the right direction. However, we need help.
Kind regards,
Peter Kriens
Hi Stephen
ReplyDeleteI agree with the sentiment that the JCP needs some reform. However, I disagree with most of your points, to whit:
1) Legal agreements: we only have Harmony's side of the story, not Sun's, AFAICT. The disagreement is based on the text of contracts which have not been made public, correct? The negotiations have also been private. Not much factual evidence to base comments (or criticism) on.
2) JCP rubber stamps--there are a large number of JCPs already approved--can you point to 5 which you feel were rubber-stamped? And doesn't that apply to OSGi, which you otherwise seem to like?
3) 5-10 year vision: What other language has a 5-10 year vision? Apart from maybe Perl 6, and we're still waiting for that. A long-term vision (outside of general commitment to portability, safety, etc.) seems unrealistic given the way our world is changing so rapidly. Still, what do you offer up as an example?
4) Duplicate JSRs--I don't know of any duplicate JSRs, can you name a few? Functionality in OSGi appears to overlap functionality proposed in a couple of other JSRs, but does that mean they are really duplicating effort? For example, two JSRs related to superpackages and modules allow you to use them independently, which seems correct--given that we haven't had superpackages until now, modules in their first release would likely just repackage libraries currently delivered as JAR files, without changing package structure (which would likely be a breaking change to library APIs in terms of access).
4a) Also, I can't see OSGi as a model for anything--I know of no one who has used this in any project I've worked on or been told about (tho I realize it is used under the covers by some tools). Let them win in the public sphere, first.
5) "Individuals" is too loose a term. I'd argue we need subject matter experts, both from academia and those with real-world (long-term) development experience. Doug Lea and Bill Pugh seem to be good examples of academics whose expertise has benefited JSR development. I don't know that we need a bunch of well-meaning, strongly opinionated individuals on the JCP--what exactly would that buy us? The suggestion means nothing in and of itself--individuals can also be advocates for big companies or for "solutions" marketed by big companies. As a specific proposal, I'd argue we need to have a certain % of academics on board the JCP, and a % of individual "hands-on domain experts" on each JSR. That said, my guess is if Google joins a JSR on search, their contribution to the expert group would already include those individuals.
Again, I agree we need reform, but disagree with your specific criticisms.
Regards
Patrick
@Patrick, Thanks for your comment.
ReplyDelete1) I am a member of the ASF, and as such have received a limited amount of non-public information.
2) I refer to the fact that it is not uncommon to see JSR voting comments that do not appear to be resolved.
3) The key example is language change. Is Java to keep on growing, to ensure it remains 'cool'. Or is it to be frozen where it is. Or if there is to only be minor language change, what is the overall goal of that (eg. 'ease of use' or 'type-safety' or 'filling in the gaps').
The same broad vision could be applies to the libraries (eg. is scientific oriented code suitable for Java platform releases, eg. JSR-275).
And also the JVM. For example, a vision here would say 'the JVM will be extended to better enable non-Java languages'.
4) OSGi already is popular, with integration from Eclipse, BEA, JBoss, Spring... My comment wrt duplicates is OSGi based, where theie implication is that 6 JSRs could be closed simply by adopting OSGi.
5) My take is that individuals would challenge the status quo on the EC. Your idea for EGs is probably also useful (but difficult).
The reason why Free Software C/C++ world is years ahead of the Java world, and has largely avoided being stuck in the JAR hell to the extent the Java world has been is due to the source code being the primary artifact, instead of binaries.
ReplyDeleteThat does not mean that an end-user needs to recompile modules in order to use them, it just means that modules can be rebuilt *on demand*. I currently happily use my browser, editor, shell, desktop system, with joyful modularity all over the place over thousands of packages, along with millions of other people, without having to recompile them. I could, though, if I had to fix a bug, or customize them, etc. quite easily do that since my distribution (Debian/Ubuntu) is set up to make that process trivial, and the C/C++ world makes the distribution's job easy, too.
That's what really crucially matters in order to avoid JAR hell: being able to verify that metadata associated with a binary, created at build time, is correct at the time of use of the binaries, and being able to fix the problem, otherwise.
And of course, being able to rebuild a module on demand makes modules a lot more useful, for environments where the original author has no access to or interest in. If you've worked with multiple revisions of the Java platform across multiple, differing vendor implementations, you probably know the feeling of having to work around bugs either in the modules due to the developers making mistakes, like assuming some unspecified com.sun API exists everywhere, or due to the vendor implementations simply having issues that are triggered by the module code.
I've never seen bug free Java code, any more than I've seen bug free C code. But the C open source developers somehow get that, and know that access to source code that lets people reproduce the binaries they produce easily is a major differentiator between being stuck with statically whacking everything together (the bad old bundle your JARs with your app model), dynamically whacking everything together and hoping it works for the end users, too (the fancy new 'download your dependencies dynamically' model), and being able to verify and guarantee the consistency of a distribution by cross-building everything.
As long as none of the modules systems gets the last part, we'll be stuck in JAR hell, only that this time around the JARs will be managed inside the VM, downloaded over the network, etc. which is certainly a tiny bit of progress, but does not address the fundamental issue of why we are in JAR hell in the first place: because it is impossible to do integration testing of JARs without rebuilding them, and no one is able to do that in an easy way, so stuff breaks all the time when dependencies get updated, and the metadata ends up being stale most of the time.
Language Change: I'm usually arguing against language changes. I'd like to see language changes (e.g. closures) remain very difficult to get through the JCP, but I agree that library changes (e.g. rename a method for consistancy) should be much easier. Your idea of a single, broad JCP sounds great.
ReplyDeleteThe current JCP process is well documented - your "what if" scenarios do have clear, specific answers at jcp.org
As for a 5-10 year plan, I don't see the need or practicality. The only existing example I can think of is with all MS languages, the unspoken plan is "we'll do whatever the heck we want with the language, the fact that it's an official standard notwithstanding".
As for the issue of corporate control vs. individual control, keep in mind that almost all of Java today was designed by people at corporations (like Sun and IBM). I really don't see where this "individual will do a better job" thinking comes from.
On the issue from other posters about C/C++ and JAR hell: if we're using C/C++ as our model for where Java should go (as a language and library), then we've jumped the track. Same goes for us seeing "JAR hell" as more of a problem than "DLL hell".
At the end, you imply that Sun should pay you for working as JSR EG lead. Doing so would presumably make you more productive. And so you'll have gotten more productive by simply changing from an "individual" to a "representative of a corporation". So much for the JCP needing more individuals and less corporations.
@Andy, have a look at how the bulk of truly open source stuff is developed. It can still very much be sponsored (Novell, Canonical etc.) but those sponsors have no special rights i.e. no JCP veto and they do not sit on trademarks. I still think there's a gap between true open source and how Sun (and some of the Java community) interprets it.
ReplyDeleteI also think that Sun should perhaps invest more wisely (into the language) rather than wobbling back and forth (i.e. armwrestling Adobe, a fight they have already lost). Assisting Stephen would be a smart thing to do, for clearly nobody inside Sun has ever been able to get date/calendar right.
I don't have to use C/C++ as an example. ;)
ReplyDeleteI could equally well point to Perl, Python and Ruby, who all have successful solutions for the same 'modularity' space. What do Perl's CPAN, Ruby's Gems and Python packages have in common? They all use source code as the fundamental unit of software exchange that everything else is built from. They don't obscure access to the source code needlessly.
On the other hand, what do all the modularity solutions for Java (JARs, OSGi, Maven, Ivy, 277 etc.) so far have in common? None of them considers access to source code to be important. Oddly enough, none of them has managed to achieve the level of acceptance in the Java developer community that matches that of the successful modularity solutions in other communities.
If there only was a simple, obvious reason for that ... ;)
Danny Coward, the spec lead for JDK7, has made it clear there will be language changes in JDK7 and outlined the kinds of changes being considered. More details will hopefully be revealed when (if) the JDK7 platform JSR is filed. If you want to influence these things, it matters more what you do rather than where you live or how often you blog. Complete your language prototype and put it out there for the world to try, and write about your experience using it. If it catches on others will write about it too.
ReplyDeleteThanks all for the thoughtful comments.
ReplyDeleteSource vs Binary. I disagree with the view that Java modules have failed because they are not source based. IMO, versioning and dependency are bigger issues. However, I would accept that access to source as an equivalent artifact should be a goal, as it does simplify many tasks. Basically, its a balance. Forcing the module system to be source only, simply wouldn't work IMO.
As to why Java modules haven't taken off yet, I'd say that was complex. Maven has always had implementation issues, Ivy has never been recognised, while OSGi has slowly been growing (but starting from the mobile space, which may be a Good Thing in the long run). Plus, many managers still only accept Sun's view of the world.
Individuals vs Corporates. I've tried to carefully balance the needs of both. The practical reality is that companies matter because they have cash. But currently, the price they extract in return (proprietry APIs, RI, TCK, and worse) is causing a lot of pain. I see the role of individuals as challenging the process, being the movers and shakers that can drag the companies kicking and screaming to a more balanced world.
Money. My comment about the cost of running a JSR is really there to serve as an example, and a warning. The task takes time, and should IMO be viewed by all as a job, and thus worthy of financial recompense.
@Neal, I was not aware that language change 'will' occur in Java 7. Can you provide a link? I was aware that change was under consideration, but you seem to imply a lot more. If so, then where is the open process? Where is the clear page saying here is what we have considered? What is rejected? What is being worked on? How to get involved?
ReplyDeleteThere are a million questions here, and IMO the community management from Sun is entirely lacking.
"I was not aware that language change 'will' occur in Java 7. Can you provide a link?"
ReplyDeleteSure: http://blogs.sun.com/dannycoward/entry/channeling_java_se_7
"Where is the clear page saying here is what we have considered? What is rejected? What is being worked on? How to get involved?"
You're looking for the JDK7 platform JSR, but as you probably know Sun/Danny are holding that pending resolution of licensing issues.
Given that you're a spec lead, you know very well how the community gets involved in the evolution of a JSR, and the typical processes expert groups use to make decisions, and why and how the expert group is necessarily limited in size. You also know that implementing something outside the JCP (for example, Jodatime), and building a supportive community for that change, is one of the most effective ways to trigger change in the platform. I believe you know how to file a JSR. I believe you also know how the JCP executive committee, which has the final say on JSRs, is elected by the community.
@Neal, I would say that blog from 14 months ago (which I have seen before) is far from clear on the topic, although I can see it being read as a positive 'there will be change'. More significantly, I would hope for an *ongoing* dialogue from Sun.
ReplyDeleteYou suggest raising JSRs as an appropriate means for change, however I disagree. Raising a JSR requires a commitment to run the JSR. Personally, I would raise 20+ JSRs tomorrow if I had the time and money to do so, but I don't, so I can't. As you know I'm sure, JSRs are a tremendously limiting and highly political process.
I would also note that not all JSR EGs are limited in size. JSR-310 is effectively unlimited, as there are no private EG mails, only public ones.
Versioning & dependency management are easy to solve (and have been solved many times 10+ years ago by many Linux distributions across runtime technologies, there is a body of academic research on all that). They are rather straightforward once one comprehends that metadata over binaries alone is not sufficient for a module system (and that's all current systems really implement, largely for political reasons).
ReplyDeleteI'm not saying that the module system should be source only. It should be source-based. Binaries are very useful as second class artifacts (and that's where the proprietary modules can have a happy place of their own), but ultimately, most of the proprietary modules will transitively depend themselves on open source artifacts. The current designs of all the fundamentally broken attempts at module systems in Java happily ignores that fact, and make everyone's code suffer as bad as proprietary modules do, completely missing the 'source' part of 'open source'.
Binary-only module systems don't scale out of their niche usage, as proprietary Java vendors various attempts at achieving that show. Source-only systems wouldn't scale either, though they probably provide a convenient straw man to bash for the proponents of binary-only module systems. We need both, and source needs to be king, for the module system to work.
Hmm perhaps Stephen should move to India:
ReplyDeletehttp://blogs.sun.com/webmink/entry/getting_paid_to_develop
Stephen - I vote you for benevolent dictator !
ReplyDeleteHaving a Czar over the JLS would be nice (the platform is perhaps less controversial). As long as it is someone outside of Sun (their stewardship in recent years has been terrible, and they only appeared to open source it out of desperation).