Wednesday, September 06, 2006

The Debate over Closures for Java

We've been continuing work on the closures proposal. I'd like to give you some feedback on what I think about the debate. Tomorrow I'll post an updated proposal, and the next day I'll outline an alternative version of the proposal that avoids function types and contrast it with the current draft; some people think that might be a better approach, though we have some doubts that I'll explain.

Our publication of the draft closures proposal resulted in a great deal of debate and excitement. Even if we end up doing nothing, I think that is very healthy. These kind of things rarely get debated in such public forums. We've been listening to the debate as it rages (on scores of blogs and message boards, and the occasional email), and we're trying to adjust our thinking accordingly. My next post will be an updated version of the proposal, and the following will be an extended discussion of one alternative direction we're considering that addresses some concerns.

There are a few different kinds of feedback we've received, both positive and negative. To those whose feedback has been mostly positive: you've been no help whatsoever. Just kidding. Seriously, thanks for the support. We don't think things are perfect yet, and maybe they will never even be good enough to put anything like closures in the Java programming language. We need to figure out what, if anything, to change to make it better, so your criticism is welcome. If you can quantify precisely how much more fun Java will be to program in, that would help too.

To those with mainly negative or mixed feedback, I'd like to try to classify some of the most common concerns and give general, blanket responses. Please don't take these to mean we're dismissing your concerns. On the contrary, we expect the proposal to evolve, and your concerns are a primary source of information on how the proposal needs to evolve. This classification doesn't cover all the constructive criticism we've received or all the issues we know we need to address, and there is a great deal of overlap between these topics.

Closures don't buy you anything beyond what you can accomplish with inner classes. That just isn't true. See my previous blog post about use cases. Interfaces and inner classes solve the asynchronous use cases just fine, mostly, but they don't touch the synchronous use cases. I don't mean to imply that they can't be extended to cover both sets of use cases, but inner classes in their current form don't do it.

The addition of function types to the language is too much of a price to pay for closures. We have a few reactions to this. First, you're likely to be surprised how much you actually get with closures, and how little it actually costs you in complexity. While it might be possible to get closures without function types - see the alternative proposal for one stab at that approach - what you lose doing things that way is not trivial.

The structural typing of function types doesn't feel right in Java, which normally does its typing via programmer-defined named types. Actually, function types dovetail nicely with interfaces due to the closure conversion. The structural typing you get with closures is much like you would get using generic interfaces and using wildcards heavily in your code, except much simpler.

The proposal introduces a second way to do things that folks already do with interfaces. The concern here is that the language should provide strong guidance on how to do any particular kind of thing, and Java has already chosen to pass snippets of code around as objects that implement interfaces, typically via anonymous classes. Moreover, some existing APIs might look silly doing things the "wrong" way once a new way is introduced. I don't think this concern holds water because virtually all of the existing APIs in the JDK that I'm aware of are best written using interfaces rather than function types. I've heard the debate regarding the JDK notification APIs (for example, in swing) and what to do about them. I think they should be left alone.

Ordinary Java programmers won't be able to understand this stuff. Many folks who say this don't make it clear which parts of the proposal they find most troubling, or perhaps they don't see the proposal as decomposable into troubling and acceptable parts. The Closures for Java draft specification was written to be more like a specification than a tutorial. It is not a suitable thing to read to learn the language construct, any more than reading the Java Language Specification is a suitable way to learn object-oriented programming. Rather you would learn the feature by examples of gradually increasing complexity. I haven't provided that, and I apologize. I've tried to remedy that in my later blog posts but I'm afraid I'm not very good at that kind of writing.

If you believe that closures are inherently a difficult concept for programmers to understand, or some kind of language "trickery", I suspect you probably haven't done much coding in a language that supports them (see Steve Yegge's blog post Language Trickery and EJB, and Curtis Poe's article Sapir-Whorf is not a Klingon). In practice they are pretty straightforward and obvious. Things are complex with anonymous inner classes, and any tutorial would need to explain them as well. I will present an alternative spec in a few days that does away with function types entirely, syntax and all; with that change the spec may seem simpler than the previous one, but I believe as a practical matter it will be harder to use. Also gone are local functions from our proposal entirely. Those who identified function types as the main problem might be satisfied with the version of the proposal lacking function types. But if you have a hard time understanding anonymous inner classes, closures won't make things any easier as we can't remove anything from the language. There also needs to be a prototype implementation that people can actually use to get a feel for what closures can do. I eagerly await an open-source javac on which we can prototype this.

How much more expressive does the language become given the complexity of the additions? The gist of this concern is that adding closures seems to increase the complexity of the language substantially. Is the additional expressiveness worth the additional complexity? The best result would that we end up with an expressive but simple programming langauge. The question is: how much more expressive can you get with minimal changes to the language? We believe that closures adds significantly to the expressiveness of Java in an area that the language is severly lacking. The asynchronous use cases are handled just fine today, but the synchronous use cases like control abstraction and some kinds of functional programming idioms are very difficult.

Are you just doing this to compete with C#/Ruby/Javascript or add the latest experimental/research language feature? Closures have been an idea simmering in the background since the earliest days of Java, and it is the subject of repeated RFEs. Closures predate Java by many years. Closures were in Simula (the original object-oriented language), Lisp, and Smalltalk. They really do make some things easy that are difficult or impossible to express without them. Anonymous inner classes were a half-measure that made some use cases easier, but left some use cases unaddressed. Competition from other languages has shaped the political landscape and the programmer mindset to make closures more acceptable than they were before, so the time is ripe to propose them.

Values of function type can't be made serializable. Agreed, a problem in the previous spec. We intend that you should be allowed to "implement" function types so that you can make them serializable.

I can't stand the syntax. We've heard this before about things that people ended up loving once they started using them. We're open to alternatives. The alternative of just using anonymous class instance creation expressions doesn't work. Closures are not just shorthand for making an anonymous class, because names appearing inside the closure are resolved in the context of the enclosing scope instead of in the context of the anonymous class being created. Perhaps there is a better way of writing a function type. Perhaps we can improve the syntax for writing a closure - but it does require its own distinct syntax in order to get the expressive power. Your suggestions and ideas are welcome.

The synchronous and asynchronous use cases are so different that they deserve completely different syntax. I'm not sure about this argument. First, in the proposal the two kinds of APIs do use completely different syntaxes: one uses function types and the other uses interfaces. A closure means precisely the same thing (if it is allowed) whether used in one kind of API or the other. Any difference in meaning is because of what the API elects to do with the closure (for example, invoke it or not invoke it). Therefore, to understand a use of a closure, you need to consult the documentation or specification of the method it is passed to. In this respect a closure is no different than a value of any other type. I suspect I don't fully understand this concern, and we'll keep listening.

Closures will be implemented using the JVM invokedynamic bytecode; won't that hurt performance? The short answer is: no, not necessarily, and there are alternative implementation strategies that use existing instructions.

Compatibility with the existing language demands an inelegant/kludgy design for closures. Frankly, I don't believe it. I believe that the Java language in JDK7 can have closures AND seem to have been designed with them from the beginning, with virtually all existing APIs using the most appropriate idiom. Some minor syntactic challenges remain: closures would probably better be written with a new keyword (lambda? fun?), but can't due to backward compatibility; a related issue is the difficulty of using "," or "|" in the syntax for disjunction types for type parameters representing thrown exception types. I believe we are likely to converge on a syntax that feels quite natural once people start trying to use it.

A related argument compares the Java programming language to a delicious Oreo® Cookie: you can add syntactic sugar to its outside and still have a nice Oreo cookie, but if you disturb its delicious design center it just won't be the same anymore. Implicit in this argument is a warning (or veiled threat) that hordes of Java programmers will abandon the platform if anything so horrible is done to the language. I think this line of reasoning is overly melodramatic. In fact, I believe the addition of closures will attract a number of programmers who have previously had no choice but to seek alternative languages due to limitations of Java.

Closures are too powerful; programmers should not have the freedom to effectively define their own control constructs, because they will write programs that are too hard to understand. Instead, the argument goes, we should add new statements to the language for each of the important use cases we know about and leave it at that. The gist of this concern is that while the Java programming language today lends itself to easily understood programs, the addition of closures will result in a language in which readers of programs have no firm ground to stand on, and will not be able to tell what a program means. I've only heard this concern raised by a small number of people and our response (thanks to James Gosling) is: by the same logic we shouldn't allow programmers to write their own methods either. If you're reading a program with invocations of unfamiliar methods, you may have to consult the documentation of those methods to understand what the code does; if the code is well-written the name of the method will give you a good hint. This is true whether or not the method receives a closure as an argument. Related to this argument is the concern that the addition of closures will make it possible to write programs that are really hard to understand, but in reality poor programs can be written in any language. This line of reasoning is made by those who consider themselves advocates of the B&D (Bondage and Discipline) school of language design, but I would characterize it as the S&M (Sadism and Masochism) school. The subtext is that we should not trust ordinary programmers with anything so expressive as closures.

Closures should work with classes that have a single abstract method, like java.util.TimerTask, in addition to single-method interfaces. OK, sure, that sounds like a good idea.

There have also been proponents of various other languages like Smalltalk, Ruby, and Javascript noting with smug amusement that the Java folks (or some of them, anyway) finally "get" closures. To them I say "Pfffft!"

15 comments:

Anonymous said...

Thank you for update. I have two comments. The first one is actually a warning and it goes to your previous post with use-cases. Please make sure that proposed implementation strategy for closures does not automatically create a class file for every closure. Otherwise, your very nice use-case with timeOperation abstraction will be simply non-usable in practice due to extreme impact on the size of the compiled code in .jar file.

The second comment is about closure conversion. I do not think there is any need to include it together with function types. There are many cases where you will be forced to write a special helper method that converts a closure to an interface or abstract class implementation anyway (typical case is a multi-method listener interface). I mean, almost all existing uses of inner classes are asynchronous and the closure conversion in that case is often non-trivial and will require special helper methods anyway.

You can easily write such helper method for any interface or abstract class and clearly document what it does for this particular case. There is no need to clutter language specification with a special closure conversion that only covers a fraction of use-cases you encounter in practice anyway. You might object that with helper methods you are loosing some efficiency as compiler will not be able to recognize it as a special case to avoid extra object creation overhead. But JVM hotspot compiler (instead of javac source compiler) can recognize this as a special case. Moreover, including the corresponding code into JVM (as opposed to javac) is a better thing because it benefits more programs (even the code that is not using closure conversion, but adapts one interface to the other might benefit).

Neal Gafter said...

Roman: the helper method approach doesn't address the asynchronous use cases, where you don't want to allow nonlocal control-flow.

Jochen "blackdrag" Theodorou said...

I think, if you find a good solution for the typing and exception bloat problem and the possible side effects of non local returns, then it will bring Java a step forward.

The only sad thing is, that you possibly won't add an "each" method or such on List or Collection, becuase they are interfaces and can't give an default implementation. And I guess adding something like an ClosureEnhancedList interface which all lists implement and extends List is not possible.. or... why not? Of course with a better name, maybe not extending List, maybe not only on Lists but also on Maps and Lists.

Maybe I should say that something like that is nearly a must.

Anonymous said...

Neal: what language rules are violated by the following wrapper method that undermines closure conversion guarantees?


Runnable getRunnable(final void f()) {
return new Runnable () {
void run() {
f();
}
};
}

Neal Gafter said...

I don't know what guarantees you're referring to. The closure conversion constraints helps you write APIs that do the right thing, it doesn't prevent you from writing APIs that do the wrong thing. It is possible for separate threads to share data without synchronizing; that's nothing new. It is possible for someone using this code to get into a situation in which a nonlocal transfer isn't to a caller that is on the stack of the current thread; that is why we added the exception UnmatchedNonlocalTransfer.

Karsten Wagner said...

Why non-local-returns? Only Smalltalk has it because they needed it to implement if-then-else via blocks - but of course Java already has if-the-else. Otherwise it's quite untypical (for a reason), can lead to runtime-errors and makes the behaviour of code less clear.

What about iteration? Do you really want to add 'map/fold/each/...' operations to every collection class (and what about parallel-iteration then?)? Is it really possible to supply all relevant swing-classes with a second variant for the new closure? And if this isn't possible whats the use of the new closures anyway?

Java isn't a functional language, closures simply don't fit well into the 'Java-way' of doing things. In functional programming closures are 'a must' because iteration isn't possible there (at least in pure fp). But Java is an imperative language so iteration is quite normal and it would be much better to make the building and use of iterators more easy. For example by extending the language with provider/consumer-reversal (like in Icon, Sather, Ruby) like I wrote about in my blog.

Understanding closures isn't the problem, closures are in fact quite easy to grasp. The problem is that you need certain new usage patterns which conflict with already existing java-usage patterns. For example iteration via loop / iteration via closure. So the average programmer has to learn more usage patterns and always choose between the alternatives. So it becomes more probable that similar problems are solved once with closures and once without which results in less compatible/maintainable code.

I know people who say hat Java even haven't closures like Lisp, Python, Ruby or ... and that because of this Java sucks. Until know I've always told them that they aren't right, because Java HAS closures for some time now and until now nobody was able to show me an example which wasn't nearly as simple (but with a bit more typing) to do in Java with inner classes. It's a bit sad that Java folks now join those people.

Neal Gafter said...

Karsten: Why non-local-returns? Because the built-in control constructs aren't the only ones you need, and programmers now have to write very contorted boilerplate for similar things. See http://gafter.blogspot.com/2006/08/use-cases-for-closures.html and keep in mind that the blocks you're encloing may naturally have a return statement in it.

We're not proposing any change to iteration or callback APIs, and we do not believe iteration should be changed in the language or the API. It is possible that static methods along the lines you suggest could be added to utility classes, but I think you're imagining much more than should be done.

To say that closures don't fit into Java because it doesn't have them begs the question as to whether or not it would be better or worse off with them. You seem to assume that the addition of closures to Java requires that we reconsider things that could have been done with closures. I disagree; existing APIs and ways of doing things should be kept as they are.

Anonymous said...

funny that Karsten Wagner wants some new control structures
http://kawagner.blogspot.com/2006/08/java-enhancements-i-would-like-to-see.html

and many of them can be easily implemented through proposed closure.

Karsten Wagner said...

The question is always what's better: Create a control-structure via some abstraction (like closures) of build it in directly into the language.

Until now, Java clearly went the second way, which has the obvious advantages that there are only a limited number of control structures which are easy to learn and to use, because they all have a certain, definite purpose.

Other languages went another way, Smalltalk is a obvious example which has no control structures at all as part of the syntax.

The question is: Do we really want to transform Java into a totally new kind of language or isn't it better to improve Java as it is? I prefer the second way, because I think Java's biggest advantage is, that the language promotes a certain way of programming. Sure, that's NOT the prefered way of programming for Lisp-hackers or the like, but it has proven quite successfull in those areas of programming Java where is in fact quite successful. And I suspect by making Java more user-extensible we will lose this advantage an Java will become just one more of those multi-paradigm-languages without a real advantage over the rest of them. In the end this will kill Java because the language will lose it's prime advantages: Being easy to use and to learn and easy to promote a certain kind of coding which leads to easy maintainable and reusable code.

I know that this kind of thinking is hard to grasp for programmers with this certain 'hacker-mentality', which always search for better abstractions and more compact representations. But for those kind of programmers there are lot's of languages available, why not just leave Java as it is and advance it in small and 'Java-like' steps?

Unknown said...

I agree with Karsten Wagner. Furthermore, closures will allow programmers to subvert the JCP and create all kinds of control constructs without community review. This will be frustrating to programmers who become responsible for code filled with improperly implemented closures. The code could be difficult to maintain. Management can conclude that the system needs to be re-designed in a different language.

Java is an object-oriented language--objects have both attributes and operations. Closures are essentially operations without attributes. Thus they do not belong in Java.

The 'boilerplate' code that is intended to be replaced by closures is not that difficult to write. Both Eclipse and NetBeans support automated completion of anonymous listeners, try-catch, etc.

Neal Gafter said...

@Joe: The same argument can be made to disallow any user-defined code: Methods allow programmers to subvert the JCP by creating APIs without community review. This will be frustrating to programmers who become responsible for code filled with improperly implemented methods. The code could be difficult to maintain. Management can conclude that the system needs to be re-designed in a different language.

The 'boilerplate' code that is intended to be replaced by closures is difficult to read, and automatic generation of boilerplate by IDEs doesn't help.

Anonymous said...

Why not allowing method literals to be assigned to function types?

class Operations
{
public int plus (int a, int b) { return a+b; }
}

Operations ops = new Operations ();

{int,int=>int} plus = ops.plus.method;

//or

{int,int=>int} plus = ops.plus;

Anonymous said...

Neal!

Thanks for the proposal, closures really rock!

But the syntax you propose (obviously scala inspired) does't feel really "native" to may Java programmers, including me. I know that I would get used to it and I don't think your proposal is ugly or anything... just not Java-"natural" (propably causing much of the unnecessary part of the debate).

Just as an idea, could the call be aligned somewhat to the enhanced for loop syntax?!?

Instead of ...
add(3, {int b => b + 5})
... I would prefer ...
add(int b : 3){ b + 5 }

I.e. move the parameters passed into the block just before the parameters passed to the method, separated by a colon. Most programmers will mainly pass closures to methods, so this is the most important syntax.

Could the syntax for a function type not look exactly like a method signature? So instead of ...
{ => int } c = { => 42 };
int answer = c.invoke();
... something like ...
int c() = { 42 };
int answer = c();
I would love to be able to omit the ".invoke"!

Maybe the type of a no-arg function could even be infered ...
int answer = { 42 }();

The definition of the method called above would change from ...
int add(int n, {int b => int} f){
return n + f.invoke(7);
}
... to ...
int add(int n, int f(int b)){
return n + f(7);
}

I am not shure what troubles all of this would cause a parser... to be honest: I don't care much as long as it can be done ;)

The syntax seems not only to produce more concise code, IMHO it also looks more like Jara. It shure still needs a huge lot of thought to be even considered, but maybe...

Regards
RĂ¼diger

Neal Gafter said...

Anonymous: We changed from a syntax similar to this between versions 0.2 and 0.3 of the spec in part because of confusion regarding the semantics of return.

Peter Lawrey said...

"Closures are too powerful; programmers should not have the freedom to effectively define their own control constructs, because they will write programs that are too hard to understand."
I would argue that programmers are free to build their own control constructs, but do so with limited language support so how this is done is down to the developers taste.
They do write programs that are too hard to understand now.
Closures won't prevent from writing obfuscated code, but I believe closures will make it easier to write control structures in a standard way.