JEP 109: Enhance Core Libraries with Lambda

Summary

Enhance the Java core library APIs using the new lambda language feature to improve the usability and convenience of the library.

Goals

The primary goal is to modernize the general library APIs by adding the use of Lambda in suitable locations. Most implementations will be provided as extension methods upon existing classes. We will target high-traffic areas of the library and add Lambda APIs where we think it will have the most benefit. Ideally, Lambda would appear in the API wherever a programmer familiar with Lambda would expect it to be. Alternatively, we'd like mainstream programmers to stumble over Lambda APIs in the library and think, "Oh cool, they added a Lambda here, and that lets me solve my problem more easily."

A secondary goal is to inform the design of Lambda language feature by using Lambda in library APIs, calling them from real code, evaluating the results, and providing feedback to the Lambda language/compiler team.

Goals can be summarized as follows:

• Introduce a new idiom, lambda functions, to the existing libraries;
• Improve utility and convenience of libraries with lambda functions;
• Demonstrate best practices for extension methods; and
• Demonstrate innovation and evolution of the familiar core libraries.

Non-Goals

It is not a goal to use Lambda in every possible place it could be used, nor will the scope extend beyond core libraries. For example, client, XML, and CORBA are not covered by this effort.

Very little new functionality will be added to the core classes with this enhancement -- only new ways of doing familiar tasks.

There are no specific goals for the proportion of APIs that are upgraded to use Lambda, that is, nothing like "we will add Lambda to xx% of the library."

Success Metrics

Success will be evaluated based upon the degree to which the new APIs and features are adopted by developers. For complete success use of the Lambda features will become the default idiom for using the core libraries by Java developers.

Motivation

"Language design is library design.
Library design is language design." -- Andrew R. Koenig

Java 8 will include a new language feature called Lambda. Having this feature in the language is useful, but with just language changes for Lambda, the platform is incomplete. It will make the platform much more valuable to have Lambda support added to appropriate areas of the library API.

In the past several years, a variety of new programming languages have emerged and are gaining popularity. Most of these languages have some kind of block, closure, or first-class function construct. While Java is still the #1 programming language, the common view is that it hasn't kept up with recent developments in programming languages. This is well recognized as a motivation for the Lambda language feature itself. However, it's also necessary to consider library APIs that support the use of Lambda. The alternative languages all have libraries, and their APIs are tuned to work well -- smoothly and idiomatically -- with the closures or function objects provided by the language.

Similarly, with Java, we expect that as time progresses the use of Lambda will become widespread and various coding idioms will develop around this feature. The Java library APIs will need to be enhanced to support idiomatic usage with Lambda alongside their current, conventional usage.

Description

This project has two phases:

1. A survey phase to identify candidates for Lambda API enhancements; and

2. A scoping and implementation phase to prioritize and select a subset of the candidates and to implement them.

Candidate Survey

There are several approaches to discovering candidate sites for adding Lambda-based APIs. One approach is to examine other systems with similar language features and to look at their libraries and see how they use closures and functions.

Consider Ruby for instance. The Ruby language supports a variety of first class function-like constructs. These are useful in and of themselves, in that they enable programmers to use a functional style of programming, to create higher order functions, to compose functions, etc. In addition, since the Ruby class library was developed along with a language that supports first-class functions, many APIs in the class library make use of them. We can look through the Ruby class libraries to find examples that we can use as inspiration for potential enhancements to the Java libraries. Some of these examples from Ruby's class library include the following:

• The Integer class has times, upto, downto, and step methods that support a variety of arithmetic iteration constructs. This removes the need to have a variety of such iteration constructs defined in the language.

• The File class has a variation of the open method that takes a block. The file is opened, passed to the block, and is closed when the block returns. This helps avoid resource leaks, and it reduces the need to have special execute-around constructs in the language (such as Java 7's try-with-resources construct).

• The HTTP API classes have several methods that take blocks. For example, the HTTPResponse class has an each method that calls a block that takes (header, value) pairs.

• In the Tk binding for Ruby, the widget creation method new takes a block that is executed in the context of the newly created widget, which provides a convenient and concise way of initializing the widget.

It's clear that there are many opportunities for using Lambda outside the collections classes. There is no requirement to implement Lambda-based APIs everywhere that Ruby does, but given Ruby's popularity and mindshare it seems reasonable to look to Ruby's class library for initial ideas. It might also be reasonable to look examine other systems, such as Groovy, Scala, Python, Clojure, and Smalltalk to find similar inspiration for Lambda use in library APIs.

Another approach is to look through existing Java code -- both within the library and outside, e.g. from the Qualitas Corpus -- and do pattern matching or synthesis to discover candidates. A set of techniques might be as follows:

• Run through the APIs and add an each() or forEach() method to anything that plausibly contains a collection of something, or that can be iterated over, even if it's not a collection. For example, a forEachLine() method might be added to java.nio.file.Files that calls a Lambda for each line in that file.

• Look for opportunities to use the execute-around idiom. For example, files must be closed after use, locks must be unlocked after use, etc.

• Look for classes that implement Iterable and consider whether a Lambda would be useful to invert the flow of control.

• Look for for-loops and while-loops in code and consider Lambda for inverting the flow of control.

• Identify abstract classes with one or two abstract methods, and consider adding constructors or factories that takes lambdas. Also consider this for interfaces. The pattern is where implementation is to be "filled in" via overriding; consider how to convert this into passing a Lambda as a parameter.

Scoping and Implementation

The resulting set of candidates will probably be too long to implement in the JDK 8 time frame. They will need to be prioritized and then truncated to fit into the available schedule, with a given set of resources (people) allocated to the project. Prioritization can be according to subjective importance of the library, but it would be good to have some usage data to back it up. For example, it might be useful to do a survey over the Qualitas Corpus to determine which areas of the library are the most heavily used.

Language Design Issues

During API development using Lambda, the following issues should be considered and relevant information should be fed back to the Lambda language design team.

1. Exception transparency: do you find a need to write lambda expressions that can throw checked exceptions? do you need extra sets of SAM types that have 'throws' clauses, maybe generic 'throws E'? is the result extremely tedious?

2. Variance: Utility methods for functional interfaces will probably make heaver use of wildcards than anything else in the API; is this extremely tedious? Would it be useful to have better/less verbose variance support?

3. Unboxed overriding: Would it be useful in the design of functional interfaces to allow primitive/void return types to override reference return types, such as Predicate<T> <: Function<T, Boolean> for example?

4. Abstract class functional interfaces: In the previous item about abstract classes for, is it the case that there are so many useful candidates that it would be nice to directly support abstract classes as functional interfaces, rather than having to manually define a subclass?

5. Generic-method functional interfaces: Do you find it useful to have a functional interface that has a generic method, like: interface MapFactory { <K,V> Map<K,V> make(); }. Note the significant point here is that it is the method that is generic, not the interface. This is not currently supported; the benefit is that new type arguments can be inferred for every invocation, which may be useful in some applications.

6. Chaining inference: In expressions like 'foo().bar(23)', the type arguments for 'foo' are inferred independently of the 'bar(23)' part of the expression. We plan to improve the use of context in inference for many cases, but we're still looking for some experience that would justify taking it this far.

7. Method reference disambiguation: Do you find a need to explicitly disambiguate method references (when the method is overloaded or there's a static/instance clash), either by writing a full signature or giving up and using a lambda expression instead?

Testing

Tests will need to be developed for each new API that is added to the system. New APIs are largely independent of each other, so it should be simple to test them individually. Furthermore, the API enhancements are likely to consist largely of adding new methods that do not impact other APIs in the same class. So, testing of these new APIs should be fairly straightforward and should not impact existing tests that use current APIs.

Risks and Assumptions

Adding an additional programming idiom increases the complexity for new users. If Lambda APIs are added piecemeal all over the system, they may tend to diverge in style. It would probably be beneficial to do a survey first and to develop a consistent style that is applied across the library.

One point that mitigates risk is that since the API enhancements are largely independent of each other, it is possible to change scope without too much project impact. That is, consider the scope to be a prioritized list of API enhancements. A line will be drawn at a certain point depending upon project schedule and staffing. If this line needs to move for any reason, this should have no impact on work already done, and little replanning should be necessary because the items are largely independent of each other. For the same reason, it should also be possible to reprioritize the list without too much effort, should new information become available while the project is underway.

Using these new APIs intermixed with the existing APIs has the prospect of making user code bases more complicated to maintain.

Dependences

This work depends only on the actual lambda implementation.

Even this is a fairly soft dependency, as lambdas are expressed as SAMs in the API. It's possible to add functional-interface-based APIs even before lambda has been integrated, though it won't be possible for callers to use lambda at that point.

Impact

• JCP: These are all changes to the public API, so they will end up modifying JCP-controlled specifications. These change to existing core library classes should be small enough not to require their own JSR; they should be covered under the platform JSR. The new classes required (functional interfaces) will be designed and reviewed by the JSR 335 EG. Of course, the appropriate review processes must all be followed.

• Other JDK components: Lambda implementation

• Compatibility: API changes are unlikely to introduce incompatibilities.

• Documentation: There will probably need to be some new tutorials to show how Lambda can be used with the new library constructs, or to have a series of small examples illustrating common usage. It may be that Lambdas are difficult to discern when looking through javadoc, so it might be useful or necessary to have some new javadoc syntax or possibly different javadoc output formatting to highlight in some unique way the APIs that use lambda. An additional possibility is to make sure that the javadoc for each of the new Lambda-accepting APIs includes an example of its use.