JEP 291: Deprecate the Concurrent Mark Sweep (CMS) Garbage Collector

2016/07/25

Attendees *present

Jungwoo Ha <jwha@google.com> Google *
Chuck Rasbold <rasbold@google.com> Google *
Jeremy Manson <jeremymanson@google.com> Google *
Janice Gee <janiceg@google.com> Google *
Jon Masamitsu <jon.masamitsu@oracle.com> Oracle GC at Oracle *
Dalibor Topic <dalibor.topic@oracle.com> Oracle Product manager for OpenJDK *
Bernard Traversat Oracle managing the JVM development team *
John Coomes <jcoomes@twitter.com> Twitter
Andrew Haley <aph@redhat.com> Red Hat *
Volker Simonis <volker.simonis@gmail.com> SAP JVM team *
Kirk Pepperdine <kirk@kodewerk.com> jClarity GC tuning for customers in the field *
Martijn Verburg <martijnverburg@gmail.com> jClarity *

Background

http://openjdk.java.net/jeps/291

Goals for the Meeting

Understand Oracle’s intentions concerning the CMS collector.
Understand everyone else’s motivations for keeping / using the CMS collector.
Understand parties’ levels of commitment to / openness to community maintenance and development. Discuss and brainstorm what forms this could take, and concerns about it.
Develop action items and roadmap for next meeting (possibly in August).

Action Items

Jeremy - schedule a followup meeting in about a month and invite the Shenandoah engineers. The agenda for this meeting will be to 1) Andrew - understand the status of Shenandoah and 2) Jon - draft proposal for refactoring.
Jeremy - schedule an informal F2F at JVMLS to build the social cohesiveness of the team
Jon - put together an early draft proposal of the separation for the next meeting. How much will be static? How many build time vs. runtime flags? List classes of changes functionality-by-functionality not line-by-line. Requirements for what you want and don’t want in the tree. What do we need to get rid of?
Andrew - for the next meeting, get status from the Shenandoah engineers about where they are in the refactoring of the interfaces, if they think that things are in a good state and ready for others to take a look at it.

Summary

Partners are interested in having CMS continue to be supported, as there are scenarios (substantially increased memory and CPU cost) that may not meet customer’s needs. Oracle will continue to provide support for CMS with JDK 9, but future JDKs are TBD. Partners are interested in moving CMS to the open source community where we can continue to add enhancements.
Separation of CMS is the key short-term engineering goal. It would be good to have simple, clean GC interfaces. The team will look at the feasibility of leveraging what Shenandoah has done / planned.
We need to figure out the effort and complexity of separating CMS and getting the ecosystem (workflow, tools) working for community development.
The developer workflow and tools look like they will work for community support.
We will communicate via the hotspot-gc-dev mailing list for now, until that no longer meets our needs.
Oracle does not consider CPU / memory cost to be a blocker for G1’s being the replacement for CMS. They have ideas to improve it in the future, but it is not on their roadmap.
Partners are interested in providing binaries of future JDK releases with CMS available, as building OpenJDK can be an inhibitor

Notes

1. Understand parties’ levels of commitment to / openness to community maintenance and development. Discuss and brainstorm what forms this could take, and concerns about it.

Summary of Oracle’s intentions for JEP 291

Jon: it is pretty much what is written in http://openjdk.java.net/jeps/291. We believe that G1 is now a better collector in some scenarios and want CMS users to migrate to G1. We would like to announce the deprecation in JDK 9. But we have not decided when we will stop support - maybe sometime ~ JDK 10. G1 is not better in all scenarios, but it is expected to be better for many applications. We would like to minimize effort on CMS.

2. Understand everyone else’s motivations for keeping / using the CMS collector.

Jeremy: what metrics and benchmarking are used?

Jon: throughput and pause times (mostly avg and outliers). We are working towards getting G1 to work on the Oracle products. We are making a lot of progress and we think that this will be reflective of other applications being run

Jeremy: we have found in tests that G1 uses substantially more CPU resources.

Jon: we have seen it use <10% more resources and we are thinking about ways to address this. You can also change the number of concurrent threads run.
Jeremy: we found that the avg cost to the mutator is due to the additional cost of the write barrier for G1
Jon: we have thought about that we measured it in terms of additional CPU and lower throughput
Jeremy: we found numbers of higher than 10% closer to 20% for CPU. 10% is untenable. We cannot even consider 20%!
Kirk: 20% is what what we see in applications. Pause times are all over the map.
You have to use more memory with G1.

Kirk: Are we going to focus on the G1 problems or discuss CMS support?

Jeremy: we want to ensure that we are all on the same page with the cost of deprecating CMS and moving to G1. When users have data centers full of machines running Java, spending >10% more CPU / memory is not tenable. If we open up this to community, that would be good.
Jon: CPU / memory cost is not a blocker for G1. But we recognize that it can be improved in the future and have some ideas

Jeremy: Is there someone else who wants to have CMS continue to be supported?

Kirk jClarity: there are scenarios and customers that cannot move due to the way G1 works. I cannot do anything about it, but I want to see CMS continue as some customers are not able to take advantage of G1.

3. Understand parties’ levels of commitment to / openness to community maintenance and development. Discuss and brainstorm what forms this could take, and concerns about it.

Jeremy: At this point, what forum can we ensure that CMS continues to be supported? Community support?

Jeremy: Kirk jClarity and Ramki Twitter have also expressed interest in seeing CMS continue.
Volker: SAP will support it for many more years. We support all Java releases for a very long time and use the same hotspot for all of them. So for us it is important that at least the CMS sources are still in the latest OpenJDK versions. For future Java releases, our CMS support depends on customer request.

Jeremy: Is it accurate to say that Oracle will continue to support it for many years?

Jon: we will still support it in previous releases but may not in 10. We will provide complete support on 9 and maybe 10 if we decide. We are just deciding to deprecate CMS. We think we will remove support in 10, but it may be 11. Support for CMS in earlier releases will be the same as it is for any other feature.
Volker SAP had concerns that we may need an agreement that licensees are allowed to get support.
Jeremy: This looks like something that can be worked out between SAP and Oracle.
Jeremy: so there is a way to get support for CMS from Oracle for older versions, but the community can enhance CMS
Volker: We only have some concerns that as Java licensees we (and others in the same situation) may get some CMS fixes from Oracle for older Java releases (8,9) which we can not easily contribute to the latest OpenJDK version (e.g. 10) if Oracle is not supporting CMS there any more. We should find a way how this would be possible.

Jeremy: What form might this take (community support)? Since this is not hosted by Oracle, what are the issues around testing, qualification? Would it be available to community support? What are the alternatives?

Jeremy: Volker said that he wants this on the mainline branch. Is this feasible to do this if we are using a different qual process across different platforms? Can the test tools be made available to the community? We need access to continuous integration, build logs, testing results, JTREG tests, … It should not be a problem to share this.

Jeremy: Volker SAP - you have collaborated with Sasha on the PowerPC little endian port. In terms of the workflow, taking patches and testing them, how did this work?
Volkner: The workflow for PowerPC all worked very well. Everyone can upload code to the code review servers. We can push PowerPC changes - there is no gatekeeper. Any OpenJDK committer can do this. If we can refactor CMS into its own directory, this should be OK

Jeremy: To what extent is there CMS code outside of the CMS directories?

Jungwoo: In JDK8, a lot of code is in the shared space. I would need to check JDK 9.
Jeremy: how much work will it be to clean that up so Oracle can be hands off.

Jon: Some progress has been made but there is a lot of hard work ahead.

Jeremy: Is it true that the intention is that Oracle can direct some of the work, but someone else has to do the work?

Jon: We need to share the work. We are open to collaboration with other groups
Jeremy: How much engineering work will this be?
Jon: I don’t know now.

Andrew: RedHat is writing another collector called Shenandoah

Andrew: We are looking at refactoring the GC for the runtime and JIT with a clean callback layer. So we are interested in simple, clean interfaces. We made progress on this, but we are not done. We found all of the places in JIT. We still need to work on runtime.

Jeremy: A follow-up conversation can be had about what needs to be done, what has been done, and reconciling that. This seems like a reasonable next step.
Volker: Question for Jon. After JDK 10 opens, are you going to work on the CMS refactoring right away?

Jon: we have not decided when to end support of CMS. We have not decided it is a doable thing. Our hope is that we will remove it but there may be roadblocks. We should starting work on this when JDK 10 is open for commits.

Kirk: Where do we want to go? What kind of world do we want to have?

Kirk: We want as near as we can have to plug-and-play GC world. This may be slightly idealistic. The major blocker to this is performance
Jeremy: from a code health perspective, we want well-defined interfaces

Kirk: Needing to build CMS from scratch will be a barrier to using it

Jeremy: we want results to be easily accessible by the community - to contribute and consume it.
Volker SAP: We need to talk to customers and find out how much they want it. The OpenJDK always has this available as sources. We want the build included in every JDK distro.

4. Develop action items and roadmap for next meeting (possibly in August).

Jeremy: Who wants to participate actively?

Jeremy: In the past, Volker, Twitter, Google, Kirk said that they want to participate. Kirk can also provide tooling for free to compare and contrast patches.

Jeremy: We would have contributed more but the process is very frustrating. It is a very high friction process. This would be a good thing to fix, but not the first thing to address.

Jon: Development of new features would be done in the CMS tree and it would be less painful.
Volker: We cannot solve the OpenJDK project infrastructure issues here. The infrastructure must improve, but we can’t solve this with CMS.
Jeremy: We can have external continuous integration, testing, … and that would be fine for CMS

Jeremy: We need a subset of engineers to figure out how to refactor this. Then we can figure out the form that this project will take.

Jeremy: The next step is to include those who like GC hacking to sit down and discuss the form the refactoring should take, how much work it is, the tasks, how long it will take.
Jeremy: We need to figure out if we need to have a GC interface. Oracle needs to commit some resources.
Jeremy: There is a vector - at one end is where we are, in the middle is what we need to do to support CMS, at the other end is provide beautiful interfaces. Maybe the short/medium term goal is to get CMS and then later figure out better interfaces.
Andrew: From the experience with Shenandoah, we would rather get the job done correctly.
Jeremy: Perhaps these are different projects we can pursue in parallel. Azul and others might want this, as well.
Jon: CMS separation is my immediate goal. It will be useful to review what was done by Shenandoah in terms of how it can be used to separate the CMS code.
Jeremy: We need to look at what has been done for Shenandoah. Who wants to be involved? Almost all of the folks on this call. We need to understand what has been done with Shenandoah about refactoring in a follow-up meeting. When can we do this?
Andrew RedHat: I need to talk to the Shenandoah engineers and ask them where they are in refactoring the interfaces. We need to know if they think that things are in a good state and ready for others to take a look.

Jeremy: We need to reconvene on this topic. We can have a meeting out of band at JVMLS, and have another meeting in a month

Volker: If we can build up the social process, it would help make this go better.

Jeremy: that would help greatly. This meeting cannot be the end of it. We need regular meetings for those working on this. We need to create a community around this.
Kirk: I thought we were going to have a F2F at JavaOne.
Jeremy: We should have a follow-up meeting before that. We can have an informal discussion at the JVM conference.

Jeremy: the homework for next meeting is for Andrew to talk to the Shenandoah team. The deliverable: status update on Shenandoah.

Volker: For the meet up, we want to invite the Shenandoah engineers

Jeremy: We need a proposal. How do we want to start putting together a proposal?

Jeremy: We need requirements for what you want and don’t want in the tree. Figure out what the shape of it should be. Functionality-by-functionality - not line-by-line. What do we need to get rid of? Provide a list of things.
Jeremy: Jon can you put together an early draft proposal on the separation? How much will be static? How many build time vs. runtime flags? Classes of changes not line-by-line.
Jon: we will try to have something by the next meeting.
Jeremy: If you want help, we can help. Post your request for help to the list. We will monitor the list. We will see what we can do to prioritize this.

Jeremy - schedule the next meeting
Jeremy: what channel will we use to communicate? We can use the gc mailing list for now until we start needing to do code reviews etc.