FULL PRODUCT VERSION :
  Bug reproducable with:
java version " 1.7.0_13 "
Java(TM) SE Runtime Environment (build 1.7.0_13-b20)
Java HotSpot(TM) 64-Bit Server VM (build 23.7-b01, mixed mode)

Works fine with:
java version " 1.6.0_35 "
Java(TM) SE Runtime Environment (build 1.6.0_35-b10)
Java HotSpot(TM) 64-Bit Server VM (build 20.10-b01, mixed mode)

ADDITIONAL OS VERSION INFORMATION :
Microsoft Windows [Version 6.1.7601]

A DESCRIPTION OF THE PROBLEM :
I am building my open-source project with maven and updated to java7. Since that site generation fails with an out of memory error:
[ERROR] javadoc: error - java.lang.OutOfMemoryError: Please increase memory.
[ERROR] For example, on the JDK Classic or HotSpot VMs, add the option -J-Xmx
[ERROR] such as -J-Xmx32m.
[ERROR]
[ERROR] Command line was: " C:\Program Files\Java\jdk1.7.0_13\jre\..\bin\javadoc.
exe " -J-Xmx1800m @options @packages

I have increased the maximum heap size to 4GB an still I get an out of memory.

  From what I analyzed javadoc is building the javadoc for all classes but then fails at the end when building indexes or overview. I observed the process with VisualVM and can see the quick and steep increase of the heap at the end shortly before the out of memory occurs. I took a heap dump and could see more that 1 million char[] have been allocated. All this gives me the impression that there is a brute infinity loop or recursion allocating memory in the case produced by my project.

Building the same with java6 works fine with no such problem.

I am not the only one with this problem - I found a german posting with the same issue where someone even tried with 8GB max heap without luck:
http://www.java-forum.org/allgemeine-java-themen/127008-maven-javadoc-plugin-javadoc-speicherverbrauch.html

FYI: I you really want to improve the quality of Java/JDK, you should have a look at this OSS project as it reveals even more bugs in JDK. It makes intensive use of generics and can not be build with java6 due to errors in javac. There is even still a bug in generics processing of javac with jdk1.7
that I worked around by down- and upcasting of
https://github.com/m-m-m/mmm/blob/master/mmm-data/mmm-data-base/src/main/java/net/sf/mmm/data/base/reflection/AbstractDataClassLoader.java
See comment " the following code is due to a generics bug in javac that does not occur in eclipse compiler " .
I would file a separate issue for this but I first want to see that you care - as previous issues I submitted have been ignored for years now and are still reproduceable in java.

REGRESSION. Last worked in version 6u31

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
1. check out https://github.com/m-m-m/mmm.git
2. set JAVA_HOME to a jdk1.7.x
3. go to mmm-setup and call
mvn install
4. go to the toplevel checked out directory and call
mvn site

EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
JavaDoc gets build properly.
ACTUAL -
[ERROR] javadoc: error - java.lang.OutOfMemoryError: Please increase memory.
[ERROR] For example, on the JDK Classic or HotSpot VMs, add the option -J-Xmx
[ERROR] such as -J-Xmx32m.
[ERROR]
[ERROR] Command line was: " C:\Program Files\Java\jdk1.7.0_13\jre\..\bin\javadoc.
exe " -J-Xmx1800m @options @packages


ERROR MESSAGES/STACK TRACES THAT OCCUR :
[ERROR] javadoc: error - java.lang.OutOfMemoryError: Please increase memory.
[ERROR] For example, on the JDK Classic or HotSpot VMs, add the option -J-Xmx
[ERROR] such as -J-Xmx32m.
[ERROR]
[ERROR] Command line was: " C:\Program Files\Java\jdk1.7.0_13\jre\..\bin\javadoc.
exe " -J-Xmx1800m @options @packages


REPRODUCIBILITY :
This bug can be reproduced always.

---------- BEGIN SOURCE ----------
Sorry, but I am unable to isolate the problem to a small project. Otherwise it would occur for everybody building javadocs.
---------- END SOURCE ----------

CUSTOMER SUBMITTED WORKAROUND :
Use java6 instead of java7.