CTW tests are frequently failing in my CI because there is not enough memory to run them.

Run this:

$ CONF=linux-x86_64-server-fastdebug make images run-test TEST=applications/ctw/modules TEST_JOBS=8

While this is running, see how many VMs have started:

$ ps -ef | grep CompileTheWorld | wc -l
8

$ ps -ef | grep AgentServer | wc -l
25

$ for P in `ps -ef | grep CompileThe | awk '{print $2;}'`; do grep VmRSS /proc/$P/status; done | awk '{ sum += $2 } END { print sum; }'
2014100

$ for P in `ps -ef | grep AgentServer | awk '{print $2;}'`; do grep VmRSS /proc/$P/status; done | awk '{ sum += $2 } END { print sum; }'
5051432

In other words, with 8 test jobs, the whole thing starts 25+8 = 33 JVMs to support the testing. Most of those are jtreg AgentServer, that take the bulk amount of RSS memory as well.

I have two questions here:
 a) Why there are AgentServer VMs at all?
 b) Why there are so many of them?

It is a separate question why jtreg does not shut some of the AgentServer VMs down.