The customer has several Amdocs environments installed on the detailed machine, each environment has several application servers. Approximately once a day one of the servers that belongs to one of their environments (randomly) starts to consume abnormal amounts of memory
 
Note: Each of the servers' memory consumption was limited using JVM –Xmx and –DmaxHeapMemory parameters to 1gb – 2gb.

Detailed description:

1. At first it was suspected to be an application memory leak but this option was ruled out once logs are cross referenced to the script log.

logs shows that no more than ~30% of the total memory that was allocated to the server was used and no exception occurred

while at the same time, the script log shows that the server used 4gb of memory which is about 4 times the memory allocated for this process. (using a bit more memory then what was allocated is normal but not using 4/5 times the amount of memory allocated).

2. The class histogram shows that no more than 917mb was used by objects in memory while at the same time the pmap script log shows that 4gb of memory was used.