advice on tuning tomcat_opts on plosone, mulgara servers

Replying to russ:

plosone started going up and down according to nagios around 4:20 AM - when the rss feeds are generated.
when i finally got a chance to look, i saw that plosone was thrashing hard.
i'm bad and hadn't installed systat yet, so i'm not sure if mulgara was also thrashing, but all three had to be force quit with -f.

vmstat, iostat, xosview, etc will all give you info on whether things are swapping like mad or not.

looking at /etc/sysconfig/plosone, it looks like the memory being given to tomcat is rather ambitious!

[snip]

the MEM variable is being set to "total memory in the system - 128MB". 128MB is really not enough overhead!
if you want to dynamically set the memory usage, you should start with column 4 (the free memory) and then subtract some about (more than 128MB!)

Agreed - it should probably be subtracted from the free mem. OTOH, what else is running on the box that needs any noticeable amount of memory?

i will hard code mine to 3GB.

Sure.

would it make sense to limit the thread stack size as well with -Xss32m or something?

That would increase the stack sizes... On linux the sizes are taken from 'ulimit -s' - that currently seems to be 10M on prodution.

is there an easy way to figure out memory usage per thread so i can tune?

Since threads by definition share the same memory space there's is not such thing as per-thread heap memory usage. As to the per-thread stacks usage, I don't see any way - neither jmx nor ps gives that info AFAIK.

yes, i used vmstat on plosone servers to confirm the swap situation.

nothing else is running on the box - there was about 350M in use with plosone stopped out of 4GB total.

the internet seems to agree that there's no way to look at memory usage per thread - is there anything else we should keep in mind when tuning tomcat java_opts for mulgara and plosone? i discovered that mulgara Xmx was set very low, and upped it quite a bit...

Replying to russ:

the internet seems to agree that there's no way to look at memory usage per thread - is there anything else we should keep in mind when tuning tomcat java_opts for mulgara and plosone? i discovered that mulgara Xmx was set very low, and upped it quite a bit...

Please don't - it is low on purpose (and much for the same reason that we said to disable swap entirely on the mulgara machine). The vast bulk of the memory usage on mulgara is due to the mmap'd db files; because they're mmap'd it's not part of the heap. The heap usage is only a few 10 MB's to maybe a bit over 100MB. If you up the heap usage all you do is potentially reduce the amount of space available to the page-cache, i.e. reducing the amount of the mmap'd files being cached in memory. So unless you have really good numbers to indicate otherwise (e.g. from monitoring the heap usage via jmx or actual out-of-memory errors in the logs) I recommend leaving the assigned heap value.

okay, i'll remove /etc/sysconfig entirely.

i still need some advice on tuning tomcat and mod_jk on the plosone servers.

when the topaz stack goes down, it's taking out all of our production web sites because all of the apache processes on the webheads get stuck waiting for their 10 minute tomcat timeout.

i need to either reduce the mod_jk timeout to something more reasonable, or reduce the timeout between plosone and mulgara so that plosone gets the hint sooner that it's not gonna happen.

any advice?

err, /etc/sysconfig/mulgara that is. i'll leave the rest!