Every time I come across a ulimit issue, It takes 4 hours to figure out. So I've decided from this point forward I will document each one.
Simple request came in…
A DBA was trying to do an Oracle Upgrade on an HPUX box, and ran into an issue.
The install was failing, and Oracle suggested that he have the sysadmins raise the maxdsiz ulimit.
So we got the call, investigated it, and hit this… like we always do.
mybox# ulimit -d 1048576
sh: ulimit: The specified value exceeds the user's allowable limit.
So, this being HPUX, we opened up SAM and took a look.
SAM reported the maxdsiz as 385875968… well above the limit 1048576 I was trying to set.
Then after a few hours, I came to the realization. SAM, that big asshole reports in bytes, whereas ulimit reports in kilobytes. Big difference. So the kernel limit in SAM when converted to Kilobytes was 376832. That's well below the 1048576 I was trying to set it to, and thus, the reason for the error.
Next, we scheduled a time to rebuild the kernel with the new settings.
When it was time, we opened up SAM, and to my dismay, SAM wanted the kernel paremeter in HEX. As if I wasn't already pissed off at it. Anyway. I converted the byte count to HEX, which ended up being 0×40000000. After that, I rebuilt the kernel and ran into another sumbling block, it wouldn't build.
I was able to trace this to a disk space issue. /stage was filled up. I found an old copy of the kernel in there that I moved off to another partition, and then finally, I was able to build the kernel using SAM with the new parameters, reboot the machine, and everyone was happy.
Another issue I encountered after the reboot is that sysdef reports the information differently than kmtune. The main difference being kmtune reports it correctly and sysdef doesn't. I didn't concern myself too much with why because sysdef is apparently depreciated anyway.
And theres the HPUX Kernel Tunable Issue Story.