Hi,
I think this could be because of the kernel, as this all is now taken care by the kernel itself. May be an update to current level may solve the problem.
Also, what the maximum heap size per process ?
Can you check this "memlimits -v off".
We cannot run this DIALOG.
Refer: 789477 - Large extended memory on AIX (64-bit) as of Kernel 6.20
May be some adjustment might work.
Regards,
Divyanshu