Hi,
Please share kernel details and if possible try to upgrade the kernel to the latest available as already suggested by Reagan Benjamin and retry the phase .Meanwhile follow SAP Notes 1563583 - SYSTEM_NO_TASK_STORAGE, DBIF_RSQL_NO_MEMORY, EXSORT_NOT_ENOUGH_MEMORY
1147109 - Heap memory shortage
323816 - AIX: User limit settings
Hope this will help you.
Regards,
Gaurav