CXX on X86 : LLVM ERROR: out of memory

[joukj]

Hi I get the following:

Code: Select all

rumba-jj) cxx/vers
clang version 10.0.1 (git@bitbucket.org:vms_software/llvm-10.0.1.git 1944c289616
bb7073af92c05d3ba6e7fd5d7ed0e)
Build date: 09-01-2023
Target: x86_64-OpenVMS
Thread model: posix
InstalledDir: RUMBA$DKA100:[SYS0.SYSCOMMON.][SYSEXE]


rumba-jj)cxx /stan=gnu14/define=("DIP_CONFIG_HAS_JPEG=1",	"DIP_CONFIG_HAS_ICS=1","
DIP_CONFIG_HAS_TIFF=1","DIP_CONFIG_HAS_FFTW=1"	,"DIP_CONFIG_HAS_FREEGLUT=1","DI
P_CONFIG_HAS_FREETYPE=1",	DIP_VERSION_STRING="3.0-vms")/l_doub=64	/include
=([],[--.include])	/warn=disable=("reserved-user-defined-literal") matrix.c
pp
LLVM ERROR: out of memory
Stack dump:
0.	Program arguments: RUMBA$DKA100:[SYS0.SYSCOMMON.][SYSEXE]CXX$COMPILER.EX
E;1 --std=gnu++98 --std=gnu++14 -DDIP_CONFIG_HAS_JPEG=1 -DDIP_CONFIG_HAS_ICS=1 -
DDIP_CONFIG_HAS_TIFF=1 -DDIP_CONFIG_HAS_FFTW=1 -DDIP_CONFIG_HAS_FREEGLUT=1 -DDIP
_CONFIG_HAS_FREETYPE=1 -DDIP_VERSION_STRING="3.0-vms" -I[] -I[--.INCLUDE] -mlong
-double-64 -Wno-reserved-user-defined-literal -Wno-nonportable-include-path -isy
stem/CXX$SYSTEM_INCLUDE -isystem/CXX$USER_INCLUDE -x c++ -D__RTTI $DSTM:[JOUKJ.p
ublic.diplib.diplib_2890.diplib.src.support]matrix.cpp;23 
1.	<eof> parser at end of file
2.	Code generation
3.	Running pass 'Function Pass Manager' on module '$DSTM:[JOUKJ.public.dipl
ib.diplib_2890.diplib.src.support]matrix.cpp;23'.
4.	Running pass 'X86 Assembly Printer' on function '@_ZN5Eigen8internal16bi
nary_evaluatorINS_13CwiseBinaryOpINS0_17scalar_product_opIddEEKNS_14CwiseNullary
OpINS0_18scalar_constant_opIdEEKNS_6MatrixIdLin1ELi1ELi0ELi2ELi1EEEEEKNS_5BlockI
KNSD_INSD_INS8_IdLi2ELi2ELi0ELi2ELi2EEELi2ELi1ELb1EEELin1ELi1ELb0EEELin1ELi1ELb0
EEEEENS0_10IndexBasedESL_ddEC2ERKSK_'
%SYSTEM-F-ACCVIO, access violation, reason mask=04, virtual address=000000000000
0011, PC=FFFF8301079FC77E, PS=0000001B

  Improperly handled condition, image exit forced.
    Signal arguments:   Number = 0000000000000005
                        Name   = 000000000000000C
                                 0000000000000004
                                 0000000000000011
                                 FFFF8301079FC77E
                                 000000000000001B
    Register dump:
    RAX = 0000000000000001  RDI = 000000007FF7BC20  RSI = 0000000000000001
    RDX = 0000000000000000  RCX = FFFFFFFFBD005FDE  R8  = 0000000000000000
    R9  = FFFFFFFF8049B900  RBX = 000000007A919F00  RBP = 000000007A90EF60
    R10 = FFFFFFFFFFFFFFD9  R11 = 0000000000000000  R12 = 000000007A90EEA8
    R13 = FFFFFFFF804B83F0  R14 = FFFFFFFF804B8558  R15 = 000000007A904602
    RIP = FFFF8301079FC77E  RSP = 000000007A90EEA0  SS  = 000000000000001B
%MMS-F-ABORT, For target matrix.obj, CLI returned abort status: %X1000000C.

What can cause this problem??? It seems that the problem is deep in the Eigen-package (see http://nchrem.tnw.tudelft.nl/openvms/so ... html#EIGEN )
I do not see here any "large" constant matrices....

[arne_v]

Have you tried increasing pgflquo?

[joukj]

They are already quite high:

Code: Select all

rumba-jj) aut
UAF> sh joukj

Username: JOUKJ                            Owner:  JOUK JANSEN
Account:  JOUKJ                            UIC:    [300,300] ([JOUKJ])
CLI:      DCL                              Tables: RUMBA$DKA100:[JOUKJ.COM]JJJ.E
XE
Default:  SYS$SYSDEVICE:[JOUKJ.COM]
LGICMD:   
Flags: 
Primary days:   Mon Tue Wed Thu Fri        
Secondary days:                     Sat Sun
No access restrictions
Expiration:            (none)    Pwdminimum: 15   Login Fails:     0
Pwdlifetime:           (none)    Pwdchange:  20-APR-2023 08:16 
Last Login: 12-SEP-2023 13:07 (interactive), 12-SEP-2023 08:04 (non-interactive)
Maxjobs:         0  Fillm:      5000  Bytlm:       2100000
Maxacctjobs:     0  Shrfillm:      0  Pbytlm:            0
Maxdetach:       0  BIOlm:     10000  JTquota:       60000
Prclm:          40  DIOlm:      1024  WSdef:      24000000
Prio:            4  ASTlm:      1200  WSquo:     100000000
Queprio:         4  TQElm:       400  WSextent:  524288000
CPU:        (none)  Enqlm:      4000  Pgflquo:  2147483632
Authorized Privileges: 
  ACNT         ALLSPOOL     ALTPRI       AUDIT        BUGCHK       BYPASS
  CMEXEC       CMKRNL       DIAGNOSE     DOWNGRADE    EXQUOTA      GROUP
  GRPNAM       GRPPRV       IMPERSONATE  IMPORT       LOG_IO       MOUNT
  NETMBX       OPER         PFNMAP       PHY_IO       PRMCEB       PRMGBL
  PRMMBX       PSWAPM       READALL      SECURITY     SETPRV       SHARE
  SHMEM        SYSGBL       SYSLCK       SYSNAM       SYSPRV       TMPMBX
  UPGRADE      VOLPRO       WORLD
Default Privileges: 
  NETMBX       TMPMBX
UAF> exit
%UAF-I-NOMODS, no modifications made to system authorization file
%UAF-I-RDBNOMODS, no modifications made to rights database
  13-SEP-2023 07:20:53
rumba-jj) sh mem
              System Memory Resources on 13-SEP-2023 07:20:57.66

Physical Memory Usage (pages):     Total        Free      In Use    Modified
  Main Memory (62.34GB)          8171282     7269020      900985        1277

Extended File Cache  (Time of last reset:  8-SEP-2023 09:22:12.37)
 Allocated (GBytes)              4.37    Maximum size (GBytes)            31.17
 Free (GBytes)                   0.00    Minimum size (GBytes)             0.00
 In use (GBytes)                 4.37    Percentage Read I/Os                84%
 Read hit rate                     98%   Write hit rate                       0%
 Read I/O count               3954962    Write I/O count                 721026
 Read hit count               3895287    Write hit count                      0
 Reads bypassing cache             41    Writes bypassing cache               1
 Files cached open                289    Files cached closed               9829
 Vols in Full XFC mode              0    Vols in VIOC Compatible mode         3
 Vols in No Caching mode            0    Vols in Perm. No Caching mode        0

Granularity Hint Regions (pages):   Total        Free      In Use    Released
  S0 Execlet data                    2048        1679         369           0
  S0 Executive data                 60416         200       60216           0
  S0 Executive RO data                512         310         202           0
  S0 Resident image code             3072        2989          83           0
  S0 Resident image data              512         512           0           0
  S0 Resident RO image data           512         512           0           0
  S2 Execlet code                    4096         489        3607           0
  S2 Execlet data                    4096        4096           0           0
  S2 Executive data                   512           0         512           0
  S2 Resident image code             4096        1512        2584           0
  S2 Resident image data              512         512           0           0

Slot Usage (slots):                Total        Free    Resident     Swapped
  Process Entry Slots                657         618          39           0

Dynamic Memory Usage:              Total        Free      In Use     Largest
  Nonpaged Dynamic Memory (MB)    912.00      903.45        8.54      902.43
  USB Addressable Memory  (KB)   1024.00     1022.87        1.12     1022.87
  Paged Dynamic Memory    (MB)     10.32        4.72        5.59        4.68
  Lock Manager Dyn Memory (MB)     47.52       37.79        9.72
  S2 Dynamic Memory Usage (MB)      3.96        3.80        0.15        3.80

Buffer Object Usage (pages):                  In Use        Peak
  32-bit System Space Windows (S0/S1)              1           1
  64-bit System Space Windows (S2)                 0           0
  Physical pages locked by buffer objects          1           0

Memory Reservations (pages):       Group    Reserved      In Use        Type
  Total (0 bytes reserved)                         0           0

Paging File Usage (8KB pages):                 Index        Free        Size
  DISK$X86SYS:[SYS0.SYSEXE]PAGEFILE.SYS;2                                       
                                                 254     1572856     1572856
  Total committed paging file usage:                                   16221

Of the physical pages in use, 314470 pages are permanently allocated to OpenVMS.

[arne_v]

They are already quite high:

Pgflquo: 2147483632

That is 1 TB.

It must be a bug.

[jreagan]

Runnig out of P1 heap is not fixable with more page file quota of 1TB. We are looking at LLVM memory usage right now from a COBOL bug report. I can't offer any additional suggestions at the moment, but a reproducer might help.

[arne_v]

Ah P1 space.

Could you move some of LLVM heap to P2?

Added in 8 minutes 58 seconds:
s/P1 heap/P0 heap/w

I presume.

[joukj]

64-bit and more memory is the future!!!!

Added in 1 minute 33 seconds:
I solved the problem by splitting the source code in (in this case 11) parts.

I will wait till VSI hassorted out the COBOL problem. Maybe that one also solves this one.

[jreagan]

Can we get access to that code so we can do a study of the memory usage patterns in the compiler?

[joukj]

Attached is a reproducer. Just type "MMS" and after a while you get the error.
The file matrix.cpp is extracted from the diplib package (see https://diplib.org/ )

[joukj]

With the latest CXX compiler I do not get the memory ,message anymore, but... now I get a "access violation and a traceback