Detached Process consuming Paging File Quota

Having difficulties when installing the system? Your system runs slowly and requires some tweaking? You can get help here.
Post Reply

Topic author
vmskostoff
Active Contributor
Posts: 39
Joined: Fri Jun 28, 2019 10:29 am
Reputation: 0
Location: Gary, Indiana
Status: Offline

Detached Process consuming Paging File Quota

Post by vmskostoff » Thu Feb 24, 2022 4:13 pm

I have a detached process on an Itanium, rx3600, OpenVMS 8.4 that starts with

PQL_DPGFLQUOTA 700000

Now observing:

Paging file quota: 32

Elapsed CPU time: 4 02:43:16.90
Connect time: 114 03:55:27.69

Not sure of what is contributing to the decline in the Paging File Quota from 700000 to 32?
Looking for some ideas of what look at.
Last edited by vmskostoff on Thu Feb 24, 2022 4:43 pm, edited 1 time in total.

User avatar

volkerhalle
Master
Posts: 196
Joined: Fri Aug 14, 2020 11:31 am
Reputation: 0
Status: Offline

Re: Detached Process consuming Paging File Quota

Post by volkerhalle » Fri Feb 25, 2022 2:49 am

Note that page file quota is a POOLED QUOTA, so it's shared between the main process and all possible sub-processes it may have created.

You can check the page file quota values of the process with SDA:

$ ANALYZE/SYS
SDA> SHOW PROC/ID=<id-of-detached-process>
SDA> READ SYSDEF
SDA> FORMAT JIB
...
FFFFFFFF.82A4D300 JIB$L_PGFLQUOTA 000009C4 <<< original page file quota when created
FFFFFFFF.82A4D304 JIB$L_PGFLCNT 00000647 <<< current remaining page file quota
...

Did the decrease of available page file quota happen over time ? If so, it very likely is related to some kind of memory leak, i.e. allocating but not freeing virtual process address space.

Volker.


Topic author
vmskostoff
Active Contributor
Posts: 39
Joined: Fri Jun 28, 2019 10:29 am
Reputation: 0
Location: Gary, Indiana
Status: Offline

Re: Detached Process consuming Paging File Quota

Post by vmskostoff » Fri Feb 25, 2022 11:06 am

Greetings Volker,

Thanks for your response.

It is funny, so to speak, we have 3 rx3600s in production, running the same EXE, on same OS OpenVMS 8.4
and one node decides to go rogue.

$ ANALYZE/SYS
SDA> SHOW PROC/ID=<id-of-detached-process>
SDA> READ SYSDEF
SDA> FORMAT JIB

FFFFFFFF.8A1CFB00 JIB$L_PGFLQUOTA 0000AAE6 <<< original page file quota = 43750 <<<
FFFFFFFF.8A1CFB04 JIB$L_PGFLCNT 00000002

When I queried SYSGEN, I find the following:
SYSGEN> SHOW PQL_DPGFLQUO
Parameter Name Current Default Min. Max. Unit Dynamic
-------------- ------- ------- ------- ------- ---- -------
PQL_DPGFLQUOTA 700000 700000 512000 -1 Pagelets D
internal value 43750 43750 32000 -1 Pages D

So why not the higher value of 700000?

2. I cannot say whether this has decreased over time. The issue just came up recently.
Is there an automated way to track?

3. Is there a way to monitor processes for these types of events?
Last edited by vmskostoff on Fri Feb 25, 2022 11:09 am, edited 2 times in total.

User avatar

volkerhalle
Master
Posts: 196
Joined: Fri Aug 14, 2020 11:31 am
Reputation: 0
Status: Offline

Re: Detached Process consuming Paging File Quota

Post by volkerhalle » Fri Feb 25, 2022 11:52 am

Note the 'unit' shown in the second to last column of the SYSGEN display. The internal value is measured in Pages (16*512=8192 bytes) , whereas the value shown in DCL is in pagelets (512 bytes) - for historical reasons, as the page size on OpenVMS VAX was just that: 512 bytes.

2. Did you observe any problems on that process with only 32 pagelets of pagefile quota left ? Or has someone just spotted this low value ?

3. consider to write some DCL procedure to monitor this process using F$GETJPI("pid-of-process","PAGFILCNT") on a regular basis. If you see this value decline to near-zero, stop and restart the process, before it will be causing problems (process hangs). Instead of just stopping it with STOP/ID, consider using SET PROC/ID=xxx/DUMP=NOW. This will create a process dump and give you a chance to diagnose a possible memory leak. Look at the high virtual address space for repeating data patterns.

Volker.
Last edited by volkerhalle on Fri Feb 25, 2022 11:54 am, edited 1 time in total.


Topic author
vmskostoff
Active Contributor
Posts: 39
Joined: Fri Jun 28, 2019 10:29 am
Reputation: 0
Location: Gary, Indiana
Status: Offline

Re: Detached Process consuming Paging File Quota

Post by vmskostoff » Fri Feb 25, 2022 6:05 pm

#2. Problem observed was: insufficient virtual memory.
This was causing the process to NOT be able to write/read to an RMS-based file.

User avatar

volkerhalle
Master
Posts: 196
Joined: Fri Aug 14, 2020 11:31 am
Reputation: 0
Status: Offline

Re: Detached Process consuming Paging File Quota

Post by volkerhalle » Sat Feb 26, 2022 3:40 am

#2 so the process crashed after an INSVIRMEM error and had to be restarted - correct ?

If true: run or start that process with /DUMP. This will cause a process dump file to be created, if the process crashes due to some unhandled error. You could than look at that dump file with ANALYZE/SYSTEM or ANALYZE/PROCESS to try to determine the underlying reason.

If true: when running with 32 pagelets of remaining pagefile quota, this problem might happen again very soon. You can SET PROC/DUMP/PID=<id-of-running process>, so a process dump will be created, if that process crashes.

Consider to collect the following information from SDA about the problematic process and one of the other ones, which do not exhibit this problem (on the other systems):

SDA> SET PROC/IND=<pid-of-process>
SDA> SET LOG filename ! causes output to be logged
SDA> SHOW PROC/IMA
SDA> SHOW PROC/PAGE/P0 ! assuming the memory leak will be in P0 space

Then look at the high end of P0 virtual address space mapped and consider to examine the contents of the last page:
...
Mapped Address PTE Address PTE Type Read Writ MLOA GH PgTyp Loc Bak RefCnt WSLX
----------------- ----------------- ----------------- ----- ---- ---- ---- - ------- ------ ----------------- ---- --------
00000000.00372000 FFFFFEFC.00000DC8 0000CBF4.0016FF09 VALID KESU KESU M-U- 0 PROCESS ACTIVE FF000000.00000000 0001 000001FA

SDA> EXA 00372000;2000

The idea is as follows: if there is a memory leak (allocating and not freeing proces virtual memory over time), the virtual address space will grow and be populated with those allocated-but-not-freed packets. You might spot data patterns in that area, which could lead to further conclusions.

Volker.


Topic author
vmskostoff
Active Contributor
Posts: 39
Joined: Fri Jun 28, 2019 10:29 am
Reputation: 0
Location: Gary, Indiana
Status: Offline

Re: Detached Process consuming Paging File Quota

Post by vmskostoff » Mon Feb 28, 2022 12:59 pm

#2 Process did not crash and is still running:

Elapsed CPU time: 4 06:03:57.78
Connect time: 118 00:23:03.17

The only indication of problem was found incidentally when reviewing log files.
So no performance issues known at this time other than the two issues:

1. INSVIRMEM
2. Unable to read/write RMS-based file.

If FALSE: Since no branch provided in your reply, any additional thoughts on this, "IF FALSE" path?

At least collect this as you discussed and investigate and compare between nodes:

SDA> SET PROC/IND=<pid-of-process>
SDA> SET LOG filename ! causes output to be logged
SDA> SHOW PROC/IMA
SDA> SHOW PROC/PAGE/P0 ! assuming the memory leak will be in P0 space

Uptime on Node C is: 200 22:01:25
Uptime on the other nodes is

Node A: Uptime 1362 20:56:39
Node B: Uptime 1362 20:58:13

User avatar

volkerhalle
Master
Posts: 196
Joined: Fri Aug 14, 2020 11:31 am
Reputation: 0
Status: Offline

Re: Detached Process consuming Paging File Quota

Post by volkerhalle » Mon Feb 28, 2022 2:05 pm

So the process is still running albeit the INSVIRMEM errors reported and the inability to read/write RMS based file - are there any error/status messages associated with those RMS problems ?

So this seems to indicate that the INSVIRMEM errors must be handled by some condition handler and maybe retried and then succeeded ? Or whatever code should have been executed and hit the INSVIRMEM did not execute ? With no visible effects ?

What's the initial and remaining page file quota values on nodes A and B ?

When I see those uptimes, I can understand, that you're reluctant to stop and restart those processes.

Volker.
Last edited by volkerhalle on Mon Feb 28, 2022 2:07 pm, edited 1 time in total.


Topic author
vmskostoff
Active Contributor
Posts: 39
Joined: Fri Jun 28, 2019 10:29 am
Reputation: 0
Location: Gary, Indiana
Status: Offline

Re: Detached Process consuming Paging File Quota

Post by vmskostoff » Mon Feb 28, 2022 6:43 pm

QN: So the process is still running ....
Answer: Yes process is still running. with the INSVIRMEM error reported and inability to read/write RMS-based file.

QN: INSVIRMEM errors must be handled by condition handler? Yes that is my understanding at this time. There are no
visible side effects.

QN: Initial and remaining page file quota on:
Answer:
Node A:
Initial:
SYSGEN> SHOW PQL_DPGFLQUO
Parameter Name Current Default Min. Max. Unit Dynamic
-------------- ------- ------- ------- ------- ---- -------
PQL_DPGFLQUOTA 700000 700000 512000 -1 Pagelets D
internal value 43750 43750 32000 -1 Pages D

Remaining:
Paging file quota: 668864

Node B:
Initial
SYSGEN> SHOW PQL_DPGFLQUO
Parameter Name Current Default Min. Max. Unit Dynamic
-------------- ------- ------- ------- ------- ---- -------
PQL_DPGFLQUOTA 700000 700000 512000 -1 Pagelets D
internal value 43750 43750 32000 -1 Pages D

Remaining:
Paging file quota: 668864

Node C:
Initial:
SYSGEN> SHOW PQL_DPGFLQUO
Parameter Name Current Default Min. Max. Unit Dynamic
-------------- ------- ------- ------- ------- ---- -------
PQL_DPGFLQUOTA 700000 700000 512000 -1 Pagelets D
internal value 43750 43750 32000 -1 Pages D

Remaining:
Paging file quota: 32

Node Uptimes:
Node uptimes and process restarts are independent.
Due to the actual physical process, there are windows when OpenVMS processes can be restarted.
Some OpenVMS processes can be restarted on the fly and some cannot.

User avatar

volkerhalle
Master
Posts: 196
Joined: Fri Aug 14, 2020 11:31 am
Reputation: 0
Status: Offline

Re: Detached Process consuming Paging File Quota

Post by volkerhalle » Tue Mar 01, 2022 2:41 am

Thanks for the summary information.

So it will really be interesting to see the differences from the SDA process output of those systems. Something must be different...

Volker.

Post Reply