OpenVMS V8.4
3 identical RX2600 – 48GB mem - with only internal disks in RAID
Cluster consisting of:
- 2 Production servers:
• Node1: (active) running the application and Oracle database, shared Queue management
• Node2: (standby) running some housekeeping tasks, third-party Backup via network, Defrag, sharing some batch queues (and sharing all print Q’s) with Node1
• The production application Shadow disks have shadowset members mounted on both nodes; system disks are local on each node with common part on a separate shadow disk mounted on both nodes, for amongst others, the shared Queue manager
- 1 Development server – Node3:
• Quorum/voting member for the cluster
• The production applications disks have a third shadowset member on this node, but these are not mounted
• Local non-shadow disks for system & application development (physical disks are double the size on this member, this being the only physical HW difference)
No changes done on the systems the past couple of months, no major changes (ex. SYSGEN) the previous years.
Problem:
Since last Friday morning, the Lock Manager Dynamic Memory has started to slowly increase on Node1 and has continued to increase ever since, from +/- 73MB in use (87MB size) the weeks before, to 84MB (91MB) on Friday morning (triggering an alert), and creeping up to 88 MB (95MB) about 48hrs later.
Investigation so far (I have limited experience on system internals and performance analysis knowledge):
- SDA ‘Show Lock’ analysis on Node1:
• 114K entries in total
• 105K from ‘Resource name’ CACHE$cmDISK4
• 1K from next most occurring resource CACHE$cmCOMMON (disk containing, amongst others, the Q mgr.)
- DISK4 is the disk with the ‘miscellaneous’ files containing temporary work files, .log files, etc. :
• 230+K files in total
• 20K new files created per day
• approx. 200-400 open files on DISK4 on Node1 (< 10 on Node2)
• daily cleanup of older file versions at 4PM
• disk less than 50% full
• monthly defragged, last run was 1 week ago
- details of Show Mem included at the end.
Monitoring the Lock Mgr memory during the day, I discovered a significant drop of 4 – 5 MB ‘in use’ after the daily file-cleanup housekeeping job had run (purging approx. 20K old file versions). But this drop in size, only compensates partially the daily growth. The files that are purged are mostly files that have not been in use/opened for several days (except by the daily VMS full backup with /IGN=INTER, that runs every other day on Node1 or Node2).
SDA ‘Show Lock’ analysis on Node1 after daily cleanup:
• 104K entries in total (down from 114K)
• 94K from ‘Resource name’ CACHE$cmDISK4 (down from 105K)
Questions:
1. Is it possible to calculate/estimate, how much longer the Lock Mgr. memory can continue to grow?
2. Can the huge amount of Lock entries for Disk4 be explained somehow, giving the low number of files open on Disk4 at any given time? How can purging mostly old/unused files decrease the number of Lock table entries?
3. Most importantly, any ideas on how to fix this sudden growth in Lock entries??
______________________________________________________________
Code: Select all
System Memory Resources on 5-DEC-2021 20:50:55.36
Nonpaged Dynamic Memory (Lists + Variable)
Current Size (MB) 75.53 Current Size (Pagelets) 154704
Initial Size (MB) 57.66 Initial Size (Pagelets) 118096
Maximum Size (MB) 261.00 Maximum Size (Pagelets) 534528
Free Space (MB) 19.14 Space in Use (MB) 56.39
Largest Var Block (KB) 270.50 Smallest Var Block (bytes) 192
Number of Free Blocks 41296 Free Blocks LEQU 64 bytes 0
Free Blocks on Lookasides 41291 Lookaside Space (MB) 18.79
Bus Addressable Memory (Lists + Variable)
Current Size (MB) 2.46 Current Size (Pagelets) 5056
Initial Size (MB) 2.46 Initial Size (Pagelets) 5056
Free Space (MB) 2.26 Space in Use (MB) 0.19
Largest Var Block (MB) 2.26 Smallest Var Block (bytes) 64
Number of Free Blocks 14 Free Blocks LEQU 64 bytes 1
Free Blocks on Lookasides 5 Lookaside Space (bytes) 384
Paged Dynamic Memory (Lists + Variable)
Current Size (MB) 12.33 Current Size (Pagelets) 25264
Free Space (MB) 6.24 Space in Use (MB) 6.09
Largest Var Block (MB) 4.84 Smallest Var Block (bytes) 16
Number of Free Blocks 13737 Free Blocks LEQU 64 bytes 22
Free Blocks on Lookasides 13731 Lookaside Space (MB) 1.39
Lock Manager Dynamic Memory
Current Size (MB) 95.12 Current Size (Pages) 12176
Free Space (MB) 9.90 Hits 1308210732
Space in Use (MB) 85.21 Misses 10452
Number of Empty Pages 685 Expansions 20568
Number of Free Packets 39119 Packet Size (bytes) 0
System Memory Resources on 5-DEC-2021 20:49:59.90
Physical Memory Usage (pages): Total Free In Use Modified
Main Memory (47.99GB) 6291024 616526 5649442 25056
Extended File Cache (Time of last reset: 16-MAR-2017 12:05:28.56)
Allocated (GBytes) 8.56 Maximum size (GBytes) 23.
Free (GBytes) 1.21 Minimum size (GBytes) 0.
In use (GBytes) 7.34 Percentage Read I/Os
Read hit rate 97% Write hit rate
Read I/O count 43642926446 Write I/O count 28671504
Read hit count 42482109518 Write hit count
Reads bypassing cache 88175862 Writes bypassing cache 20298857
Files cached open 663 Files cached closed 710
Vols in Full XFC mode 0 Vols in VIOC Compatible mode
Vols in No Caching mode 0 Vols in Perm. No Caching mode
Granularity Hint Regions (pages): Total Free In Use Released
Execlet code region 8192 4427 3765 0
Execlet data region 2048 1123 925 0
S0S1 Executive data region 7721 0 7721 0
S0S1 Resident image code region 65536 61810 3726 0
S0S1 Resident image data region 2048 1888 160 0
Slot Usage (slots): Total Free Resident Swapped
Process Entry Slots 1017 782 235 0
Balance Set Slots 1015 782 233 0
Dynamic Memory Usage: Total Free In Use Largest
Nonpaged Dynamic Memory (MB) 75.53 19.10 56.43 0.26
Bus Addressable Memory (MB) 2.46 2.26 0.19 2.26
Paged Dynamic Memory (MB) 12.33 6.19 6.14 4.84
Lock Manager Dyn Memory (MB) 95.12 9.86 85.25
Buffer Object Usage (pages): In Use Peak
32-bit System Space Windows (S0/S1) 6 19
64-bit System Space Windows (S2) 0 0
Physical pages locked by buffer objects 6 19
Memory Reservations (pages): Group Reserved In Use Type
ORA_SGA SYSGBL 3200 3200 Page Table
ORA_SGA SYSGBL 3145728 3145728 Allocated
ORA_SGA SYSGBL 131072 130803 Allocated
Total (25.02 GBytes reserved) 3280000 3279731
Write Bitmap (WBM) Memory Summary
Local bitmap count: 4 Local bitmap memory usage (KB) 1016.00
Master bitmap count: 5 Master bitmap memory usage (MB) 1.19[/size]