3 identical RX2600 – 48GB mem - with only internal disks in RAID
Cluster consisting of:
- 2 Production servers:
• Node1: (active) running the application and Oracle database, shared Queue management
• Node2: (standby) running some housekeeping tasks, third-party Backup via network, Defrag, sharing some batch queues (and sharing all print Q’s) with Node1
• The production application Shadow disks have shadowset members mounted on both nodes; system disks are local on each node with common part on a separate shadow disk mounted on both nodes, for amongst others, the shared Queue manager
- 1 Development server – Node3:
• Quorum/voting member for the cluster
• The production applications disks have a third shadowset member on this node, but these are not mounted
• Local non-shadow disks for system & application development (physical disks are double the size on this member, this being the only physical HW difference)
No changes done on the systems the past couple of months, no major changes (ex. SYSGEN) the previous years.
Since last Friday morning, the Lock Manager Dynamic Memory has started to slowly increase on Node1 and has continued to increase ever since, from +/- 73MB in use (87MB size) the weeks before, to 84MB (91MB) on Friday morning (triggering an alert), and creeping up to 88 MB (95MB) about 48hrs later.
Investigation so far (I have limited experience on system internals and performance analysis knowledge):
- SDA ‘Show Lock’ analysis on Node1:
• 114K entries in total
• 105K from ‘Resource name’ CACHE$cmDISK4
• 1K from next most occurring resource CACHE$cmCOMMON (disk containing, amongst others, the Q mgr.)
- DISK4 is the disk with the ‘miscellaneous’ files containing temporary work files, .log files, etc. :
• 230+K files in total
• 20K new files created per day
• approx. 200-400 open files on DISK4 on Node1 (< 10 on Node2)
• daily cleanup of older file versions at 4PM
• disk less than 50% full
• monthly defragged, last run was 1 week ago
- details of Show Mem included at the end.
Monitoring the Lock Mgr memory during the day, I discovered a significant drop of 4 – 5 MB ‘in use’ after the daily file-cleanup housekeeping job had run (purging approx. 20K old file versions). But this drop in size, only compensates partially the daily growth. The files that are purged are mostly files that have not been in use/opened for several days (except by the daily VMS full backup with /IGN=INTER, that runs every other day on Node1 or Node2).
SDA ‘Show Lock’ analysis on Node1 after daily cleanup:
• 104K entries in total (down from 114K)
• 94K from ‘Resource name’ CACHE$cmDISK4 (down from 105K)
1. Is it possible to calculate/estimate, how much longer the Lock Mgr. memory can continue to grow?
2. Can the huge amount of Lock entries for Disk4 be explained somehow, giving the low number of files open on Disk4 at any given time? How can purging mostly old/unused files decrease the number of Lock table entries?
3. Most importantly, any ideas on how to fix this sudden growth in Lock entries??
Code: Select all
System Memory Resources on 5-DEC-2021 20:50:55.36 Nonpaged Dynamic Memory (Lists + Variable) Current Size (MB) 75.53 Current Size (Pagelets) 154704 Initial Size (MB) 57.66 Initial Size (Pagelets) 118096 Maximum Size (MB) 261.00 Maximum Size (Pagelets) 534528 Free Space (MB) 19.14 Space in Use (MB) 56.39 Largest Var Block (KB) 270.50 Smallest Var Block (bytes) 192 Number of Free Blocks 41296 Free Blocks LEQU 64 bytes 0 Free Blocks on Lookasides 41291 Lookaside Space (MB) 18.79 Bus Addressable Memory (Lists + Variable) Current Size (MB) 2.46 Current Size (Pagelets) 5056 Initial Size (MB) 2.46 Initial Size (Pagelets) 5056 Free Space (MB) 2.26 Space in Use (MB) 0.19 Largest Var Block (MB) 2.26 Smallest Var Block (bytes) 64 Number of Free Blocks 14 Free Blocks LEQU 64 bytes 1 Free Blocks on Lookasides 5 Lookaside Space (bytes) 384 Paged Dynamic Memory (Lists + Variable) Current Size (MB) 12.33 Current Size (Pagelets) 25264 Free Space (MB) 6.24 Space in Use (MB) 6.09 Largest Var Block (MB) 4.84 Smallest Var Block (bytes) 16 Number of Free Blocks 13737 Free Blocks LEQU 64 bytes 22 Free Blocks on Lookasides 13731 Lookaside Space (MB) 1.39 Lock Manager Dynamic Memory Current Size (MB) 95.12 Current Size (Pages) 12176 Free Space (MB) 9.90 Hits 1308210732 Space in Use (MB) 85.21 Misses 10452 Number of Empty Pages 685 Expansions 20568 Number of Free Packets 39119 Packet Size (bytes) 0 System Memory Resources on 5-DEC-2021 20:49:59.90 Physical Memory Usage (pages): Total Free In Use Modified Main Memory (47.99GB) 6291024 616526 5649442 25056 Extended File Cache (Time of last reset: 16-MAR-2017 12:05:28.56) Allocated (GBytes) 8.56 Maximum size (GBytes) 23. Free (GBytes) 1.21 Minimum size (GBytes) 0. In use (GBytes) 7.34 Percentage Read I/Os Read hit rate 97% Write hit rate Read I/O count 43642926446 Write I/O count 28671504 Read hit count 42482109518 Write hit count Reads bypassing cache 88175862 Writes bypassing cache 20298857 Files cached open 663 Files cached closed 710 Vols in Full XFC mode 0 Vols in VIOC Compatible mode Vols in No Caching mode 0 Vols in Perm. No Caching mode Granularity Hint Regions (pages): Total Free In Use Released Execlet code region 8192 4427 3765 0 Execlet data region 2048 1123 925 0 S0S1 Executive data region 7721 0 7721 0 S0S1 Resident image code region 65536 61810 3726 0 S0S1 Resident image data region 2048 1888 160 0 Slot Usage (slots): Total Free Resident Swapped Process Entry Slots 1017 782 235 0 Balance Set Slots 1015 782 233 0 Dynamic Memory Usage: Total Free In Use Largest Nonpaged Dynamic Memory (MB) 75.53 19.10 56.43 0.26 Bus Addressable Memory (MB) 2.46 2.26 0.19 2.26 Paged Dynamic Memory (MB) 12.33 6.19 6.14 4.84 Lock Manager Dyn Memory (MB) 95.12 9.86 85.25 Buffer Object Usage (pages): In Use Peak 32-bit System Space Windows (S0/S1) 6 19 64-bit System Space Windows (S2) 0 0 Physical pages locked by buffer objects 6 19 Memory Reservations (pages): Group Reserved In Use Type ORA_SGA SYSGBL 3200 3200 Page Table ORA_SGA SYSGBL 3145728 3145728 Allocated ORA_SGA SYSGBL 131072 130803 Allocated Total (25.02 GBytes reserved) 3280000 3279731 Write Bitmap (WBM) Memory Summary Local bitmap count: 4 Local bitmap memory usage (KB) 1016.00 Master bitmap count: 5 Master bitmap memory usage (MB) 1.19[/size]