RMS Files - Best Practices

Management of storage subsystems: SAN, volume shadowing, logical disks, file systems, and more.
Post Reply

Topic author
vmskostoff
Active Contributor
Posts: 39
Joined: Fri Jun 28, 2019 10:29 am
Reputation: 0
Location: Gary, Indiana
Status: Offline

RMS Files - Best Practices

Post by vmskostoff » Wed Sep 08, 2021 1:21 pm

Looking for suggestions on how to optimize RMS files on OpenVMS 8.4 and OpenVMS 7.3-1.
Our applications use RMS files.

Would like some suggestions on best practices for the optimization of the RMS files.
1. RMS file Maintenance.
2. Analysis - first - to know where starting before any optimization.
3. Index reset.
4. Index maintenance.
5. Compression?

Anything I need to know and have not asked?

Thanks.
Vince
Last edited by vmskostoff on Wed Sep 08, 2021 3:33 pm, edited 1 time in total.


hein
Active Contributor
Posts: 41
Joined: Fri Dec 25, 2020 5:20 pm
Reputation: 0
Status: Offline

Re: RMS Files - Best Practices

Post by hein » Wed Sep 08, 2021 6:29 pm

Hi there Vince.

Now there's a loaded question, or rather series of questions, you ask.
What have you done so far to get up to speed on this topic.
Read the manuals? Tried some stuff? gathered (T4) data to suggest whether the is or might not be an issue. What triggered the questions now?

Some folks have spent almost a lifetime trying to figure this stuff out (me!)
Back in 2008 I made a powerpoint presentation for the OpenVMS Bootcamp to get folks going.
I called it RMS Accidental Indexed File Owner Maybe you are such person? 🙂.
Perhaps that get you going some. Follow that link!

Main thing to look at

Convert!
Once every year whether your files need it or not.
If there is not a properly designed FDL available, then for bonus point just use ANAL/RMS/FDL and EDIT/FDL/NOINTERACTIVE

Global Buffers - they are magic. SET FILE/GLOB=500/SHARE
You probably need them badly. Start with 500 for larger, frequently used files.

Local buffers
SET RMS/BUF=20/IND/SYS

Good luck!
Hein


hein
Active Contributor
Posts: 41
Joined: Fri Dec 25, 2020 5:20 pm
Reputation: 0
Status: Offline

Re: RMS Files - Best Practices

Post by hein » Thu Sep 09, 2021 9:36 am

Link failed to come over.
Try this as one line

https://docs.google.com/presentation/d/ ... ue&sd=true

Hein


Topic author
vmskostoff
Active Contributor
Posts: 39
Joined: Fri Jun 28, 2019 10:29 am
Reputation: 0
Location: Gary, Indiana
Status: Offline

Re: RMS Files - Best Practices

Post by vmskostoff » Fri Sep 10, 2021 9:59 am

Greetings Hein,

You are a brave man for responding as I was brave to post.

I have been reading manuals and trying to piece together a plan and background from there.
I was hoping to augment that process with the post of my questions.

I have not ventured down the path of T4 yet.

Why now? Trying to optimize our environment. People resources more and more limited so trying to be
more "Admin-like" and get ahead of any potential issues.

I have attended the OpenVMS Bootcamp, once, but not in 2008. I attended in 2011. Thoroughly enjoyed it.

Thanks for the link. I will review.

As I initially did some simple "googles" it became apparent as you indicate that it is
a very deep subject.

I appreciate you taking the time to reply to my series of questions.

Keep you posted on my progress.
Vince

User avatar

arne_v
Master
Posts: 299
Joined: Fri Apr 17, 2020 7:31 pm
Reputation: 0
Location: Rhode Island, USA
Status: Offline
Contact:

Re: RMS Files - Best Practices

Post by arne_v » Fri Sep 10, 2021 10:42 am

@Hein

General question - not related to specific question posted here.

1991: VAX 6000 - 2 cores @ 35 Mhz, 32 MB RAM, some 1 GB rotating disks, 1 million block index-sequential file (500 MB)

maintaining index-sequential files are critical for performance.

2021: Itanium - 4 cores @ 1.5 GHz, 32 GB RAM, some 1 TB SSD's, 10 million block index-sequential file (5 GB)

how much maintenance is still needed?
Arne
arne@vajhoej.dk
VMS user since 1986


willemgrooters
Valued Contributor
Posts: 87
Joined: Fri Jul 12, 2019 1:59 pm
Reputation: 0
Location: Netherlands
Status: Offline
Contact:

Re: RMS Files - Best Practices

Post by willemgrooters » Fri Sep 10, 2021 3:13 pm

It's never a bad idea to maintain index-sequential files.

On Alpha, not that long ago, I worked for a customer with many separate installations of the same software. It accessed quite a number of indexed-sequential files, and the application should be available "24/7, all year though". The recommendation was to maintain these files (convert/fdl, recreate indexes etc) regularly; most heavily changed files more often (some once a week), less heavily changes less. That would cause some downtime in the night but that was acceptable.
We knew that some sites followed the recommendation and that some were more relaxed - for whatever reason.

At some point, all sites were required to periodically extract data from their files to feed a centralized system. It put quite a load on our software: the larger the number of records in our system, the heavier.
Here, the difference was clear immediately: Large sites that had their files maintained regularly, had no trouble at all to extract their data in just a few (nightly) hours. Sites that were more relaxed in their maintenance, managed to do so in somewhat longer elapsed times, eventually finishing early the next office day.
Except for one site, where the run was aborted day after day because it blocked normal usage.
When I investigated the case, it turned out that NONE of the files that were accessed by the extracting program, had EVER been maintained; we learned that people at that site had been complaining about lack of performance for a long time.
It would take at least one, perhaps 2 or 3 days to re-organize the files: Convert to sequential on an empty disk, and re-create the indexes afterwards during which time the system would - of course - be unavailable (since these files could not be updated during that period). Even worse: Since there was no empty disk, not sufficient free space to execute this, it could even be longer.


hein
Active Contributor
Posts: 41
Joined: Fri Dec 25, 2020 5:20 pm
Reputation: 0
Status: Offline

Re: RMS Files - Best Practices

Post by hein » Sat Sep 11, 2021 7:50 pm

Arne,
Absolutely. Times have changed. Back in the day I might have recommend 10-record buckets and 20 global buffers for a critical file. Now just for for 16 or even 63 block buckets and 500 global buffers and no worries about memory and little about CPU. We can now do 2000 IO/sec or even 5000 versus 50 or 100 back in the day,,, it makes a difference.
Still, some files my still define or have an extreme impact, on overall application performance in highly contentious environments. The pain may have shifted to locking from IO but it may still warrant some critical path analysis and tuning.
Global buffers (and XFC) are often the solution.

Vince,
Thanks for your humorous approach. Yup it was a 'big' question to approach.
Do check out T4, It will help you focus what elements (locks, cpu, IO) to focus on.
Next thing is to try to get an honest view of the 'hot files', not just which files you think are critical.
It's the old 80/20 rule. Most files are not worth your time worrying about them, but some...
Hint: $SHOW MEM/CACHE=(TOPQIO=2O, VOLUME=xxx).


Willem,
Good write-up that's indeed how it goes too often.
Still, one good whack and all will be well for a good bit of time - 'convert once a year, whether you need it or not'.
The extreme slow converts where probably due to minimal bucket sizes - barely big enough to hold a single record - and data buckets scattered all over causing a single io for every one or two records. Once you convert to 20+ records/bucket that trouble typically gone, but the first time is indeed painful to watch.,
btw... there is a technical, proven solution to this problem but commercially unlikely to be viable. There is an RMS CDC product capturing all CRUD (put,update,delete) events for a set of files. Enable that for the file to be converted.
Start converts. Once finished with now a or two day old data, just apply the journal to the freshly converted file in a few minutes. Boom!

Cheers,
Hein
Last edited by hein on Sat Sep 11, 2021 7:51 pm, edited 1 time in total.

User avatar

imiller
Master
Posts: 130
Joined: Fri Jun 28, 2019 8:45 am
Reputation: 0
Location: South Tyneside, UK
Status: Offline
Contact:

Re: RMS Files - Best Practices

Post by imiller » Mon Sep 13, 2021 4:59 am

I was probably there for that presentation by Hein and still have a copy to refer to occasionally. ;)

The advice about using SHOW MEM/CACHE to spot the hot files should be followed - these are the ones to concentrate on.

It's surprising what big improvement a few changes can made.

Do use T4 with the RMS data collector addon. You may not understand the data yet but you will be collecting the data for when you need it. Have a look at the new T4 viewer PDview http://xdelta.co.uk/xdelta_t4viewer.html
Ian Miller
[ personal opinion only. usual disclaimers apply. Do not taunt happy fun ball ].

Post Reply