HP Basic Reverse file read


Topic author
kwfeese
Contributor
Posts: 10
Joined: Fri Nov 18, 2022 11:34 am
Reputation: 0
Status: Offline

HP Basic Reverse file read

Post by kwfeese » Tue Feb 28, 2023 6:32 pm

I'm trying to read a file in reverse using HP Basic and I came across this paragraph in Guide to OpenVMS File Applications:

8.4.3 Processing Indexed Files
Indexed files provide the most record-processing flexibility. Your program can read existing records from the file in sequential, random access by RFA mode or random access by key mode. RMS also allows you to write any number of new records into an indexed file if you do not violate a specified key constraint, such as not allowing duplicate key values.

In random access by key mode, RMS provides two forward search key options for use with one of four match options (see Section 8.4.3.2). A reverse search key option permits reverse random access when used in combination with either of the two forward search key options.

There was an example but it was written in C and, unfortunately, my current company doesn't have a C license.

Has anyone ever accomplished this and if so can you provide examples?


sms
Master
Posts: 310
Joined: Fri Aug 21, 2020 5:18 pm
Reputation: 0
Status: Offline

Re: HP Basic Reverse file read

Post by sms » Tue Feb 28, 2023 8:32 pm

Code: Select all

> I'm trying to read a file in reverse [...]

   Is this an existing file (with some kind of structure), or some new
file which you'll be creating in some way or other, or what?

   It might be helpful to describe in more detail the actual problem
which you are trying to solve, rather than asking how to implement some
particular "solution" or technique ("read a file in reverse").

> [...] in Guide to OpenVMS File Applications: [...]
> There was an example but it was written in C [...]

   Have you looked at the BASIC documentation?

      https://docs.vmssoftware.com/   (Look for "COMPILERS"?)

   The "VSI BASIC for OpenVMS User Manual" includes "Chapter 13. File
Input and Output", which includes section 13.2.4 on "Indexed Files".

User avatar

arne_v
Master
Posts: 299
Joined: Fri Apr 17, 2020 7:31 pm
Reputation: 0
Location: Rhode Island, USA
Status: Offline
Contact:

Re: HP Basic Reverse file read

Post by arne_v » Tue Feb 28, 2023 8:32 pm

You have seen some C code that are working with an index-sequential file using RMS API specifically SYS$FIND with rab.rab$l_rop set to something | RAB$M_REV and you want to do the same in Basic?

Added in 10 minutes 42 seconds:
In Basic you would most likely not use RMS directly but instead use Basic builtin statements. Instead of SYS$FIND and SYS$GET just use FIND and GET statements.

I am not a Basic person, but I believe that the way to read reverse is to specify PRIMARY KEY ... DESCENDING in the OPEN statement.

https://docs.vmssoftware.com/vsi-basic- ... nce-manual
https://docs.vmssoftware.com/vsi-basic- ... ser-manual

has all the details.

If you get stuck then I could try create a working example.
Arne
arne@vajhoej.dk
VMS user since 1986


fim
Active Contributor
Posts: 31
Joined: Wed Jan 04, 2023 5:14 am
Reputation: 0
Status: Offline

Re: HP Basic Reverse file read

Post by fim » Wed Mar 01, 2023 10:57 am

From what I can see in Basic manuals you can read indexed file only forward. Then there is an option, read the entire file from start to finish in key order, store the keys in an array in memory. Then you retrieve the keys in your array in backward order, with each key you use the basic command FIND with EQ and then the command GET.
/Fim W.


Topic author
kwfeese
Contributor
Posts: 10
Joined: Fri Nov 18, 2022 11:34 am
Reputation: 0
Status: Offline

Re: HP Basic Reverse file read

Post by kwfeese » Wed Mar 01, 2023 12:59 pm

sms wrote:
Tue Feb 28, 2023 8:32 pm

Code: Select all

> I'm trying to read a file in reverse [...]

   Is this an existing file (with some kind of structure), or some new
file which you'll be creating in some way or other, or what?

   It might be helpful to describe in more detail the actual problem
which you are trying to solve, rather than asking how to implement some
particular "solution" or technique ("read a file in reverse").

> [...] in Guide to OpenVMS File Applications: [...]
> There was an example but it was written in C [...]

   Have you looked at the BASIC documentation?

      https://docs.vmssoftware.com/   (Look for "COMPILERS"?)

   The "VSI BASIC for OpenVMS User Manual" includes "Chapter 13. File
Input and Output", which includes section 13.2.4 on "Indexed Files".
It's an indexed History file currently at 67m blocks and the file is corrupt so I'm trying to extract as much data from the file as possible by sequentially reading the file forwards and backwards. I've tried to use the convert utility to no avail.


sms
Master
Posts: 310
Joined: Fri Aug 21, 2020 5:18 pm
Reputation: 0
Status: Offline

Re: HP Basic Reverse file read

Post by sms » Wed Mar 01, 2023 2:07 pm

Code: Select all

> [...] I've tried to use the convert utility to no avail.

   "tried" is not a very detailed description of what you did, and "to
no avail" is no better as a description of what happened when you did
it.

> [...] read the entire file from start to finish in key order, store
> the keys in an array in memory [...]

   I know nothing about that array stuff, but it did leap out at me as I
was skimming the BASIC docs.

User avatar

arne_v
Master
Posts: 299
Joined: Fri Apr 17, 2020 7:31 pm
Reputation: 0
Location: Rhode Island, USA
Status: Offline
Contact:

Re: HP Basic Reverse file read

Post by arne_v » Wed Mar 01, 2023 8:11 pm

Unfortunately primary key descending is for the actual index in the file and can not be changed as one want.

So options are:
- use RMS API (yuck)
- read all keys into array and process in reverse order (sounds like that will not work based on the description)
- read chunks where each chunk is read forward but the order of which chunks in the file is read is reversed (hack)

Illustration of the last:

Code: Select all

$ type gen.pas
program gen(input,output);

type
   fixstring = packed array [1..8] of char;
   mydata = record
               nam : [KEY(0)] fixstring;
               v   : integer;
            end;

var
   f : file of mydata;
   d : mydata;

begin
   open(f, 'fwdrev.isq', new, organization := indexed, access_method := keyed);
   rewrite(f);
   d.nam := 'A A';
   d.v   := 1;
   f^ := d;
   put(f);
   d.nam := 'B B';
   d.v   := 2;
   f^ := d;
   put(f);
   d.nam := 'C C';
   d.v   := 3;
   f^ := d;
   put(f);
   d.nam := 'D D';
   d.v   := 4;
   f^ := d;
   put(f);
   close(f);
end.
$ pas gen
$ link gen
$ run gen
$ type fwd.bas
program fwd

record mydata
    string nam = 8
    integer v
end record
map (buf) mydata d

open "fwdrev.isq" as file #1, indexed fixed, recordtype none, map buf, primary key d::nam
handler eof_handler
end handler
when error use eof_handler
    while 1 = 1
        get #1
        print d::nam,d::v
    next
end when
close #1

end program
$ bas fwd
$ link fwd
$ run fwd
A A            1
B B            2
C C            3
D D            4
$ type rev.bas
program rev

record mydata
    string nam = 8
    integer v
end record
map (buf) mydata d
declare string target
declare integer i

handler eof_handler
end handler

open "fwdrev.isq" as file #1, indexed fixed, recordtype none, map buf, primary key d::nam
for i = 1 to 26
    target = chr$(91 - i)
    when error use eof_handler
        find #1, key #0 ge target
        get #1
        while mid(d::nam, 1, 1) = target
            print d::nam,d::v
            get #1
        next
    end when
next i
close #1

end program
$ bas rev
$ link rev
$ run rev
D D            4
C C            3
B B            2
A A            1
Arne
arne@vajhoej.dk
VMS user since 1986


hein
Active Contributor
Posts: 41
Joined: Fri Dec 25, 2020 5:20 pm
Reputation: 0
Status: Offline

Re: HP Basic Reverse file read

Post by hein » Sun Apr 16, 2023 10:47 am

OP @kwfeese wrote in a reply>>
"It's an indexed History file currently at 67m blocks and the file is corrupt so I'm trying to extract as much data from the file as possible by sequentially reading the file forwards and backwards. I've tried to use the convert utility to no avail."

Hmmm... Why didn't you say so in the first place? That makes any suggestion to read (and store key values) useless.
Why do yo think reading backward would offer significant help (it might - if there is only 1 bad 'spot') ?

I only saw this submission just now. I suspect/hope you solved the issue by now possibly using a backup?
Or maybe, just maybe CONVERT works when reading by ALTERNATE key, which will take a long time and requires sorting afterwards.

Anyway to answer the initial question: To read an indexed file backwards in BASIC one supposedly needs to use a USEROPEN to snarf the RAB address. Next while processing, set the RAB$V_REV bit in the rab::rab$l_rop. This is probably best set by ORring with the RAB$M_REV mask as Arne indicated. Then go about with Basic's native GETs. There is a small chance BASIC fill nuke the whole ROP with FIND/GET by key. We (the group, me) could verify that easily, but by now the whole original problem probably went away already and it would likely not be the best solution for the underlying problem anyway. But please do ask if an example/test is still deemed valuable.

For the benefit of others, How 'bad' is/was the corruption? When (in the processing), Where (in the file) does it occur?
As Arne already suggested, then best way is probably to 'chunk' your way through the file in a binary search fashion, possibly guided by a (stale) backup of the file.
Supposedly there is a 'structure' and predictability to the Primary key. Perhaps there is a date-time component, or a slowly increasing value, if none (the proverbial 'Social Security Number') there is still an order. So is key 12345678 fails the you could try 123456000 or 123460000 until you find a spot that does work and step or binary search backwards (with forward searches :-). Or read a backup starting with 12345678 and start trying the next values from the backup on the target file.
Repeat until the end is reach.

OR, you could try fixing the file (or hire someone to do so).
For that you may want to take a peek at a presentation I wrote years ago (nothing changed)

https://docs.google.com/presentation/d/ ... ue&sd=true

Good luck!
Hein.
Last edited by hein on Sun Apr 16, 2023 10:50 am, edited 1 time in total.


Topic author
kwfeese
Contributor
Posts: 10
Joined: Fri Nov 18, 2022 11:34 am
Reputation: 0
Status: Offline

Re: HP Basic Reverse file read

Post by kwfeese » Tue May 02, 2023 4:53 pm

Thanks Hein,

You're presention shed light on my objective. Now I'm struggling with sorting out how to read by RFA, it's out of realm of expertise.

Basically, I want to start with REC 1 and read every subsequent record until end of the file. With any luck that will resolve the issue my client has.


hein
Active Contributor
Posts: 41
Joined: Fri Dec 25, 2020 5:20 pm
Reputation: 0
Status: Offline

Re: HP Basic Reverse file read

Post by hein » Wed May 03, 2023 10:00 am

Reading a file 'by RFA' is relatively easy in Basic - easier than any other language - with the GETRFA function and GET/FIND RFA <rfavariable> statements.
I don't think it will offer much though, and RFA's are 'opaque' and while you'll recognize some structure (VBN/ID) in them and may think you'll be able to 'predict' (VBN last VBN + bucket size) the next RFA or brute-force your way though the file you'll find it jumps around and skews (on file extends)

You are better off focusing on primary key values, aided as I indicated by values from backups.

It may well be still better to try and fully understand the actual corruption as such (chunk of file zeroed out? missing update causing a missing link? Random data in some block?) and try to 'patch' around that. Again a more or less recent back may help a lot with that process. I appreciate it is a pretty tall order to learn the indexed file internals needed for this, but on the other hand it is quite doable once you set your mind to it. Maybe someone can help you with this (for money)? It will be much better for your customer.

Hein.

Post Reply