Sending savesets over TCP/IP to Linux.

[martin]

Background
My backup scripts consist of a master script running on the main server and a small slave on each VM or remote machine. The master calls up the slave via SSH and directs stderr to the log and stdout to the dumpfile. The slave MUST send ONLY the dump over stdout, all other reporting is done over stderr.

Problem
Integrating a VMS VM into this! I've hit two problems:

• I can't find a way to cleanly link SYS$OUTPUT to stdout and SYS$ERROR to stderr, and then ensure that ONLY the saveset flows over SYS$OUTPUT/stdout.
• Alternatively, I've tried running the BACKUP command through OpenSSH, but specifying a remote file for the saveset won't work. I know it's easy with DECnet (done it often enough in the past) but I'd like to do it over TCP/IP/OpenSSH.

Latest (failing) attempt:

Code: Select all

$ BACKUP /IMAGE /RECORD /COMMENT='gemini-dka100' dka100: "<FQDN>:/tmp/bups-gemini-dka100-rkMXUm"/save_set

Where <FQDN> is of course the host's name.

Any suggestions/pointer please?

[sms]

Code: Select all

> My backup scripts consist of a master script running on the main
> server and a small slave on each VM or remote machine. [...]

   With my weak psychic powers, I can't see any of that.

> I can't find a way [...]

   This might make sense to you, ...

> Alternatively, I've tried running the BACKUP command through OpenSSH,
> [...]

   I don't know what that means to you.  _How_, exactly?

> [...] but specifying a remote file for the saveset won't work. [...]

      Which kind of "a remote file"?  Which kind of "won't work" is this?

   As usual, showing what you actually did (actual commands) and what
actually happened when you did it (actual behavior, error messages)
might be more helpful than a vague description of what does _not_
happen.

> Latest (failing) attempt:
> 
> $ BACKUP /IMAGE /RECORD /COMMENT='gemini-dka100' dka100: "<FQDN>:/tmp/bups-gemini-dka100-rkMXUm"/save_set

   In what sense is "<FQDN>:/tmp/bups-gemini-dka100-rkMXUm" a valid file
spec on VMS?  And what was the error message you got when that failed?
(I'd expect something like %BACKUP-F-PARSE/-RMS-F-FNM.)

> Any suggestions/pointer please?

   1. My success rate using an NFS client on VMS is low, but if your
(unspecified) Linux system offers an NFS server, then you might try
using NFS.  At least that way, a legal VMS file spec might refer to some
remote storage.

   2. Use some VMS-local storage to hold the BACKUP save set
temporarily, and then transfer it from there to its ultimate
destination using the file-transfer scheme of your choice.

   In any case, I'd think some about how to deal with the extra software
needed to restore any save set which is not accessible locally.
Especially for a system disk.  (What to do when booting the distribution
kit is not good enough.)

[martin]

"With my weak psychic powers, I can't see any of that"
Not relevant to the problem in hand, but to assist your comprehension:

1. The cli-based script starts running, loads the status codes (for later) and the functions in the master file then parses the command line before invoking the action OR the GUI starts running and performs similar steps but uses a GTK+ interface to load instructions before handing over control to the master.
2. A configuration file, either the default or as specified on the command line is loaded.
3. The master script then reads the global default items.
4. Depending upon any command line options/GTK specified the master script determines which sections to perform, in what order and whether in parallel. Importantly it then determines the correct dump level.
5. For each filesystem that needs to be backed up the master script determines an appropriate name for the dump, what zipping is required and ultimately calls an appropriate slave for the filesystem type being backed up using OpenSSH.
6. Eventually all tasks having completed, the master script reads the log files, summarises them and emails a report to root.

Slaves are provided for EFS2/3/4, for XFS and where neither of those are appropriate then either a tar or cpio based slave is available that uses a sqlite3 database to provide the multi-level functionality of other slaves.

As I said, not really relevant but might assist your weak psychic powers.

"I can't find a way [...]"
What I initially hoped to do was to ensure that only the actual saveset went to SYS$OUTPUT and that all error messages went to SYS$ERROR. However for the purposes of OpenSSH the *NIX streams stdout and stderr are more appropriate names. This would fit in perfectly with the existing system, but I was not able to find a way to ensure it worked.

Make more sense now?

"I don't know what that means to you. _How_, exactly?"

ssh gemini.home "BACKUP /IMAGE /RECORD /COMMENT='gemini-dka100' dka100: '<FQDN>:/tmp/bups-gemini-dka100-rkMXUm'/save_set >${target_dir}/${target}.${base_name}" 2>"${log_dir}/${target}.log"

Which kind of "a remote file"? Which kind of "won't work" is this?

A remote file is a file on a system other than the one executing the command. "Won't work" means that it does not function as wanted.

As usual ... happen
Really? Several hundred lines of code, two pages of configuration file, two dokuwiki articles, running to about 10 sheets of paper when printed out?

There is a good reason for a "vague description", I really think that the details of how to back up a *NIX system would not help, only cloud the problem.

In what sense ... a valid file spec on VMS
Compare:

Code: Select all

$ scp "<FQDN>:~/readme" .
readme                                        100% 2314   226.0KB/s   00:00
$ dir readme.*

Directory DISK$USERS:[JMR]

readme.;1

Total of 1 file.

Your final points:

1. NFS is a possibility. Not optimal but a fallback.
2. Again sub-optimal. The minimum size for a "spool" disk would be at least equal to the used space on my largest disk. Do-able, but a wast of spinning rust.

Extra software is not really an issue. The saveset should remain in native VMS format, but simply on a disk in my safe. If the VM is lost, then it's relatively straightforward to build a new vanilla machine just for the purpose of rebuilding the system disk image. It would be nice to use standalone backup, but that really is revealing my age!

[cct]

Martin, I used to use backup, followed by ZIP on the saveset, then copy the .ZIP file onto another platform

Worked for me!

Chris

[arne_v]

That would also be my take.

If the needed disk space is there, then local BACKUP, ZIP "-V" and whatever file transfer seems like the path of least resistance.

[sms]

Code: Select all

> Which kind of "a remote file"? Which kind of "won't work" is this?
> 
> A remote file is a file on a system other than the one executing the
> command. "Won't work" means that it does not function as wanted.

   Defining "remote file" does not answer the question posed there.  
"Which kind" and "Please define" are spelled differently for a reason.

   Similarly, defining "Won't work" does not answer the other question
posed there.  I'll try again:

   As usual, showing what you actually did (actual commands) and what
actually happened when you did it (actual behavior, error messages)
might be more helpful than a vague description of what does _not_
happen.

> $ scp "<FQDN>:~/readme" .

   The fact that a program like SCP accepts notation like <node>:<path>"
does not make that a valid VMS file spec, and BACKUP expects/requires a
valid VMS file spec.


> [...] it's relatively straightforward to build a new vanilla machine
> just for the purpose of rebuilding the system disk image. [...]

   If it's easy enough for you, then that's what matters.  I'd want to
run the experiment before I started to depend on backups which required
that to be done.


>  If the needed disk space is there, then local BACKUP, ZIP "-V" and
> whatever file transfer seems like the path of least resistance.

   People keep recommending Zip for this, but I see it as more software
which will be needed to do a restoration.  (Might be harmless, if VSI
now includes the Info-ZIP tools with the OS.)  If its compression is
useful, and "the needed disk space is there", then it might be harmless,
or even beneficial.  Without it, the block-size attribute of the save
set would need to be restored before BACKUP would tolerate it, but
that's trivially easy to do using BACKUP /REPAIR or a pretty simple DCL
command, without dragging in more software.

[joukj]

Another option is to use RESET_BACKUP_SAVESET_ATTRIBUTES.COM found here:
https://github.com/fernandezcuesta/smsc ... IBUTES.COM

Iyou can make the saveset; copy it to linux; restore it on OpenVMs; the DCL-procedure aboave restores the file-header.

[martin]

@cct, @arne and @joukj. Thanks for the suggestions. I must admit I'd be a bit wary of ZIP, and your comments about the file header are interesting. I've probably spent too long with various *NIXs and forget how much extra detail VMS keeps.

Added in 19 minutes 57 seconds:
>> Which kind of "a remote file"? Which kind of "won't work" is this?
>>
>> A remote file is a file on a system other than the one executing the
>> command. "Won't work" means that it does not function as wanted.

> Defining "remote file" does not answer the question posed there.
>"Which kind" and "Please define" are spelled differently for a reason.

> Similarly, defining "Won't work" does not answer the other question
>posed there. I'll try again:

I'm having a little difficulty seeing your problem here. You keep asking what "kind" of file, well clearly the file will hold a BACKUP saveset. As regards the filesystem, it will typically be held on an EXT3 external disk, but why would that be relevant?

> As usual, showing what you actually did (actual commands) and what
>actually happened when you did it (actual behavior, error messages)
>might be more helpful than a vague description of what does _not_
>happen.

Just repeating a cut-and-paste section doesn't help, shall we quietly ignore it?

>> $ scp "<FQDN>:~/readme" .

> The fact that a program like SCP accepts notation like <node>:<path>"
>does not make that a valid VMS file spec, and BACKUP expects/requires a
>valid VMS file spec.

At last some useful information. Are you stating that BACKUP cannot use TCP/IP in the output filespec? It can use DECnet (along the lines of BIGVAX::DISK$BACKUPS:[BACKUPS]GEMINI.BCK) but with only (currently) one VMS VM hosted by Linux then DECnet isn't an option. If BACKUP can't use TCP/IP, then there is no point in pursuing this development, which is where we came in.

[martinv]

Are you stating that BACKUP cannot use TCP/IP in the output filespec?

Yes, that's where the problem lies.

It can use DECnet

DECnet is more deeply integrated with VMS than TCP/IP.

And that is where the "BACKUP/ZIP, then copy" suggestions stem from - you have to decouple BACKUP from the network transfer. An advanteage of ZIP is that when used with the "-V" option, it saves the record format information, and UNZIP restores it.

HTH,
Martin

[martin]

Thank you. A simple statement that what I was attempting is not possible, so I will need to adopt an alternative approach. Thanks also for the heads up about ZIP -V.

I'll have to park things for a while. A failing SATA controller on the host system has just knocked out a couple of disks and crashed my RAIDset. I'll be a bit busy rebuilding from dumps. I don't back up my /virt filesystem on the host (one small change in a VM causes the whole virtual disk to be backed up), only by using the VMs themselves - and my VMS machine doesn't yet back up! Ouch!