cgi module on VMS x86-64

User avatar

Topic author
arne_v
Master
Posts: 450
Joined: Fri Apr 17, 2020 7:31 pm
Reputation: 0
Location: Rhode Island, USA
Status: Offline
Contact:

cgi module on VMS x86-64

Post by arne_v » Thu May 09, 2024 9:17 am

(I know this is very old, officially deprecated and going away)

It does not seem to work.

Code: Select all

import cgi
...
form = cgi.FieldStorage()
and form is all empty despite being a valid form submission.

I assume it has something to do with the fact that VMS Apache - VMS Python is different from *nix Apache - *nix Python.

I have tried the obvious of doing a:

$ define/nolog sys$input apache$input

in the COM wrapper before starting Python, but it did not solve the problem.

(obvious because in C or whatever then one actually has to read form input from apache$input not stdin / sys$input - and it was what was needed to make it work on Alpha using JFP Python 2.7)
Arne
arne@vajhoej.dk
VMS user since 1986

User avatar

neilrieck
Active Contributor
Posts: 30
Joined: Tue Jan 10, 2023 10:41 am
Reputation: 0
Location: Waterloo, Ontario, Canada
Status: Offline
Contact:

Re: cgi module on VMS x86-64

Post by neilrieck » Wed Jun 19, 2024 9:50 am

This will be a complicated reply:
1) your "cgi library" demo should work but is probably broken for several reasons
2) In my initial demo code I was unable to read shell variable "REQUEST_METHOD" until I modified the behaviour of Apache on OpenVMS by defining these system-level logical names (APACHE$CREATE_SYMBOLS_GLOBAL = "Y") and ("APACHE$CGI_MODE = 0) then restarting Apache. BTW, this stuff is missing from the current VSI Apache release notes, so an incomplete list can be found here: https://neilrieck.net/docs/openvms_note ... r-logicals
IMHO, VSI would be wise to copy this stuff to their Apache release notes ASAP.
3) After these changes, I was now able to read "REQUEST_METHOD" but discovered that there might be an error in how the python "os library" has been implemented by VSI in all platforms. I'm no Python expert but was under the impression that "os.getenv is a wrapper to os.environ.get" so was surprised when the first worked while the second did not. Speculation: the "cgi library" might not work properly depending upon which method was used. Anyway here's some demo code for use from the command line (works differently on OpenVMS than Linux)

Code: Select all

'''
==================================================
title  : python_demo_100.py
author : Neil Rieck
created: 2024-06-10
notes  :
1) no shebang on OpenVMS
2) OpenVMS usage:
        $ TEST1="111"   ! local
        $ TEST2=="222"  ! global (like an exported symbol in BASH)
        $ sho sym TEST*
        $ def/proc     "TEST3" "333"
        $ def/job      "TEST4" "444"
        $ def/job/exec "TEST5" "555"
        $ python python_demo_101.py
history:
ver who when   what
--- --- ------ -----------------------------------
100 NSR 240610 original effort
101 NSR 240618 added support for logical names
==================================================
notes  :
'''
import cgi
import os
import sys

TITLE = "python_demo_100"

def main():
    print(f"-i-program: {sys.argv[0]}")
    #
    #   these two calls should work but do not
    #
    symbol1 = os.environ.get("TEST1", "blank")
    print(f"symbol1  = '{symbol1}'")
    symbol2 = os.environ.get("TEST2", "blank")
    print(f"symbol2  = '{symbol2}'")
    #
    #   and now we know why
    #
    yada = os.environ
    print("-----")
    for i in yada:
        print(i)
    print("-----")
    #
    #   the second call should work and does
    #
    symbol1 = os.getenv("TEST1", "blank")
    print(f"symbol1  = '{symbol1}'")
    symbol2 = os.getenv("TEST2", "blank")
    print(f"symbol2  = '{symbol2}'")

if __name__ == "__main__":
    main()

#this is the last line 
4) changing logical APACHE$CGI_MODE to 2 (or 1 if the data is greater than 970 bytes) means you will be dealing with logical names rather than shell variables, I have written my own routines for translating logical names, and it works perfectly, except with Apache on OpenVMS x86. At this point in time "I think" the LNM$PROCESS logical name table is being clobbered when I start python. I'll keep testing.
Last edited by neilrieck on Wed Jun 19, 2024 11:02 am, edited 1 time in total.

User avatar

Topic author
arne_v
Master
Posts: 450
Joined: Fri Apr 17, 2020 7:31 pm
Reputation: 0
Location: Rhode Island, USA
Status: Offline
Contact:

Re: cgi module on VMS x86-64

Post by arne_v » Sat Jun 22, 2024 4:41 pm

$ def/sys APACHE$CREATE_SYMBOLS_GLOBAL "Y"
$ def/sys APACHE$CGI_MODE 0

before starting Apache did not populate FieldStorage with form data.
Arne
arne@vajhoej.dk
VMS user since 1986

User avatar

neilrieck
Active Contributor
Posts: 30
Joined: Tue Jan 10, 2023 10:41 am
Reputation: 0
Location: Waterloo, Ontario, Canada
Status: Offline
Contact:

Re: cgi module on VMS x86-64

Post by neilrieck » Mon Jun 24, 2024 9:41 am

As I mentioned previously, os.getenv is implemented as a wrapper around os.environ on most other systems, but this is not the case on OpenVMS (and only VSI would be able to tell us why). A little python hacking shows that os.getenv can read global ("==") DCL variables (like REQUEST_METHOD + QUERY_STRING + CONTENT_LENGTH) but os.environ cannot. Inspecting "cpi.py" on OpenVMS shows that it employs os.environ so now we know why this doesn't work for you. So at this point it appears that you will need to roll your own CGI solution based up os.getenv. You start off by reading REQUEST_METHOD. If is is defined as "GET" then all your information will be in QUERY_STRING. If it is defined as "POST" then you will need to open a read-only connection back to stdin then read bytes up to CONTENT_LENGTH. Anyway, this is the way it has been for me for the past 20 years with VMS-BASIC and DEC-C.

Suggestion: be prepared to play with APACHE$COMMON:[000000.CGI-BIN]TEST-CGI-VMS.COM (or call this script from the DCL you are using to call python on OpenVMS x86.

Added in 5 hours 53 minutes 36 seconds:
(only working on this during my coffee breaks).

I took a peek at os.py and now know why os.getenv is different compared to os.enron so I wrote a little stub to copy from DCL into os.environ and now this thing works in GET mode (but not post) :mrgreen:

Code: Select all

'''
======================================================
title  : apache$common:[scripts]python_demo_101.py
author : Neil Rieck
notes  : no shebang on OpenVMS
history:
ver who when   what
--- --- ------ -----------------------------------
100 NSR 240610 original effort (DCL variables)
101 NSR 240618 added support for logical names
102 NSR 240524 now copy some DCL stuff into os.environ
======================================================
'''
import cgi
import os
import sys
# import logical_name as ovms

TITLE = "python_cgi_demo"

stuff = {'CONTENT_LENGTH', 'CONTENT_TYPE',
         'QUERY_STRING', 'REQUEST_METHOD'}

def main():
    debug = 0
    script = sys.argv[0]
    print("Status: 200\n"
          "content-type: text/html\n\n"
          "<html>\n<head>\n"
          f"<title>{TITLE}</title>\n"
          "</head>\n<body>\n"
          '<form action="/scripts/python_cgi_demo" method="get">'
          'A <input type="text" name="a"/> '
          'B <input type="text" name="b"/> '
          '<input type="submit" value="submit"/>'
          '</form>'
          f"<pre>hello from: {script}")
    if debug > 0:
        print("---------- 0a")
        print(os.environ)
        print("---------- 0b")
    for i in stuff:
        # print(i)
        yada = os.getenv(i, "")
        if yada != "":
            os.environ[i] = yada
    if debug > 0:
        print("---------- 0c")
        print(os.environ)
        print("---------- 1")
    fld = cgi.FieldStorage(keep_blank_values=True)
    if len(fld) > 0:
        for name in fld:
            value = fld[name].value
            print(name," = ",value)
    print("</pre>"
          "</body></html>")

if __name__ == "__main__":
    main()

#this is the last line

User avatar

Topic author
arne_v
Master
Posts: 450
Joined: Fri Apr 17, 2020 7:31 pm
Reputation: 0
Location: Rhode Island, USA
Status: Offline
Contact:

Re: cgi module on VMS x86-64

Post by arne_v » Mon Jun 24, 2024 8:13 pm

os.environ look generally useless in 3.10 / x86-64.

Code: Select all

$ type env.py
import os

def dump(nam):
    print('%-10s %-10s %-10s' % (nam, os.getenv(nam), os.environ.get(nam)))

print('           os.getenv  os.environ')
for nam in ['lclsym', 'glbsym', 'usrlog', 'sprlog']:
    dump(nam)
$ lclsym = "OK"
$ glbsym == "OK"
$ def/nolog/user usrlog "OK"
$ def/nolog/super sprlog "OK"
$ python env.py
VSI Python 3.10 / VMS 9.2-2 x86-64:

Code: Select all

           os.getenv  os.environ
lclsym     OK         None
glbsym     OK         None
usrlog     OK         None
sprlog     OK         None
JFP Python 2.7 / VMS 8.4-2L2:

Code: Select all

           os.getenv  os.environ
lclsym     OK         OK
glbsym     OK         OK
usrlog     OK         OK
sprlog     OK         OK
Added in 31 minutes 28 seconds:
But I will give the workaround a try.
Arne
arne@vajhoej.dk
VMS user since 1986


sergey_vorfolomeev
VSI Expert
Master
Posts: 104
Joined: Thu Aug 22, 2019 12:17 am
Reputation: 0
Status: Offline

Re: cgi module on VMS x86-64

Post by sergey_vorfolomeev » Tue Jun 25, 2024 3:56 am

The issue will be fixed in the next release

Code: Select all

VMS4> create env_test.py
import os
def dump(nam):
    print('%-10s %-10s %-10s' % (nam, os.getenv(nam), os.environ.get(nam)))

print('           os.getenv  os.environ')
for nam in ['lclsym', 'glbsym', 'usrlog', 'sprlog']:
    dump(nam)
 Exit
VMS4> lclsym = "OK"
VMS4> glbsym == "OK"
VMS4> def/nolog/user usrlog "OK"
VMS4> def/nolog/super sprlog "OK"
VMS4> python env_test.py
           os.getenv  os.environ
lclsym     OK         OK
glbsym     OK         OK
usrlog     OK         OK
sprlog     OK         OK
VMS4>

User avatar

neilrieck
Active Contributor
Posts: 30
Joined: Tue Jan 10, 2023 10:41 am
Reputation: 0
Location: Waterloo, Ontario, Canada
Status: Offline
Contact:

Re: cgi module on VMS x86-64

Post by neilrieck » Tue Jun 25, 2024 8:29 am

Another quick coffee-break hack (did this just as an intellectual challenge). I added this to my mainline python script and it works:

Code: Select all

    try:
        CL = os.environ['CONTENT_LENGTH']
        CL = int(CL)
        if CL > 0:
            f = open("/APACHE$INPUT/", "r")
            print(f.read(CL))
    except Exception as e:
        # print(f"-e-error: {e}")
        pass
Just as there is some specialty code for OpenVMS added to library "os.py", something similar could easily be added to library "cgi.py" to get you over the hump.

p.s. This is one of the joys of opensource. 99 people out of 100 might not think a given problem is not urgent, but 1 person out of 100 may disagree then can do something about it. Two years ago, I was working on a SOAP problem in the zeep library where I found an actual bug. I fixed the problem at my end then notified the maintainer of zeep. My fix still has not made it into the zeep code maintained at github :mrgreen:

Added in 13 minutes 13 seconds:
sergey_vorfolomeev wrote:
Tue Jun 25, 2024 3:56 am
The issue will be fixed in the next release

Code: Select all

VMS4> create env_test.py
import os
def dump(nam):
    print('%-10s %-10s %-10s' % (nam, os.getenv(nam), os.environ.get(nam)))

print('           os.getenv  os.environ')
for nam in ['lclsym', 'glbsym', 'usrlog', 'sprlog']:
    dump(nam)
 Exit
VMS4> lclsym = "OK"
VMS4> glbsym == "OK"
VMS4> def/nolog/user usrlog "OK"
VMS4> def/nolog/super sprlog "OK"
VMS4> python env_test.py
           os.getenv  os.environ
lclsym     OK         OK
glbsym     OK         OK
usrlog     OK         OK
sprlog     OK         OK
VMS4>
Can hardly wait. Many thanks for your efforts. :mrgreen:

User avatar

Topic author
arne_v
Master
Posts: 450
Joined: Fri Apr 17, 2020 7:31 pm
Reputation: 0
Location: Rhode Island, USA
Status: Offline
Contact:

Re: cgi module on VMS x86-64

Post by arne_v » Wed Jun 26, 2024 10:54 am

sergey_vorfolomeev wrote:
Tue Jun 25, 2024 3:56 am
The issue will be fixed in the next release

Code: Select all

VMS4> create env_test.py
import os
def dump(nam):
    print('%-10s %-10s %-10s' % (nam, os.getenv(nam), os.environ.get(nam)))

print('           os.getenv  os.environ')
for nam in ['lclsym', 'glbsym', 'usrlog', 'sprlog']:
    dump(nam)
 Exit
VMS4> lclsym = "OK"
VMS4> glbsym == "OK"
VMS4> def/nolog/user usrlog "OK"
VMS4> def/nolog/super sprlog "OK"
VMS4> python env_test.py
           os.getenv  os.environ
lclsym     OK         OK
glbsym     OK         OK
usrlog     OK         OK
sprlog     OK         OK
VMS4>
Nice.

I think that may avoid some confusion in the future.

There still seems to be problems with cgi module regarding POST. But since cgi module will be removed in Python 3.13 then maybe not as important.
Arne
arne@vajhoej.dk
VMS user since 1986

User avatar

neilrieck
Active Contributor
Posts: 30
Joined: Tue Jan 10, 2023 10:41 am
Reputation: 0
Location: Waterloo, Ontario, Canada
Status: Offline
Contact:

Re: cgi module on VMS x86-64

Post by neilrieck » Thu Jun 27, 2024 4:22 pm

neilrieck wrote:
Tue Jun 25, 2024 8:42 am
Another quick coffee-break hack (did this just as an intellectual challenge) for use with POST. I added this to my mainline python script and it works:

Code: Select all

    try:
        CL = os.environ['CONTENT_LENGTH']
        CL = int(CL)
        if CL > 0:
            f = open("/APACHE$INPUT/", "r")
            print(f.read(CL))
    except Exception as e:
        # print(f"-e-error: {e}")
        pass
Just as there is some specialty code for OpenVMS added to library "os.py", something similar could easily be added to library "cgi.py" to get you over the hump. Or you could copy cgi.py from its current location to your working directory, then modify the copy,

p.s. This is one of the joys of opensource. 99 people out of 100 might not think a given problem is not urgent, but 1 person out of 100 may disagree then can do something about it. Two years ago, I was working on a SOAP problem in the zeep library where I found an actual bug. I fixed the problem at my end then notified the maintainer of zeep. My fix still has not made it into the zeep code maintained at github :mrgreen:

Added in 13 minutes 13 seconds:
sergey_vorfolomeev wrote:
Tue Jun 25, 2024 3:56 am
The issue will be fixed in the next release

Code: Select all

VMS4> create env_test.py
import os
def dump(nam):
    print('%-10s %-10s %-10s' % (nam, os.getenv(nam), os.environ.get(nam)))

print('           os.getenv  os.environ')
for nam in ['lclsym', 'glbsym', 'usrlog', 'sprlog']:
    dump(nam)
 Exit
VMS4> lclsym = "OK"
VMS4> glbsym == "OK"
VMS4> def/nolog/user usrlog "OK"
VMS4> def/nolog/super sprlog "OK"
VMS4> python env_test.py
           os.getenv  os.environ
lclsym     OK         OK
glbsym     OK         OK
usrlog     OK         OK
sprlog     OK         OK
VMS4>
Can hardly wait. Many thanks for your efforts. :mrgreen:

User avatar

neilrieck
Active Contributor
Posts: 30
Joined: Tue Jan 10, 2023 10:41 am
Reputation: 0
Location: Waterloo, Ontario, Canada
Status: Offline
Contact:

Re: cgi module on VMS x86-64

Post by neilrieck » Wed Jul 03, 2024 4:24 pm

This just popped into my head:

Code: Select all

'''
===========================================================
title  : getenv_via_decc.py
author : Neil Rieck
created: 2024-07-03
notes  : I remembered that a "getenv" function in c stdlib
would look for a DCL symbols then (upon failure) look for a
logical name so I wrote this little stub to see if the VSI
version of python for OpenVMS could do something similar.
It appears that it can. This could be used to write an
alternate version of "cgi"
DCL use:
 $ neil123="neil123"            ! create local symbol
 $ neil456=="neil456"           ! create global symbol
 $ def/log "neil789" "neil789"  ! create logical name
 $ python getenv_via_decc.py
============================================================
'''
import _decc as DECC

test = DECC.getenv("neil123")
print(test)
test = DECC.getenv("neil456")
print(test)
test = DECC.getenv("neil789")
print(test)

Post Reply