VSI OpenVMS Forum

The official board to discuss OpenVMS-related topics
https://forum.vmssoftware.com/

Embedding VSI Python 3.10 IA64

https://forum.vmssoftware.com/viewtopic.php?f=32&t=8600

Page 1 of 2

Embedding VSI Python 3.10 IA64

Posted: Mon Apr 10, 2023 2:08 am
by mgdaniel
According to https://docs.python.org/3/extending/emb ... -embedding I should be able to embed a Python engine in an independent C program with

Code: Select all

#define PY_SSIZE_T_CLEAN
#include <Python.h>

int
main(int argc, char *argv[])
{
    setenv ("PYTHONDONTWRITEBYTECODE", "*", 1);
    setenv ("PYTHONHOME", "/python$root", 1);
    setenv ("PYTHONPATH", "", 1);
    
    wchar_t *program = Py_DecodeLocale(argv[0], NULL);
    if (program == NULL) {
        fprintf(stderr, "Fatal error: cannot decode argv[0]\n");
        exit(1);
    }
    Py_SetProgramName(program);  /* optional but recommended */
    Py_Initialize();
    PyRun_SimpleString("from time import time,ctime\n"
                       "print('Today is', ctime(time()))\n");
    if (Py_FinalizeEx() < 0) {
        exit(120);
    }
    PyMem_RawFree(program);
    return 0;
}
But cannot get past the get-go...

Code: Select all

HAVEN$ mcr wasd_exe:pyrte3
Python path configuration:
  PYTHONHOME = '/python$root'
  PYTHONPATH = (not set)
  program name = '$10$dkd1:[wasd_root.][ia64]pyrte3.exe;66'
  isolated = 0
  environment = 1
  user site = 1
  import site = 1
  sys._base_executable = ''
  sys.base_prefix = '/python$root'
  sys.base_exec_prefix = '/python$root'
  sys.platlibdir = 'lib'
  sys.executable = ''
  sys.prefix = '/python$root'
  sys.exec_prefix = '/python$root'
  sys.path = [
    '/python$root/lib/python310.zip',
    '/python$root/lib/python3.10',
    '/python$root/lib/python3.10/lib-dynload',
  ]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the file
system encoding
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'

Current thread 0x7b6d0ec0 (most recent call first):
  <no Python frame>
I have tried a great many permutations of PYTHON$HOME, PYTHON$PATH, script paths, whatever, without success.

Any Python specialists who can tell me how to progress this?

It was working with JFP's 3.10 BETA release a couple of years ago (and still does) but not with VSI Python 3.10 release.

Re: Embedding VSI Python 3.10 IA64

Posted: Wed Apr 12, 2023 4:27 am
by jfp
Hello Mark,

I have found this:
https://stackoverflow.com/questions/651 ... f-the-file

Not related:
I have build a LD image based on 3.10 VSI with modules vms and rdb (rename ovms and ordb) mostly compatible with the one provided with 2.7.18
One important feature added in the ovms module is support to VMS AST, a few new module(mbxqio and fileqio) added allow use QIO on mailboxes and files. This may be useful for WASD ?

If you have any interest let me known and I will provide links for download, LD image, ovms and ordb module which can be installed on windows so with vscode you will have documentaion/completion, etc... for thoses modules.

Source and examples:
https://foss.vmsgenerations.org/openvms ... vms_module

Amicalement,

JF

Re: Embedding VSI Python 3.10 IA64

Posted: Wed Apr 12, 2023 5:07 am
by imiller
Interesting to read about the AST and QIO support.
One problem is that my employers' security software won't let me get to that web site.

Website blocked
Access denied due to bad server certificate
You tried to visit:https://foss.vmsgenerations.org/openvms ... efault/pyt...

Re: Embedding VSI Python 3.10 IA64

Posted: Wed Apr 12, 2023 5:44 am
by martinv
imiller wrote: ↑
Wed Apr 12, 2023 5:07 am
 Website blocked
Access denied due to bad server certificate
The "bad server certificate" is a Let's Encrypt certificate that has expired on 3-Jan-2023.

Re: Embedding VSI Python 3.10 IA64

Posted: Wed Apr 12, 2023 9:00 am
by jfp
Correct, fixed.

JF

Re: Embedding VSI Python 3.10 IA64

Posted: Wed Apr 12, 2023 9:30 am
by mgdaniel
jfp wrote: ↑
Wed Apr 12, 2023 4:27 am
 Hello Mark,

I have found this:
https://stackoverflow.com/questions/651 ... f-the-file
8< snip 8<
Always good to see your script J-F. With the demise of the https://www.vmspython.org I thought we may have seen the last of you as well. Amicalement.

One of the many of my own hits against a search term "init_fs_encoding: failed to get the Python codec of the file
system encoding". Nothing along these lines seems to budge the issue.

Code: Select all

HAVEN$ mcr wasd_exe:pyrte3
Could not find platform independent libraries <prefix>
Could not find platform dependent libraries <exec_prefix>
Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
Python path configuration:
  PYTHONHOME = (not set)
  PYTHONPATH = (not set)
  program name = '$10$dkd1:[wasd_root.][ia64]pyrte3.exe;67'
  isolated = 0
  environment = 1
  user site = 1
  import site = 1
  sys._base_executable = ''
  sys.base_prefix = '/usr/local'
  sys.base_exec_prefix = '/usr/local'
  sys.platlibdir = 'lib'
  sys.executable = ''
  sys.prefix = '/usr/local'
  sys.exec_prefix = '/usr/local'
  sys.path = [
    '/usr/local/lib/python310.zip',
    '/usr/local/lib/python3.10',
    '/usr/local/lib/lib-dynload',
  ]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the file
system encoding
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'

Current thread 0x7b6d0ec0 (most recent call first):
  <no Python frame>
And even (getting ever more desperate) ...

Code: Select all

HAVEN$ mcr wasd_exe:pyrte3
Could not find platform independent libraries <prefix>
Could not find platform dependent libraries <exec_prefix>
Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
Python path configuration:
  PYTHONHOME = (not set)
  PYTHONPATH = (not set)
  program name = '/WASD_ROOT/ia64/pyrte3.exe'
  isolated = 0
  environment = 1
  user site = 1
  import site = 1
  sys._base_executable = '/wasd_root/ia64/pyrte3.exe'
  sys.base_prefix = '/usr/local'
  sys.base_exec_prefix = '/usr/local'
  sys.platlibdir = 'lib'
  sys.executable = '/wasd_root/ia64/pyrte3.exe'
  sys.prefix = '/usr/local'
  sys.exec_prefix = '/usr/local'
  sys.path = [
    '/usr/local/lib/python310.zip',
    '/usr/local/lib/python3.10',
    '/usr/local/lib/lib-dynload',
  ]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the file
system encoding
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'

Current thread 0x7b6d0ec0 (most recent call first):
  <no Python frame>
... and ...

Code: Select all

HAVEN$ mcr wasd_exe:pyrte3
Python path configuration:
  PYTHONHOME = '/python$root'
  PYTHONPATH = (not set)
  program name = '/wasd_root/ia64/pyrte3.exe'
  isolated = 0
  environment = 1
  user site = 1
  import site = 1
  sys._base_executable = '/wasd_root/ia64/pyrte3.exe'
  sys.base_prefix = '/python$root'
  sys.base_exec_prefix = '/python$root'
  sys.platlibdir = 'lib'
  sys.executable = '/wasd_root/ia64/pyrte3.exe'
  sys.prefix = '/python$root'
  sys.exec_prefix = '/python$root'
  sys.path = [
    '/python$root/lib/python310.zip',
    '/python$root/lib/python3.10',
    '/python$root/lib/python3.10/lib-dynload',
  ]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the file
system encoding
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'

Current thread 0x7b6d0ec0 (most recent call first):
  <no Python frame>
etc., etc., etc.

Extensive accomodations of the original program along the lines of

Code: Select all

#if PY_MAJOR_VERSION >= 3
   pInterpState = PyInterpreterState_Main ();
#else
   pInterpState = PyInterpreterState_Head ();
#endif
were and are working with your original

Code: Select all

HAVEN$ dir disk$jfppy3100i:[000000]

Directory DISK$JFPPY3100I:[000000]

000000.DIR;1                  1  24-JUN-2020 22:13
BACKUP.SYS;1                  0  24-JUN-2020 22:13
BADBLK.SYS;1                  0  24-JUN-2020 22:13
BADLOG.SYS;1                  0  24-JUN-2020 22:13
BITMAP.SYS;1                 21  24-JUN-2020 22:13
compile_all.com;58            2  30-JUL-2020 00:43
CONTIN.SYS;1                  0  24-JUN-2020 22:13
CORIMG.SYS;1                  0  24-JUN-2020 22:13
GPT.SYS;1                    96  24-JUN-2020 22:13
INDEXF.SYS;1              10085  24-JUN-2020 22:13
mercurial.DIR;1               1  14-AUG-2020 22:23
python3100.DIR;1              1  24-JUN-2020 22:14
SECURITY.SYS;1                1  24-JUN-2020 22:13
VOLSET.SYS;1                  0  24-JUN-2020 22:13

Total of 14 files, 10208 blocks.
But not with the VSI (sorta) equivalent release.

Re: Embedding VSI Python 3.10 IA64

Posted: Wed Apr 12, 2023 10:09 am
by sergey_vorfolomeev
I tried on my local machine

Code: Select all

$ cc /names=(as_is,short) /include="/python$root/include" py_call
$ link py_call,opt.opt/opt
$ run py_call
Today is Wed Apr 12 16:07:27 2023
opt file:

Code: Select all

sys$library:pthread$rtl.exe/SHARE
python$root:[lib]python$shr.exe/SHARE
GSMATCH=LEQUAL,1,0
CASE_SENSITIVE=YES
So... what problem is?

Re: Embedding VSI Python 3.10 IA64

Posted: Wed Apr 12, 2023 10:33 am
by mgdaniel
sergey_vorfolomeev wrote: ↑
Wed Apr 12, 2023 10:09 am
 I tried on my local machine

Code: Select all

$ cc /names=(as_is,short) /include="/python$root/include" py_call
$ link py_call,opt.opt/opt
$ run py_call
Today is Wed Apr 12 16:07:27 2023
opt file:

Code: Select all

sys$library:pthread$rtl.exe/SHARE
python$root:[lib]python$shr.exe/SHARE
GSMATCH=LEQUAL,1,0
CASE_SENSITIVE=YES
So... what problem is?
Hello Sergey. I modified the build LINK to explicitly include your items

Code: Select all

$ LINK /THREADS_ENABLE/EXECUTABLE=WASD_EXE:PYRTE.EXE -
[.OBJ_'ARCH_NAME']PYRTE.OBJ, -
SYS$INPUT:/OPTIONS
IDENTIFICATION="PYRTE 3.1.0"
sys$library:pthread$rtl.exe/SHARE
python$root:[lib]python$shr.exe/SHARE
GSMATCH=LEQUAL,1,0
CASE_SENSITIVE=YES
with no change to the run-time behaviour

Code: Select all

HAVEN$ mcr wasd_exe:pyrte3
Python path configuration:
  PYTHONHOME = '/python$root'
  PYTHONPATH = (not set)
  program name = '/wasd_root/ia64/pyrte3.exe'
  isolated = 0
  environment = 1
  user site = 1
  import site = 1
  sys._base_executable = '/wasd_root/ia64/pyrte3.exe'
  sys.base_prefix = '/python$root'
  sys.base_exec_prefix = '/python$root'
  sys.platlibdir = 'lib'
  sys.executable = '/wasd_root/ia64/pyrte3.exe'
  sys.prefix = '/python$root'
  sys.exec_prefix = '/python$root'
  sys.path = [
    '/python$root/lib/python310.zip',
    '/python$root/lib/python3.10',
    '/python$root/lib/python3.10/lib-dynload',
  ]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the file
system encoding
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'

Current thread 0x7b6d0ec0 (most recent call first):
  <no Python frame>
The image seems to be barfing during the Py_Initialize();.

It must something allied with the system's Python install but does seem otherwise pretty standard

Code: Select all

HAVEN$ show log python*

(LNM$PROCESS_TABLE)

(LNM$JOB_9401D800)

(WASD_TABLE)

(LNM$GROUP_000202)

(LNM$SYSTEM_TABLE)

  "PYTHON$ROOT" = "$10$DKD0:[SYS0.SYSCOMMON.PYTHON.]"
  "PYTHON$SHR" = "PYTHON$ROOT:[LIB]PYTHON$SHR.EXE"
  "PYTHONHOME" = "/python$root"
  "PYTHON_WHEELS$ROOT" = "$10$DKD0:[SYS0.SYSCOMMON.WHEELS.]"

(LNM$SYSCLUSTER_TABLE)

(DECW$LOGICAL_NAMES)
HAVEN$ product show history py*
------------------------------------ ----------- ----------- --- -----------
PRODUCT                              KIT TYPE    OPERATION   VAL DATE
------------------------------------ ----------- ----------- --- -----------
VSI I64VMS PYTHWHLS A1.1-6FIX03      Full LP     Install     Val 04-JAN-2023
VSI I64VMS PYTHON A3.10-0RELEASE001  Full LP     Install     Val 04-JAN-2023
------------------------------------ ----------- ----------- --- -----------
2 items found

Re: Embedding VSI Python 3.10 IA64

Posted: Wed Apr 12, 2023 10:43 am
by sergey_vorfolomeev
Does python work?

Code: Select all

VMSLOC> python
Python 3.10.0 (default, Apr 12 2023, 13:45:29) [C] on OpenVMS
Type "help", "copyright", "credits" or "license" for more information.
>>> exit()
VMSLOC>

Re: Embedding VSI Python 3.10 IA64

Posted: Wed Apr 12, 2023 10:56 am
by mgdaniel
sergey_vorfolomeev wrote: ↑
Wed Apr 12, 2023 10:43 am
 Does python work?

Code: Select all

VMSLOC> python
Python 3.10.0 (default, Apr 12 2023, 13:45:29) [C] on OpenVMS
Type "help", "copyright", "credits" or "license" for more information.
>>> exit()
VMSLOC>
Sure...

Code: Select all

HAVEN$ python
Python 3.10.0 (default, Nov 25 2021, 10:52:09) [C] on OpenVMS
Type "help", "copyright", "credits" or "license" for more information.
>>>

Code: Select all

HAVEN$ python ./scripts/test.py
environ({'PATH': '/wasd_root/src/python', 'HOME': '/D2/DANIELM', 'TERM': 'vt200-
80', 'USER': 'DANIELM', 'PYTHONHOME': '/python$root'})
/wasd_root/src/python/./scripts
/python$root/lib/python310.zip
/python$root/lib/python3.10
/python$root/lib/python3.10/lib-dynload
/python$root/lib/python3.10/site-packages
sys.path:  ['/wasd_root/src/python/./scripts', '/python$root/lib/python310.zip',
 '/python$root/lib/python3.10', '/python$root/lib/python3.10/lib-dynload', '/pyt
hon$root/lib/python3.10/site-packages']
sys.argv:  ['./scripts/test.py']
sys.builtin_module_names:  ('_abc', '_ast', '_codecs', '_collections', '_functoo
ls', '_imp', '_io', '_locale', '_operator', '_signal', '_sre', '_stat', '_string
', '_symtable', '_thread', '_tracemalloc', '_warnings', '_weakref', 'atexit', 'b
uiltins', 'errno', 'faulthandler', 'gc', 'itertools', 'marshal', 'posix', 'pwd',
 'sys', 'time')
All times are UTC-04:00
Page 1 of 2
Powered by phpBB® Forum Software © phpBB Limited
https://www.phpbb.com/