Embedding VSI Python 3.10 IA64


Topic author
mgdaniel
Valued Contributor
Posts: 62
Joined: Mon Feb 28, 2022 5:16 pm
Reputation: 0
Location: Adelaide, South Australia
Status: Offline
Contact:

Embedding VSI Python 3.10 IA64

Post by mgdaniel » Mon Apr 10, 2023 2:08 am

According to https://docs.python.org/3/extending/emb ... -embedding I should be able to embed a Python engine in an independent C program with

Code: Select all

#define PY_SSIZE_T_CLEAN
#include <Python.h>

int
main(int argc, char *argv[])
{
    setenv ("PYTHONDONTWRITEBYTECODE", "*", 1);
    setenv ("PYTHONHOME", "/python$root", 1);
    setenv ("PYTHONPATH", "", 1);
    
    wchar_t *program = Py_DecodeLocale(argv[0], NULL);
    if (program == NULL) {
        fprintf(stderr, "Fatal error: cannot decode argv[0]\n");
        exit(1);
    }
    Py_SetProgramName(program);  /* optional but recommended */
    Py_Initialize();
    PyRun_SimpleString("from time import time,ctime\n"
                       "print('Today is', ctime(time()))\n");
    if (Py_FinalizeEx() < 0) {
        exit(120);
    }
    PyMem_RawFree(program);
    return 0;
}
But cannot get past the get-go...

Code: Select all

HAVEN$ mcr wasd_exe:pyrte3
Python path configuration:
  PYTHONHOME = '/python$root'
  PYTHONPATH = (not set)
  program name = '$10$dkd1:[wasd_root.][ia64]pyrte3.exe;66'
  isolated = 0
  environment = 1
  user site = 1
  import site = 1
  sys._base_executable = ''
  sys.base_prefix = '/python$root'
  sys.base_exec_prefix = '/python$root'
  sys.platlibdir = 'lib'
  sys.executable = ''
  sys.prefix = '/python$root'
  sys.exec_prefix = '/python$root'
  sys.path = [
    '/python$root/lib/python310.zip',
    '/python$root/lib/python3.10',
    '/python$root/lib/python3.10/lib-dynload',
  ]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the file
system encoding
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'

Current thread 0x7b6d0ec0 (most recent call first):
  <no Python frame>
I have tried a great many permutations of PYTHON$HOME, PYTHON$PATH, script paths, whatever, without success.

Any Python specialists who can tell me how to progress this?

It was working with JFP's 3.10 BETA release a couple of years ago (and still does) but not with VSI Python 3.10 release.


jfp
Contributor
Posts: 16
Joined: Wed Jun 17, 2020 6:34 am
Reputation: 0
Status: Offline

Re: Embedding VSI Python 3.10 IA64

Post by jfp » Wed Apr 12, 2023 4:27 am

Hello Mark,

I have found this:
https://stackoverflow.com/questions/651 ... f-the-file

Not related:
I have build a LD image based on 3.10 VSI with modules vms and rdb (rename ovms and ordb) mostly compatible with the one provided with 2.7.18
One important feature added in the ovms module is support to VMS AST, a few new module(mbxqio and fileqio) added allow use QIO on mailboxes and files. This may be useful for WASD ?

If you have any interest let me known and I will provide links for download, LD image, ovms and ordb module which can be installed on windows so with vscode you will have documentaion/completion, etc... for thoses modules.

Source and examples:
https://foss.vmsgenerations.org/openvms ... vms_module

Amicalement,

JF

User avatar

imiller
Master
Posts: 136
Joined: Fri Jun 28, 2019 8:45 am
Reputation: 0
Location: South Tyneside, UK
Status: Offline
Contact:

Re: Embedding VSI Python 3.10 IA64

Post by imiller » Wed Apr 12, 2023 5:07 am

Interesting to read about the AST and QIO support.
One problem is that my employers' security software won't let me get to that web site.

Website blocked
Access denied due to bad server certificate
You tried to visit:https://foss.vmsgenerations.org/openvms ... efault/pyt...
Ian Miller
[ personal opinion only. usual disclaimers apply. Do not taunt happy fun ball ].

User avatar

martinv
Master
Posts: 102
Joined: Fri Jun 14, 2019 11:05 pm
Reputation: 0
Location: Goslar, Germany
Status: Offline
Contact:

Re: Embedding VSI Python 3.10 IA64

Post by martinv » Wed Apr 12, 2023 5:44 am

imiller wrote:
Wed Apr 12, 2023 5:07 am
Website blocked
Access denied due to bad server certificate
The "bad server certificate" is a Let's Encrypt certificate that has expired on 3-Jan-2023.
Working hard for something we don't care about is called stress;
working hard for something we love is called passion.
(Simon Sinek)


jfp
Contributor
Posts: 16
Joined: Wed Jun 17, 2020 6:34 am
Reputation: 0
Status: Offline

Re: Embedding VSI Python 3.10 IA64

Post by jfp » Wed Apr 12, 2023 9:00 am

Correct, fixed.

JF
Last edited by jfp on Wed Apr 12, 2023 9:07 am, edited 1 time in total.


Topic author
mgdaniel
Valued Contributor
Posts: 62
Joined: Mon Feb 28, 2022 5:16 pm
Reputation: 0
Location: Adelaide, South Australia
Status: Offline
Contact:

Re: Embedding VSI Python 3.10 IA64

Post by mgdaniel » Wed Apr 12, 2023 9:30 am

jfp wrote:
Wed Apr 12, 2023 4:27 am
Hello Mark,

I have found this:
https://stackoverflow.com/questions/651 ... f-the-file
8< snip 8<
Always good to see your script J-F. With the demise of the https://www.vmspython.org I thought we may have seen the last of you as well. Amicalement.

One of the many of my own hits against a search term "init_fs_encoding: failed to get the Python codec of the file
system encoding
". Nothing along these lines seems to budge the issue.

Code: Select all

HAVEN$ mcr wasd_exe:pyrte3
Could not find platform independent libraries <prefix>
Could not find platform dependent libraries <exec_prefix>
Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
Python path configuration:
  PYTHONHOME = (not set)
  PYTHONPATH = (not set)
  program name = '$10$dkd1:[wasd_root.][ia64]pyrte3.exe;67'
  isolated = 0
  environment = 1
  user site = 1
  import site = 1
  sys._base_executable = ''
  sys.base_prefix = '/usr/local'
  sys.base_exec_prefix = '/usr/local'
  sys.platlibdir = 'lib'
  sys.executable = ''
  sys.prefix = '/usr/local'
  sys.exec_prefix = '/usr/local'
  sys.path = [
    '/usr/local/lib/python310.zip',
    '/usr/local/lib/python3.10',
    '/usr/local/lib/lib-dynload',
  ]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the file
system encoding
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'

Current thread 0x7b6d0ec0 (most recent call first):
  <no Python frame>
And even (getting ever more desperate) ...

Code: Select all

HAVEN$ mcr wasd_exe:pyrte3
Could not find platform independent libraries <prefix>
Could not find platform dependent libraries <exec_prefix>
Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
Python path configuration:
  PYTHONHOME = (not set)
  PYTHONPATH = (not set)
  program name = '/WASD_ROOT/ia64/pyrte3.exe'
  isolated = 0
  environment = 1
  user site = 1
  import site = 1
  sys._base_executable = '/wasd_root/ia64/pyrte3.exe'
  sys.base_prefix = '/usr/local'
  sys.base_exec_prefix = '/usr/local'
  sys.platlibdir = 'lib'
  sys.executable = '/wasd_root/ia64/pyrte3.exe'
  sys.prefix = '/usr/local'
  sys.exec_prefix = '/usr/local'
  sys.path = [
    '/usr/local/lib/python310.zip',
    '/usr/local/lib/python3.10',
    '/usr/local/lib/lib-dynload',
  ]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the file
system encoding
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'

Current thread 0x7b6d0ec0 (most recent call first):
  <no Python frame>
... and ...

Code: Select all

HAVEN$ mcr wasd_exe:pyrte3
Python path configuration:
  PYTHONHOME = '/python$root'
  PYTHONPATH = (not set)
  program name = '/wasd_root/ia64/pyrte3.exe'
  isolated = 0
  environment = 1
  user site = 1
  import site = 1
  sys._base_executable = '/wasd_root/ia64/pyrte3.exe'
  sys.base_prefix = '/python$root'
  sys.base_exec_prefix = '/python$root'
  sys.platlibdir = 'lib'
  sys.executable = '/wasd_root/ia64/pyrte3.exe'
  sys.prefix = '/python$root'
  sys.exec_prefix = '/python$root'
  sys.path = [
    '/python$root/lib/python310.zip',
    '/python$root/lib/python3.10',
    '/python$root/lib/python3.10/lib-dynload',
  ]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the file
system encoding
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'

Current thread 0x7b6d0ec0 (most recent call first):
  <no Python frame>
etc., etc., etc.

Extensive accomodations of the original program along the lines of

Code: Select all

#if PY_MAJOR_VERSION >= 3
   pInterpState = PyInterpreterState_Main ();
#else
   pInterpState = PyInterpreterState_Head ();
#endif
were and are working with your original

Code: Select all

HAVEN$ dir disk$jfppy3100i:[000000]

Directory DISK$JFPPY3100I:[000000]

000000.DIR;1                  1  24-JUN-2020 22:13
BACKUP.SYS;1                  0  24-JUN-2020 22:13
BADBLK.SYS;1                  0  24-JUN-2020 22:13
BADLOG.SYS;1                  0  24-JUN-2020 22:13
BITMAP.SYS;1                 21  24-JUN-2020 22:13
compile_all.com;58            2  30-JUL-2020 00:43
CONTIN.SYS;1                  0  24-JUN-2020 22:13
CORIMG.SYS;1                  0  24-JUN-2020 22:13
GPT.SYS;1                    96  24-JUN-2020 22:13
INDEXF.SYS;1              10085  24-JUN-2020 22:13
mercurial.DIR;1               1  14-AUG-2020 22:23
python3100.DIR;1              1  24-JUN-2020 22:14
SECURITY.SYS;1                1  24-JUN-2020 22:13
VOLSET.SYS;1                  0  24-JUN-2020 22:13

Total of 14 files, 10208 blocks.
But not with the VSI (sorta) equivalent release.


sergey_vorfolomeev
VSI Expert
Valued Contributor
Posts: 99
Joined: Thu Aug 22, 2019 12:17 am
Reputation: 0
Status: Offline

Re: Embedding VSI Python 3.10 IA64

Post by sergey_vorfolomeev » Wed Apr 12, 2023 10:09 am

I tried on my local machine

Code: Select all

$ cc /names=(as_is,short) /include="/python$root/include" py_call
$ link py_call,opt.opt/opt
$ run py_call
Today is Wed Apr 12 16:07:27 2023
opt file:

Code: Select all

sys$library:pthread$rtl.exe/SHARE
python$root:[lib]python$shr.exe/SHARE
GSMATCH=LEQUAL,1,0
CASE_SENSITIVE=YES
So... what problem is?
Last edited by sergey_vorfolomeev on Wed Apr 12, 2023 10:10 am, edited 1 time in total.


Topic author
mgdaniel
Valued Contributor
Posts: 62
Joined: Mon Feb 28, 2022 5:16 pm
Reputation: 0
Location: Adelaide, South Australia
Status: Offline
Contact:

Re: Embedding VSI Python 3.10 IA64

Post by mgdaniel » Wed Apr 12, 2023 10:33 am

sergey_vorfolomeev wrote:
Wed Apr 12, 2023 10:09 am
I tried on my local machine

Code: Select all

$ cc /names=(as_is,short) /include="/python$root/include" py_call
$ link py_call,opt.opt/opt
$ run py_call
Today is Wed Apr 12 16:07:27 2023
opt file:

Code: Select all

sys$library:pthread$rtl.exe/SHARE
python$root:[lib]python$shr.exe/SHARE
GSMATCH=LEQUAL,1,0
CASE_SENSITIVE=YES
So... what problem is?
Hello Sergey. I modified the build LINK to explicitly include your items

Code: Select all

$ LINK /THREADS_ENABLE/EXECUTABLE=WASD_EXE:PYRTE.EXE -
[.OBJ_'ARCH_NAME']PYRTE.OBJ, -
SYS$INPUT:/OPTIONS
IDENTIFICATION="PYRTE 3.1.0"
sys$library:pthread$rtl.exe/SHARE
python$root:[lib]python$shr.exe/SHARE
GSMATCH=LEQUAL,1,0
CASE_SENSITIVE=YES
with no change to the run-time behaviour

Code: Select all

HAVEN$ mcr wasd_exe:pyrte3
Python path configuration:
  PYTHONHOME = '/python$root'
  PYTHONPATH = (not set)
  program name = '/wasd_root/ia64/pyrte3.exe'
  isolated = 0
  environment = 1
  user site = 1
  import site = 1
  sys._base_executable = '/wasd_root/ia64/pyrte3.exe'
  sys.base_prefix = '/python$root'
  sys.base_exec_prefix = '/python$root'
  sys.platlibdir = 'lib'
  sys.executable = '/wasd_root/ia64/pyrte3.exe'
  sys.prefix = '/python$root'
  sys.exec_prefix = '/python$root'
  sys.path = [
    '/python$root/lib/python310.zip',
    '/python$root/lib/python3.10',
    '/python$root/lib/python3.10/lib-dynload',
  ]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the file
system encoding
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'

Current thread 0x7b6d0ec0 (most recent call first):
  <no Python frame>
The image seems to be barfing during the Py_Initialize();.

It must something allied with the system's Python install but does seem otherwise pretty standard

Code: Select all

HAVEN$ show log python*

(LNM$PROCESS_TABLE)

(LNM$JOB_9401D800)

(WASD_TABLE)

(LNM$GROUP_000202)

(LNM$SYSTEM_TABLE)

  "PYTHON$ROOT" = "$10$DKD0:[SYS0.SYSCOMMON.PYTHON.]"
  "PYTHON$SHR" = "PYTHON$ROOT:[LIB]PYTHON$SHR.EXE"
  "PYTHONHOME" = "/python$root"
  "PYTHON_WHEELS$ROOT" = "$10$DKD0:[SYS0.SYSCOMMON.WHEELS.]"

(LNM$SYSCLUSTER_TABLE)

(DECW$LOGICAL_NAMES)
HAVEN$ product show history py*
------------------------------------ ----------- ----------- --- -----------
PRODUCT                              KIT TYPE    OPERATION   VAL DATE
------------------------------------ ----------- ----------- --- -----------
VSI I64VMS PYTHWHLS A1.1-6FIX03      Full LP     Install     Val 04-JAN-2023
VSI I64VMS PYTHON A3.10-0RELEASE001  Full LP     Install     Val 04-JAN-2023
------------------------------------ ----------- ----------- --- -----------
2 items found


sergey_vorfolomeev
VSI Expert
Valued Contributor
Posts: 99
Joined: Thu Aug 22, 2019 12:17 am
Reputation: 0
Status: Offline

Re: Embedding VSI Python 3.10 IA64

Post by sergey_vorfolomeev » Wed Apr 12, 2023 10:43 am

Does python work?

Code: Select all

VMSLOC> python
Python 3.10.0 (default, Apr 12 2023, 13:45:29) [C] on OpenVMS
Type "help", "copyright", "credits" or "license" for more information.
>>> exit()
VMSLOC>


Topic author
mgdaniel
Valued Contributor
Posts: 62
Joined: Mon Feb 28, 2022 5:16 pm
Reputation: 0
Location: Adelaide, South Australia
Status: Offline
Contact:

Re: Embedding VSI Python 3.10 IA64

Post by mgdaniel » Wed Apr 12, 2023 10:56 am

sergey_vorfolomeev wrote:
Wed Apr 12, 2023 10:43 am
Does python work?

Code: Select all

VMSLOC> python
Python 3.10.0 (default, Apr 12 2023, 13:45:29) [C] on OpenVMS
Type "help", "copyright", "credits" or "license" for more information.
>>> exit()
VMSLOC>
Sure...

Code: Select all

HAVEN$ python
Python 3.10.0 (default, Nov 25 2021, 10:52:09) [C] on OpenVMS
Type "help", "copyright", "credits" or "license" for more information.
>>>

Code: Select all

HAVEN$ python ./scripts/test.py
environ({'PATH': '/wasd_root/src/python', 'HOME': '/D2/DANIELM', 'TERM': 'vt200-
80', 'USER': 'DANIELM', 'PYTHONHOME': '/python$root'})
/wasd_root/src/python/./scripts
/python$root/lib/python310.zip
/python$root/lib/python3.10
/python$root/lib/python3.10/lib-dynload
/python$root/lib/python3.10/site-packages
sys.path:  ['/wasd_root/src/python/./scripts', '/python$root/lib/python310.zip',
 '/python$root/lib/python3.10', '/python$root/lib/python3.10/lib-dynload', '/pyt
hon$root/lib/python3.10/site-packages']
sys.argv:  ['./scripts/test.py']
sys.builtin_module_names:  ('_abc', '_ast', '_codecs', '_collections', '_functoo
ls', '_imp', '_io', '_locale', '_operator', '_signal', '_sre', '_stat', '_string
', '_symtable', '_thread', '_tracemalloc', '_warnings', '_weakref', 'atexit', 'b
uiltins', 'errno', 'faulthandler', 'gc', 'itertools', 'marshal', 'posix', 'pwd',
 'sys', 'time')

Post Reply