python performance on OpenVMS I64
-
- Contributor
- Posts: 21
- Joined: Tue Jan 10, 2023 10:41 am
- Reputation: 0
- Location: Waterloo, Ontario, Canada
- Status: Offline
- Contact:
Re: python performance on OpenVMS I64
Okay so I think I know what is happening.
If the python source file was created with edt or eve then it will be recompiled every time.
Record format: Variable length, maximum 255 bytes, longest 123 bytes
Record attributes: Carriage return carriage control
If the python source file was created with vi or vim (or copied in from a Linux system) then it will only be compiled once.
Record format: Stream_LF, maximum 0 bytes, longest 32767 bytes
Record attributes: Carriage return carriage control
Added in 8 minutes 11 seconds:
BTW, I also found a typo in library file "dtr.py"
$ set process/parse_style=extended
$ python :== $python$root:[bin]python.exe
$ set default python$root:[lib.python3^.10]
$ edit PYTHON$ROOT:[lib.python3^.10.vms]dtr.py
~~~~~~~~~~~~~~~~~~~~
change 'form' to 'from' then save - exit
~~~~~~~~~~~~~~~~~~~~
python compileall.py ./
If the python source file was created with edt or eve then it will be recompiled every time.
Record format: Variable length, maximum 255 bytes, longest 123 bytes
Record attributes: Carriage return carriage control
If the python source file was created with vi or vim (or copied in from a Linux system) then it will only be compiled once.
Record format: Stream_LF, maximum 0 bytes, longest 32767 bytes
Record attributes: Carriage return carriage control
Added in 8 minutes 11 seconds:
BTW, I also found a typo in library file "dtr.py"
$ set process/parse_style=extended
$ python :== $python$root:[bin]python.exe
$ set default python$root:[lib.python3^.10]
$ edit PYTHON$ROOT:[lib.python3^.10.vms]dtr.py
~~~~~~~~~~~~~~~~~~~~
change 'form' to 'from' then save - exit
~~~~~~~~~~~~~~~~~~~~
python compileall.py ./
-
Topic author - Master
- Posts: 157
- Joined: Fri Jun 28, 2019 8:45 am
- Reputation: 0
- Location: South Tyneside, UK
- Status: Offline
- Contact:
Re: python performance on OpenVMS I64
interesting.
Does performance then improve?
Does performance then improve?
Ian Miller
[ personal opinion only. usual disclaimers apply. Do not taunt happy fun ball ].
[ personal opinion only. usual disclaimers apply. Do not taunt happy fun ball ].
Re: python performance on OpenVMS I64
Time to ask for EDT and/or EVE to create files with Stream_LF?neilrieck wrote: ↑Wed Aug 16, 2023 11:01 amOkay so I think I know what is happening.
If the python source file was created with edt or eve then it will be recompiled every time.
Record format: Variable length, maximum 255 bytes, longest 123 bytes
Record attributes: Carriage return carriage control
If the python source file was created with vi or vim (or copied in from a Linux system) then it will only be compiled once.
Record format: Stream_LF, maximum 0 bytes, longest 32767 bytes
Record attributes: Carriage return carriage control
If copied in from a Windows system, what's the record format? I admit I only use systems where I have access to the sources and can build my own ... Being tired of using CONVERT/FDL, I built a modified version of EDT$SHR that does just that:
Code: Select all
$ edit/edt new.txt
Input file does not exist
[EOB]
*i
new
Exit
[EOB]
*exit
WORK18:[HB]NEW.TXT;1 1 line
$
$ pipe dir/full new.txt |search sys$pipe new,"record format"
NEW.TXT;1 File ID: (29491,90,0)
Record format: Stream_LF, maximum 255 bytes, longest 3 bytes
$
-
- Master
- Posts: 385
- Joined: Fri Apr 17, 2020 7:31 pm
- Reputation: 0
- Location: Rhode Island, USA
- Status: Offline
- Contact:
Re: python performance on OpenVMS I64
That is really weird.neilrieck wrote: ↑Wed Aug 16, 2023 11:01 amOkay so I think I know what is happening.
If the python source file was created with edt or eve then it will be recompiled every time.
Record format: Variable length, maximum 255 bytes, longest 123 bytes
Record attributes: Carriage return carriage control
If the python source file was created with vi or vim (or copied in from a Linux system) then it will only be compiled once.
Record format: Stream_LF, maximum 0 bytes, longest 32767 bytes
Record attributes: Carriage return carriage control
How does one write code trying to compare the last modified date of two files so that the test get impacted by the record format?
I don't think I could write such code.
Added in 1 minute 4 seconds:
It should improve startup performance - some work is going away, but it should not impact running performance.
Added in 6 minutes 35 seconds:
I suspect that depends on how it is copied. DEC/CPQ/HP/HPE/VSI TCP/IP FTP, Multinet FTP, HGFTP, old Pathworks, Samba. or inside ZIP file.
The first gives VAR.
-
- Contributor
- Posts: 21
- Joined: Tue Jan 10, 2023 10:41 am
- Reputation: 0
- Location: Waterloo, Ontario, Canada
- Status: Offline
- Contact:
Re: python performance on OpenVMS I64
I tried it several ways.
(1) created a pair of new files on Linux then used SFTP to send them to OpenVMS
(2) created a pair of new files on OpenVMS using VIM (from the dudes at polar home in Sweden)
(3) created a pair of new files on OpenVMS using EDT
I got 100% failure with item 3 and 100% success with items 1+2
The tiny speedup is only seen with my loader example since it is the "import" statement which invokes the JIT compile with file save. (you might want use something similar for use from Apache-cgi because so you can use compiled code)
There is no speedup if running like so: python something.py (which has to be JIT compiled but the bytecode is discarded when the PVM exits)
(1) created a pair of new files on Linux then used SFTP to send them to OpenVMS
(2) created a pair of new files on OpenVMS using VIM (from the dudes at polar home in Sweden)
(3) created a pair of new files on OpenVMS using EDT
I got 100% failure with item 3 and 100% success with items 1+2
The tiny speedup is only seen with my loader example since it is the "import" statement which invokes the JIT compile with file save. (you might want use something similar for use from Apache-cgi because so you can use compiled code)
There is no speedup if running like so: python something.py (which has to be JIT compiled but the bytecode is discarded when the PVM exits)
-
Topic author - Master
- Posts: 157
- Joined: Fri Jun 28, 2019 8:45 am
- Reputation: 0
- Location: South Tyneside, UK
- Status: Offline
- Contact:
Re: python performance on OpenVMS I64
the code I'm running is CPU bound. Uses 100% of one CPU and keeps growing, presumably as it reads in more data into pandas dataframe
Here are a few CONTROL-T results as it runs.
NODE::USER 17:26:21 python CPU=00:20:16.33 PF=174501 IO=112074 MEM=66928
NODE::USER 17:27:19 python CPU=00:21:13.94 PF=180589 IO=112084 MEM=73016
NODE::USER 17:28:33 python CPU=00:22:27.90 PF=187431 IO=112097 MEM=79858
NODE::USER 17:29:49 python CPU=00:23:43.71 PF=187621 IO=112110 MEM=80048
NODE::USER 17:32:55 python CPU=00:26:50.33 PF=203802 IO=112141 MEM=96229
Changing to StreamLF format for my source code does not made a difference. I note modules such as pandas have .py files in StreamLF
Here are a few CONTROL-T results as it runs.
NODE::USER 17:26:21 python CPU=00:20:16.33 PF=174501 IO=112074 MEM=66928
NODE::USER 17:27:19 python CPU=00:21:13.94 PF=180589 IO=112084 MEM=73016
NODE::USER 17:28:33 python CPU=00:22:27.90 PF=187431 IO=112097 MEM=79858
NODE::USER 17:29:49 python CPU=00:23:43.71 PF=187621 IO=112110 MEM=80048
NODE::USER 17:32:55 python CPU=00:26:50.33 PF=203802 IO=112141 MEM=96229
Changing to StreamLF format for my source code does not made a difference. I note modules such as pandas have .py files in StreamLF
Last edited by imiller on Thu Aug 17, 2023 12:41 pm, edited 1 time in total.
Ian Miller
[ personal opinion only. usual disclaimers apply. Do not taunt happy fun ball ].
[ personal opinion only. usual disclaimers apply. Do not taunt happy fun ball ].
Re: python performance on OpenVMS I64
I keep an FDL file in my syslogin just to simplify converting VAR files to stream LF format:neilrieck wrote: ↑Thu Aug 17, 2023 12:16 pmI tried it several ways.
(1) created a pair of new files on Linux then used SFTP to send them to OpenVMS
(2) created a pair of new files on OpenVMS using VIM (from the dudes at polar home in Sweden)
(3) created a pair of new files on OpenVMS using EDT
I got 100% failure with item 3 and 100% success with items 1+2
Code: Select all
$ type sys$login:stmlf.fdl
IDENT FDL_VERSION 02 "18-MAR-2016 15:42:49 OpenVMS ANALYZE/RMS_FILE Utility"
SYSTEM
SOURCE OpenVMS
FILE
BEST_TRY_CONTIGUOUS no
CONTIGUOUS no
EXTENSION 0
FILE_MONITORING no
ORGANIZATION sequential
GLOBAL_BUFFER_COUNT 0
GLBUFF_CNT_V83 0
GLBUFF_FLAGS_V83 none
RECORD
BLOCK_SPAN yes
CARRIAGE_CONTROL carriage_return
FORMAT stream_lf
SIZE 0
Last edited by jonesd on Thu Aug 17, 2023 12:47 pm, edited 1 time in total.
Re: python performance on OpenVMS I64
Code: Select all
> Time to ask for EDT and/or EVE to create files with Stream_LF?
I suspect that a thorough search would find abundant requests of that
nature over the years.
Count me in on a /RFM = { STM | STMCR | STMLF | VAR | <others?> }
(or similar) qualifier on every VSI-supplied editor there is. Nor would
I complain about some logical name which would change the default record
format (on a new file). (Of course, VIM doesn't need the help.)
> TPU and EDT will write out a stream-LF file if the input file was
> stream-LF.
Sure, but:
edit /tpu <file>
is handier than, say:
edit /tpu STMLF /output = <file>
where STMLF is a logical name which points to an empty template file
with the desired attribute(s).
-
- Master
- Posts: 136
- Joined: Sat Aug 15, 2020 9:00 am
- Reputation: 0
- Location: Cambridge, UK
- Status: Offline
Re: python performance on OpenVMS I64
I would rather that opensource prorams cou;d honour the native text file format
Chris
Chris
--
Chris
Chris
-
- Master
- Posts: 385
- Joined: Fri Apr 17, 2020 7:31 pm
- Reputation: 0
- Location: Rhode Island, USA
- Status: Offline
- Contact:
Re: python performance on OpenVMS I64
You don't really need the file nowadays.jonesd wrote: ↑Thu Aug 17, 2023 12:47 pmI keep an FDL file in my syslogin just to simplify converting VAR files to stream LF format:Code: Select all
$ type sys$login:stmlf.fdl ... RECORD ... FORMAT stream_lf ...
$ rfmvar :== convert/fdl=nla0:
$ rfmstmlf :== convert/fdl="""record; format stream_lf"""