python performance on OpenVMS I64

[neilrieck]

Okay so I think I know what is happening.

If the python source file was created with edt or eve then it will be recompiled every time.

Record format: Variable length, maximum 255 bytes, longest 123 bytes
Record attributes: Carriage return carriage control

If the python source file was created with vi or vim (or copied in from a Linux system) then it will only be compiled once.

Record format: Stream_LF, maximum 0 bytes, longest 32767 bytes
Record attributes: Carriage return carriage control

Added in 8 minutes 11 seconds:
BTW, I also found a typo in library file "dtr.py"

$ set process/parse_style=extended
$ python :== $python$root:[bin]python.exe
$ set default python$root:[lib.python3^.10]
$ edit PYTHON$ROOT:[lib.python3^.10.vms]dtr.py
~~~~~~~~~~~~~~~~~~~~
change 'form' to 'from' then save - exit
~~~~~~~~~~~~~~~~~~~~
python compileall.py ./

[imiller]

interesting.

Does performance then improve?

[hb]

Okay so I think I know what is happening.

If the python source file was created with edt or eve then it will be recompiled every time.

Record format: Variable length, maximum 255 bytes, longest 123 bytes
Record attributes: Carriage return carriage control

If the python source file was created with vi or vim (or copied in from a Linux system) then it will only be compiled once.

Record format: Stream_LF, maximum 0 bytes, longest 32767 bytes
Record attributes: Carriage return carriage control

Time to ask for EDT and/or EVE to create files with Stream_LF?

If copied in from a Windows system, what's the record format? I admit I only use systems where I have access to the sources and can build my own ... Being tired of using CONVERT/FDL, I built a modified version of EDT$SHR that does just that:

Code: Select all

$ edit/edt new.txt
Input file does not exist
[EOB]
*i
            new
             Exit 
[EOB]
*exit
WORK18:[HB]NEW.TXT;1 1 line
$
$ pipe dir/full new.txt |search sys$pipe new,"record format"
NEW.TXT;1                     File ID:  (29491,90,0)          
Record format:      Stream_LF, maximum 255 bytes, longest 3 bytes
$

And uses the Alternate Screen feature of xterm.

[arne_v]

Okay so I think I know what is happening.

If the python source file was created with edt or eve then it will be recompiled every time.

Record format: Variable length, maximum 255 bytes, longest 123 bytes
Record attributes: Carriage return carriage control

If the python source file was created with vi or vim (or copied in from a Linux system) then it will only be compiled once.

Record format: Stream_LF, maximum 0 bytes, longest 32767 bytes
Record attributes: Carriage return carriage control

That is really weird.

How does one write code trying to compare the last modified date of two files so that the test get impacted by the record format?

I don't think I could write such code.

Added in 1 minute 4 seconds:

interesting.

Does performance then improve?

It should improve startup performance - some work is going away, but it should not impact running performance.

Added in 6 minutes 35 seconds:

If copied in from a Windows system, what's the record format?

I suspect that depends on how it is copied. DEC/CPQ/HP/HPE/VSI TCP/IP FTP, Multinet FTP, HGFTP, old Pathworks, Samba. or inside ZIP file.

The first gives VAR.

[neilrieck]

I tried it several ways.

(1) created a pair of new files on Linux then used SFTP to send them to OpenVMS
(2) created a pair of new files on OpenVMS using VIM (from the dudes at polar home in Sweden)
(3) created a pair of new files on OpenVMS using EDT

I got 100% failure with item 3 and 100% success with items 1+2

The tiny speedup is only seen with my loader example since it is the "import" statement which invokes the JIT compile with file save. (you might want use something similar for use from Apache-cgi because so you can use compiled code)

There is no speedup if running like so: python something.py (which has to be JIT compiled but the bytecode is discarded when the PVM exits)

[imiller]

the code I'm running is CPU bound. Uses 100% of one CPU and keeps growing, presumably as it reads in more data into pandas dataframe

Here are a few CONTROL-T results as it runs.

NODE::USER 17:26:21 python CPU=00:20:16.33 PF=174501 IO=112074 MEM=66928
NODE::USER 17:27:19 python CPU=00:21:13.94 PF=180589 IO=112084 MEM=73016
NODE::USER 17:28:33 python CPU=00:22:27.90 PF=187431 IO=112097 MEM=79858
NODE::USER 17:29:49 python CPU=00:23:43.71 PF=187621 IO=112110 MEM=80048
NODE::USER 17:32:55 python CPU=00:26:50.33 PF=203802 IO=112141 MEM=96229

Changing to StreamLF format for my source code does not made a difference. I note modules such as pandas have .py files in StreamLF

[jonesd]

I tried it several ways.

(1) created a pair of new files on Linux then used SFTP to send them to OpenVMS
(2) created a pair of new files on OpenVMS using VIM (from the dudes at polar home in Sweden)
(3) created a pair of new files on OpenVMS using EDT

I got 100% failure with item 3 and 100% success with items 1+2

I keep an FDL file in my syslogin just to simplify converting VAR files to stream LF format:

Code: Select all

$ type sys$login:stmlf.fdl
IDENT   FDL_VERSION 02 "18-MAR-2016 15:42:49   OpenVMS ANALYZE/RMS_FILE Utility"

SYSTEM
        SOURCE                  OpenVMS

FILE
        BEST_TRY_CONTIGUOUS     no
        CONTIGUOUS              no
        EXTENSION               0
        FILE_MONITORING         no
        ORGANIZATION            sequential
        GLOBAL_BUFFER_COUNT     0
        GLBUFF_CNT_V83          0
        GLBUFF_FLAGS_V83        none

RECORD
        BLOCK_SPAN              yes
        CARRIAGE_CONTROL        carriage_return
        FORMAT                  stream_lf
        SIZE                    0

TPU and EDT will write out a stream-LF file if the input file was stream-LF.

[sms]

Code: Select all

> Time to ask for EDT and/or EVE to create files with Stream_LF?

   I suspect that a thorough search would find abundant requests of that
nature over the years.

   Count me in on a /RFM = { STM | STMCR | STMLF | VAR | <others?> }
(or similar) qualifier on every VSI-supplied editor there is.  Nor would
I complain about some logical name which would change the default record
format (on a new file).  (Of course, VIM doesn't need the help.)


> TPU and EDT will write out a stream-LF file if the input file was
> stream-LF.

   Sure, but:
      edit /tpu <file>
is handier than, say:
      edit /tpu STMLF /output = <file>
where STMLF is a logical name which points to an empty template file
with the desired attribute(s).

[cct]

I would rather that opensource prorams cou;d honour the native text file format

Chris

[arne_v]

I keep an FDL file in my syslogin just to simplify converting VAR files to stream LF format:

Code: Select all

$ type sys$login:stmlf.fdl
...
RECORD
...
        FORMAT                  stream_lf
...

You don't really need the file nowadays.

$ rfmvar :== convert/fdl=nla0:
$ rfmstmlf :== convert/fdl="""record; format stream_lf"""