Upload failed (cmdTimeout helps a little) ?

Here you can discuss the universal Integrated Development Environment for OpenVMS.

Topic author
l.cedric
Valued Contributor
Posts: 51
Joined: Thu Jul 18, 2019 8:18 am
Reputation: 0
Status: Offline

Upload failed (cmdTimeout helps a little) ?

Post by l.cedric » Wed Apr 13, 2022 8:52 am

Hello,

since few versions I have this issue :

Code: Select all

…  
   inflating: [.src.server.dmtr]db_serializerlistbusiness.pc  
Upload failed. [my_project]
:(

Note that our source code lies in few sub-folders

As our code base is made of ~14 000 files, the Upload lasts about 3 minutes ... So I tried to play with the cmdTimeout parameter and here's my observations :
  • When my OpenVMS destination folder is empty or hosts 1 succeded upload (so ~ 14 000 files), a 3600 value for cmdTimeout allows a succeded Upload
  • If I decrease this value (1800 for example), it fails in a previous file
  • Perhaps a clue ?
    • I noticed that the "purge": true parameter (in vmssoftware.synchronizer-settings.json) does not purge my source files on this OpenVMS destination folder ... So 2 tries give ~ 28 000 files in this folder
    • And the next Uploads fail, except if I purge by myself this OpenVMS folder :!: Which is in fact my workaround
  • Never have an issue with the Quick Upload (very few files in comparison ...)

Despite your Wiki's page which says :
NOTE: Value "0" means, that timeout is not used.
WARN: Do not change timeout settings unless necessary.
If needed, here's some informations :
  • I've just tried today with the new VMS-IDE v1.5.52 (VSCode v1.66.1)
  • SSD and a VPN towards my company thru fiber internet

Is there any new configuration I missed ?
Thanks in advance

User avatar

puder
VSI Expert
Contributor
Posts: 10
Joined: Thu Aug 29, 2019 1:44 pm
Reputation: 0
Status: Offline

Re: Upload failed (cmdTimeout helps a little) ?

Post by puder » Wed Apr 13, 2022 11:16 am

What version of TCPIP are you using?


Topic author
l.cedric
Valued Contributor
Posts: 51
Joined: Thu Jul 18, 2019 8:18 am
Reputation: 0
Status: Offline

Re: Upload failed (cmdTimeout helps a little) ?

Post by l.cedric » Wed Apr 13, 2022 11:23 am

$ prod sho prod *tcp*
------------------------------------ ----------- ---------
PRODUCT KIT TYPE STATE
------------------------------------ ----------- ---------
HP I64VMS TCPIP V5.7-13ECO5 Full LP Installed
------------------------------------ ----------- ---------

1 item found


sms
Master
Posts: 310
Joined: Fri Aug 21, 2020 5:18 pm
Reputation: 0
Status: Offline

Re: Upload failed (cmdTimeout helps a little) ?

Post by sms » Wed Apr 13, 2022 12:56 pm

Code: Select all

> $ prod sho prod *tcp*

   For more/better info:

      tcpip show version
      product show product /full tcpip*

User avatar

arne_v
Master
Posts: 299
Joined: Fri Apr 17, 2020 7:31 pm
Reputation: 0
Location: Rhode Island, USA
Status: Offline
Contact:

Re: Upload failed (cmdTimeout helps a little) ?

Post by arne_v » Wed Apr 13, 2022 2:48 pm

It does not solve the VS Code problem, but I think you would be better off restructuring that code in multiple projects.

If you instead of 1 project with 14000 files had 10 projects at an average of 1400 files, then thing would become more manageable.

The 9 projects would produce either an object library or a shareable image. The last project would produce an executable image. Or whatever works.
Arne
arne@vajhoej.dk
VMS user since 1986


sergey_vorfolomeev
VSI Expert
Valued Contributor
Posts: 98
Joined: Thu Aug 22, 2019 12:17 am
Reputation: 0
Status: Offline

Re: Upload failed (cmdTimeout helps a little) ?

Post by sergey_vorfolomeev » Thu Apr 14, 2022 12:45 am

the "purge" command is issued only when "upload" is succeeded.
may be the disk is full? or amount of files on the disk exceed "Maximum files allowed" value?

Added in 35 minutes 48 seconds:
test uploading a huge project. ok:

Code: Select all

$ dir [.llvm_src...] /grand
Grand total of 11791 directories, 111411 files.

Code: Select all

Upload completed successfully. [llvm-10.0.1]


Topic author
l.cedric
Valued Contributor
Posts: 51
Joined: Thu Jul 18, 2019 8:18 am
Reputation: 0
Status: Offline

Re: Upload failed (cmdTimeout helps a little) ?

Post by l.cedric » Thu Apr 14, 2022 3:22 am

@SMS
$ tcpip show version
HP TCP/IP Services for OpenVMS Industry Standard 64 Version V5.7 - ECO 5
on an HP rx2660 (1.67GHz/9.0MB) running OpenVMS V8.4

$ product show product /full tcpip*
--------------------- ------ --------- --------- ------------------------------------ ------------
PRODUCT KIT TYPE STATE MAINTENANCE REFERENCED BY
--------------------- ------ --------- --------- ------------------------------------ ------------
HP I64VMS TCPIP V5.7-13ECO5 Full LP Installed HP I64VMS TELNET_PAT V5.7-13ECO5A HP I64VMS OPENVMS V8.4
------------------------- ----------- --------- ------------------------------------ ------------
1 item found
VMSINSTAL history file DISK$SYSTEM:[VMS$COMMON.][SYSUPD]VMSINSTAL.HISTORY;1 contains additional information


@ARNE_V
I completly agree with you, but it's a ~30 years old ... information system, not well structured
=> monolithic architecture, which cannot be easily restructured.


@SERGEY
the "purge" command is issued only when "upload" is succeeded
As I said, even when the download seems successful, the purge of the OpenVMS folder is not actually done (= several versions of the same file remain). For examples after 2 successful Upload :

Code: Select all

$ dir [.src.server.sp]/col=1
Directory DSA9:[users.****.mantis.src.server.sp]
...
sp_mmsgen.mms;2
sp_mmsgen.mms;1
sp_mmsgen.txt;2
sp_mmsgen.txt;1

Total of 36 files.
Is there a way to enable more detailed output logging in VSCode ?

may be the disk is full? or amount of files on the disk exceed "Maximum files allowed" value?

Code: Select all

$ dir DSA9:[000000...]/grand
Grand total of 2868 directories, 426786 files.

$ sh dev /full dsa9
Disk DSA9:, device type Generic SCSI disk, is online, mounted, file-oriented
    device, shareable, available to cluster, shadow set virtual unit, error
    logging is enabled, device supports bitmaps (no bitmaps active).

    Error count                    0    Operations completed           17142233
    Owner process                 ""    Owner UIC                      [****]
    Owner process ID        00000000    Dev Prot            S:RWPL,O:RWPL,G:R,W
    Reference count               24    Default buffer size                 512
    Total blocks           314572800    Sectors per track                   128
    Total cylinders            19200    Tracks per cylinder                 128
    Logical Volume Size    314572800    Expansion Size Limit          315555840

    Volume label               "GIT"    Relative volume number                0
    Cluster size                  16    Transaction count                    22
    
========>    Free blocks            178135328    Maximum files allowed           9252141       <========
    
    Extend quantity                5    Mount count                           1
    Mount status              System    Cache name             "_DSA1:XQPCACHE"
    Extent cache size             64    Max blocks in extent cache     17813532
    File ID cache size            64    Blocks in extent cache           128144
    Quota cache size               0    Maximum buffers in FCP cache       5140
    Volume owner UIC  [****,****]
                                        Vol Prot    S:RWCD,O:RWCD,G:RWCD,W:RWCD

  Volume Status:  ODS-5, subject to mount verification, write-through XFC
      caching enabled, write-back XQP caching enabled, special files enabled.

Disk $1$DGA28:, device type 3PARdata VV, is online, device has multiple I/O
    paths, member of shadow set DSA9:, error logging is enabled.

    Error count                    0    Shadow member operation count  12533526
    Current preferred CPU Id       1    Fastpath                              1
    WWID   01000010:6000-2AC0-0000-0000-0000-001C-0001-F8D5
    Allocation class               1

  I/O paths to device              3

  Path FGA0.2101-0002-AC01-F8D5 (MANCED), primary
    Error count                    0    Operations completed                 93
    Last switched to time:   Never                     Count                  0
    Last switched from time: 10-APR-2022 12:12:11.76

  Path FGB0.2021-0002-AC01-F8D5 (MANCED), current
    Error count                    0    Operations completed           12533340
    Last switched to time:   10-APR-2022 12:12:11.76   Count                  1
    Last switched from time: Never

  Path FGB0.2121-0002-AC01-F8D5 (MANCED)
    Error count                    0    Operations completed                 93
    Last switched to time:   Never                     Count                  0
    Last switched from time: Never

Disk $1$DGA424:, device type HSV300, is online, device has multiple I/O paths,
    member of shadow set DSA9:, error logging is enabled.

    Error count                    0    Shadow member operation count  12487753
    Current preferred CPU Id       1    Fastpath                              1
    WWID   01000010:6005-08B4-0009-979C-0000-C000-0456-0000
    Allocation class               1

  I/O paths to device              2

  Path FGA0.5001-4380-025B-53DC (MANCED), primary, current
    Error count                    0    Operations completed           12482076
    Last switched to time:   10-APR-2022 12:12:53.75   Count                  1
    Last switched from time: 10-APR-2022 12:12:10.76

  Path FGA0.5001-4380-025B-53D8 (MANCED)
    Error count                    0    Operations completed               5677
    Last switched to time:   10-APR-2022 12:12:10.76   Count                  1
    Last switched from time: 10-APR-2022 12:12:53.75
Added in 1 hour 48 minutes 5 seconds:
My tests this morning with more error details (VMS-IDE VSCode output)

Code: Select all

...
Compressed: src/server/sp/sp_mmsgen.mms
Compressed: src/server/sp/sp_mmsgen.txt
upload succeeded: _8052.zip

=========>   Unzip command failed: Error: timeout  <=============

Unzip command output:
 unzip -oo _8052.zip
unzip -oo _8052.zip
Archive:  DISK$GIT:[users.*****.mantis]_8052.zip;1
  inflating: [.src._cicd_config]mms$rules.mms  
  inflating: [.src._cicd_config.test]test_functional_smoke_preprod_.com 
...
If I done by myself the Unzip directly on OpenVMS (thru a PuTTY terminal, with ssh) => it works, no timeout, but it hangs sometimes ~20s on just 1 file (perhaps a clue ?)

Code: Select all

$ dir _8052.zip/size=unit=byte
Directory DISK$GIT:[users.****.mantis]
_8052.zip;1          34.51MB
$
$ sh ti
  14-APR-2022 10:51:25
$
$ unzip -oo _8052.zip
Archive:  DISK$GIT:[users.****.mantis]_8052.zip;1
  inflating: [.src._cicd_config]mms$rules.mms
  inflating: [.src._cicd_config.test]test_functional_smoke_preprod_.com
...
  inflating: [.src.server.sp]sp_mmsgen.mms
  inflating: [.src.server.sp]sp_mmsgen.txt
$
$ sh ti
   14-APR-2022 10:52:46
Last edited by l.cedric on Thu Apr 14, 2022 6:00 am, edited 5 times in total.


sergey_vorfolomeev
VSI Expert
Valued Contributor
Posts: 98
Joined: Thu Aug 22, 2019 12:17 am
Reputation: 0
Status: Offline

Re: Upload failed (cmdTimeout helps a little) ?

Post by sergey_vorfolomeev » Thu Apr 14, 2022 6:19 am

Oops
I found a hardcoded timeout for unzipping 3 sec

Code: Select all

const unzipResult = await shell.execCmd(unzipCmd, undefined, 3000);
We will fix this soon.
PS: Is there any suggestion? New configuration value? Or use existing 'cmdTimeout'?


Topic author
l.cedric
Valued Contributor
Posts: 51
Joined: Thu Jul 18, 2019 8:18 am
Reputation: 0
Status: Offline

Re: Upload failed (cmdTimeout helps a little) ?

Post by l.cedric » Thu Apr 14, 2022 6:23 am

Just one word GREAT :D
use existing 'cmdTimeout'?
seems a good option (?)
But give the units in the documentation (second ? milisecond ? ...)

Thanks Sergey
Last edited by l.cedric on Thu Apr 14, 2022 6:27 am, edited 2 times in total.

User avatar

arne_v
Master
Posts: 299
Joined: Fri Apr 17, 2020 7:31 pm
Reputation: 0
Location: Rhode Island, USA
Status: Offline
Contact:

Re: Upload failed (cmdTimeout helps a little) ?

Post by arne_v » Mon Apr 18, 2022 10:13 am

l.cedric wrote:
Thu Apr 14, 2022 5:11 am
@ARNE_V
I completly agree with you, but it's a ~30 years old ... information system, not well structured
=> monolithic architecture, which cannot be easily restructured.
That situation is not unusual.

:-)

But an obvious question is: how many years is this application expected to live?

2 year, 3 years, 5 years? It probably make sense just to do the minimum necessary.

20 years, 30 years, 50 years? The effort to restructure it will pay off later.
Arne
arne@vajhoej.dk
VMS user since 1986

Post Reply