SPS-632 - X86E92OE FTP client hangs in GET from another (older) VMS system

[sms]

Code: Select all

   Fool that I am, I recently tried to use FTP to fetch some files from
an x86_64 system to an IA64 system:

ITS $ ftp v87
220-   Antinode V87 FTP Server.  Please be nice.
220 v87.antinode.info FTP Server (Version 6.0) Ready.
Connected to v87.antinode.info. 
Name (v87.antinode.info:system): 
331 Username system requires a Password
Password: 
230 User logged in.
FTP> get debug_v87-m_2deb_len_256_srv.log
200 TYPE set to IMAGE.
200 PORT command successful.
150 Opening data connection for SYS$SYSROOT:[SYSMGR]debug_v87-m_2deb_len_256_srv
.log; (10.0.0.140,63891) (21576 bytes)
<Ctrl/T>
ITS::_FTA165: 22:46:53 TCPIP$FTP CPU=00:00:03.80 PF=8939 IO=123568 MEM=750
GET (VMS+)                  0 bytes 00:00:18.52 elapsed (0.00 B/S)
      Local: ITS$DKA0:[UTILITY.SOURCE.KERMIT.debug]debug_v87-m_2deb_len_256_srv.
log;
     Remote: debug_v87-m_2deb_len_256_srv.log
<Ctrl/T>
ITS::_FTA165: 22:47:19 TCPIP$FTP CPU=00:00:03.80 PF=8939 IO=123572 MEM=750
GET (VMS+)                  0 bytes 00:00:45.20 elapsed (0.00 B/S)
      Local: ITS$DKA0:[UTILITY.SOURCE.KERMIT.debug]debug_v87-m_2deb_len_256_srv.
log;
     Remote: debug_v87-m_2deb_len_256_srv.log
<Ctrl/C>
 Cancel 

%TCPIP-E-FTP_FDLFAIL, cannot process input FDL file
-SYSTEM-W-CANCEL, operation canceled
FTP> quit
221 Goodbye.
ITS $ dire
%DIRECT-W-NOFILES, no files found

   This all looked vaguely familiar, and a little digging through the
local correspondence archives found some complaints which I submitted
(a while ago) against the E9.2 field test release, SPS-632.

   As a peon, I now seem to be unable to see any of that stuff on the
Service Platform, so I don't know if any relevant improvements were ever
made.

   If someone with better access could check the status, I'd be vaguely
interested (and slightly grateful).  My quick impression is that it's
all the same at V9.2-2, but what do I know?

   For completeness:

ITS $ tcpip show version

  HP TCP/IP Services for OpenVMS Industry Standard 64 Version V5.7 - ECO 5
  on an HP rx2600  (1.50GHz/6.0MB) running OpenVMS V8.4-2L3

V87 $ tcpip show version

  VSI TCP/IP Services for OpenVMS x86_64 Version V6.0
  on a VMware, Inc. VMware20,1 running OpenVMS V9.2-2  

   Motivation:
   Now that there's no longer any need, I've been poking around at the
possibility of using Kermit to transfer a PAK file like our old pal,
"X86Community-20240401.txt" (preferably without the nonsensical
corruption which spoiled the original one), over a VMware Fusion Telnet
"serial port" onto a fresh VMS installation (where no real networking
has been licensed/configured).

   Kermit works as expected over a normal Telnet connection, but there
are problems when using the pseudo-serial-port.  So, I have a bunch of
debug log files in different places, and it's annoying to have to
remember which file-transfer schemes don't work where.

   Yes, for this to work, the VMS kit would need to include/install a
Kermit executable.  (SYS$COMMON:[SYSHLP.UNSUPPORTED.Kermit]?)  I'd
expect 4.5MB (1.5MB compressed) to fit easily enough.

[cct]

Have you tried passive mode?

[dgordon]

$ TCPIP SHOW VERSION/ALL will give you better info about the exact version of each component at the cost of more info than you'd really like in most cases.

I checked SPS-632 and it was closed with "fixed in the next release" where the tested x86 version for this claim was X86VMS TCPIP X6.0-18. My V9.2-2 system is V6.0-23.

That said, I (lazily) use FTP all the time to both push and pull files between V8.4-* and x86 V9 systems, albeit via the COPY/FTP command. One of the reasons I prefer COPY/FTP is that it supports VMS lists and wildcards.

[sms]

Code: Select all

> Have you tried passive mode?

   Yes, but as I expected, no effect:

FTP> passive on
Passive is ON.
FTP> get debug_v87-m_2deb_len_256_srv.log
200 TYPE set to IMAGE.
227 Entering Passive Mode (10,0,0,184,192,82)
150 Opening data connection for SYS$SYSROOT:[SYSMGR]debug_v87-m_2deb_len_256_srv
.log; (10.0.0.140,64798) (21576 bytes)
ITS::_FTA2: 12:38:21 TCPIP$FTP CPU=00:01:44.13 PF=58987 IO=303303 MEM=798
GET (VMS+)                  0 bytes 00:00:05.63 elapsed (0.00 B/S)
      Local: ITS$DKA0:[prod.VSI]debug_v87-m_2deb_len_256_srv.log;
     Remote: debug_v87-m_2deb_len_256_srv.log
ITS::_FTA2: 12:38:27 TCPIP$FTP CPU=00:01:44.13 PF=58988 IO=303307 MEM=799
GET (VMS+)                  0 bytes 00:00:12.06 elapsed (0.00 B/S)
      Local: ITS$DKA0:[prod.VSI]debug_v87-m_2deb_len_256_srv.log;
     Remote: debug_v87-m_2deb_len_256_srv.log
<Ctrl/C>
 Cancel 

%TCPIP-E-FTP_FDLFAIL, cannot process input FDL file
-SYSTEM-W-CANCEL, operation canceled
FTP> 

   As I understand passive FTP, it solves a particular class of problems
usually associated with passage through a NAT router.  These systems are
on the same LAN.

   As I believe that I mentioned in the original report, I can fetch the
file from VMS-x86_64 to a Mac, and from the Mac to VMS-IA64, so I had
more hope for "DISABLE VMS_PLUS" (which I hadn't yet discovered at that
time), but that also disappointed.


> $ TCPIP SHOW VERSION/ALL will give you better info [...]

ITS $ pipe tcpip show version /all | search sys$pipe $ftp
  TCPIP$FTP_CHILD;1         "V5.7-ECO5F"      30-SEP-2016  SYS$COMMON:[SYSEXE]
  TCPIP$FTP_CLIENT;1        "V5.7-ECO5K"      23-JAN-2018  SYS$COMMON:[SYSEXE]
  TCPIP$FTP_SERVER;1        "V5.7-ECO5F"      30-SEP-2016  SYS$COMMON:[SYSEXE]


V87 $ pipe tcpip show version /all | search sys$pipe $ftp
  TCPIP$FTP_CHILD;1         "V6.0-24"         23-JAN-2024  SYS$COMMON:[SYSEXE]  
         
  TCPIP$FTP_CLIENT;1        "V6.0-24"         23-JAN-2024  SYS$COMMON:[SYSEXE]  
         
  TCPIP$FTP_SERVER;1        "V6.0-24"         23-JAN-2024  SYS$COMMON:[SYSEXE]  
         

   Expanding the fixed-width "Image location" field in that report
solved the problem of truncated names like "SYS$COMMON:[SYS$LDR":

ITS $ pipe tcpip show version /all | search sys$pipe $tn
  TCPIP$TNDRIVER;1          "V5.7-ECO5A"       5-FEB-2015  SYS$COMMON:[SYS$LDR
ITS $ 

   But leaving the trailing white space did no favors for people using
an 80-column terminal/window.

V87 $ pipe tcpip show version /all | search sys$pipe $tn
  TCPIP$TNDRIVER;1          "V6.0-24"         23-JAN-2024  SYS$COMMON:[SYS$LDR] 
         <--- End of line.
V87 $ 


> [...] via the COPY/FTP command. [...]

   When I get more time to kill I might try that.

[debbee.west]

I am also not having this issue in our lab.

SYSTEM IA18> ftp 10.10.112.108
220 twinkle.eng.vmssoftware.com FTP Server (Version 6.0) Ready.
Connected to ip-10-10-112-219.ec2.internal.
Name (ip-10-10-112-219.ec2.internal:system):
331 Username system requires a Password
Password:
230 User logged in.
FTP> get ucx$config.com test_transfer.delete
200 TYPE set to IMAGE.
200 PORT command successful.
150 Opening data connection for SYS$COMMON:[SYSMGR]ucx$config.com; (10.10.118.5,
58830) (1498 bytes)
226 Transfer complete.
local: SYS$SYSROOT:[SYSMGR]test_transfer.delete;1 remote: ucx$config.com
1498 bytes received in 00:00:00.00 seconds (208.98 Kbytes/s)
FTP> set def sys$system
250-CWD command successful.
250 New default directory is SYS$SYSROOT:[SYSEXE]
FTP> get tcpip$configuration.dat tcpip$configuration.dat_delete
200 PORT command successful.
150 Opening data connection for SYS$COMMON:[SYSEXE]tcpip$configuration.dat; (10.
10.118.5,58835) (57344 bytes)
226 Transfer complete.
local: SYS$SYSROOT:[SYSMGR]tcpip$configuration.dat_delete;1 remote: tcpip$confi
guration.dat
57344 bytes received in 00:00:00.00 seconds (6.84 Mbytes/s)
FTP> quit

The client runs TCPIP 5.7 eco 5
SYSTEM IA18> pip tcpip sho ver/all | search sys$pipe ftp
File Transfer Protocol (FTP):
TCPIP$FTP_CHILD;1 "V5.7-ECO5F" 30-SEP-2016 SYS$COMMON:[SYSEXE]
TCPIP$FTP_CLIENT;1 "V5.7-ECO5K" 23-JAN-2018 SYS$COMMON:[SYSEXE]
TCPIP$FTP_SERVER;1 "V5.7-ECO5F" 30-SEP-2016 SYS$COMMON:[SYSEXE]
TCPIP$TFTP;1 "V5.7-ECO5" 8-NOV-2014 SYS$COMMON:[SYSEXE]
TCPIP$SSH_SFTP-SERVER2;1 "V5.7-ECO5O" 9-OCT-2017 SYS$COMMON:[SYSEXE]
TCPIP$SSH_SFTP2;1 "V5.7-ECO5O" 9-OCT-2017 SYS$COMMON:[SYSEXE]
SYSTEM IA18>

The server is running the latest release of TCPIP
TWNKLE $ pipe tcpip sho ver/all | search sys$pipe ftp
File Transfer Protocol (FTP):
TCPIP$FTP_CHILD;1 "V6.0-24" 23-JAN-2024 SYS$COMMON:[SYSEXE]
TCPIP$FTP_CLIENT;1 "V6.0-24" 23-JAN-2024 SYS$COMMON:[SYSEXE]
TCPIP$FTP_SERVER;1 "V6.0-24" 23-JAN-2024 SYS$COMMON:[SYSEXE]
TCPIP$TFTP;1 "V6.0-24" 23-JAN-2024 SYS$COMMON:[SYSEXE]

There could be an issue with a specific type of file. Are you having issues with ALL files? If its just some files, a dir/full on one will help with testing.

Also, I tested to machines running x86 9.2-2 and 9.2-2 update 1, along with the tcpip -23 and -24 kits. (FYI)

Thank you

Added in 4 minutes 47 seconds:
One more thing,,,

Your file is a log file. Is it really an image file? Is there really an fdl file for this log file? Your error thinks there is an FDL file.

%TCPIP-E-FTP_FDLFAIL, cannot process input FDL file

Can you post the dir for that directory/file?

Also, do you have a system running 5.7 eco 5 that you can try as a client?

Thanks again!

[arne_v]

GET from another (older) VMS system
...
FTP> get debug_v87-m_2deb_len_256_srv.log
...
GET (VMS+) 0 bytes 00:00:18.52 elapsed (0.00 B/S)
...
%TCPIP-E-FTP_FDLFAIL, cannot process input FDL file

This may be stating the obvious, but is what is happening here that:
- FTP realize that this is VMS to VMS (VMS+)
- FTP tries to move file attributes over via FDL file
- recipient can't process the FDL received
?

Added in 5 minutes 14 seconds:
Don't know if it provide any insight, but which of these fail:
- FTP on Itanium to x86-64 and get
- FTP on x86-64 to Itanium and put
- COPY/FTP on Itanium
- COPY/FTP on x86-64
?

[sms]

Code: Select all

   As always, thanks for looking into this.

> Can you post the dir for that directory/file?

   It's whatever Kermit creates for a "debug.log" file (renamed for
uniqueness).  Apparently that's an old-fashioned text file.  (I should
look into changing that to StreamLF, as an option and/or the default).

V87 $ dire /full sys$manager:debug_v87-m_2deb_len_256_srv.log

Directory SYS$SYSROOT:[SYSMGR]

debug_v87-m_2deb_len_256_srv.log;1        File ID:  (46442,106,0)         
Size:           43/48         Owner:    [SYSTEM]
Created:    11-APR-2024 22:29:51.75
Modified:   11-APR-2024 22:34:33.25 (2)
Expires:    <None specified>
Backup:     <No backup recorded>
Effective:  <None specified>
Recording:  <None specified>
Accessed:   <None specified>
Attr Mod:   11-APR-2024 22:34:33.25
Data Mod:   11-APR-2024 22:29:51.75
Linkcount:  1
File organization:  Sequential
Shelved state:      Online 
Caching attribute:  Writethrough
File attributes:    Allocation: 48, Extend: 0, Global buffer count: 0
                    No version limit
Record format:      Variable length, maximum 254 bytes, longest 171
bytes
Record attributes:  Carriage return carriage control
RMS attributes:     None
Journaling enabled: None
File protection:    System:RWED, Owner:RWED, Group:RE, World:
Access Cntrl List:  None
Client attributes:  None

Total of 1 file, 43/48 blocks.


   I'll try more file types when I get a chance.


> Also, do you have a system running 5.7 eco 5 that you can try as a
> client?

   ITS (IA64, V5.7 ECO 5) _is_ the client.  I have a spare/experimental
rx2600 (REX) which should have very similar software, so I should be
able to run some realistic tests between IA64 systems.  It might be a
day or two.


> - FTP realize that this is VMS to VMS (VMS+)

   Yes, but "DISABLE VMS_PLUS" didn't help.


> Don't know if it provide any insight, but which of these fail:
> [...]

   I'll try these as time allows.

[sms]

Code: Select all

   First:

> Also, do you have a system running 5.7 eco 5 that you can try as a
> client?

   I failed to notice that the original complaint (SPS-632) faced in the
opposite direction.  All the recent trouble has been with an IA64 client
and x86_64 server.  I haven't tried an IA64 server lately, but I should
be able to do that soonish, if anyone cares.

   I _have_ run some tests with clients on two different IA64 systems,
with two different x86_64 servers, and two sets of files, one of which
works, and the other of which fails.  With any luck at all, this might
make for a test set which will replicate the problem on someone else's
systems.

 Test file collections ("zip -V", so UnZip on VMS should restore
attributes):

      Real-life files (about 32KB each):
      http://antinode.info/ftp/VSI/debug_v87xxx.zip

debug_v87-m_2deb_len_256_srv.log;1
debug_v87-m_2deb_short_srv.log;1
debug_v87_2deb_len_256_srv.log;1
debug_v87_2deb_short_srv.log;1

      Small test files (less than 512B):
      http://antinode.info/ftp/VSI/ftp_test_xxx.zip

ftp_test_fix-512.dat;1
ftp_test_stm.txt;1
ftp_test_stmlf.txt;1
ftp_test_var.txt;1

   The small files (ftp_test_*.*) have different record formats; the
real-life files (debug_*.log) all have Record format: Variable length.

   Being lazy, I put all the source test files into SYS$MANAGER on the
servers, and fetched them into empty directories on the clients.

   Transferring the small files works; transferring the larger files
fails.

   Systems:

      IA64 (clients):

ITS $ pipe tcpip show vers /all | search sys$pipe "p services", $ftp
  HP TCP/IP Services for OpenVMS Industry Standard 64 Version V5.7 - ECO 5
  TCPIP$FTP_CHILD;1         "V5.7-ECO5F"      30-SEP-2016  SYS$COMMON:[SYSEXE]
  TCPIP$FTP_CLIENT;1        "V5.7-ECO5K"      23-JAN-2018  SYS$COMMON:[SYSEXE]
  TCPIP$FTP_SERVER;1        "V5.7-ECO5F"      30-SEP-2016  SYS$COMMON:[SYSEXE]

REX $ pipe tcpip show vers /all | search sys$pipe "p services", $ftp
  HP TCP/IP Services for OpenVMS Industry Standard 64 Version V5.7 - ECO 5
  TCPIP$FTP_CHILD;1         "V5.7-ECO5F"      30-SEP-2016  SYS$COMMON:[SYSEXE]
  TCPIP$FTP_CLIENT;1        "V5.7-ECO5K"      23-JAN-2018  SYS$COMMON:[SYSEXE]
  TCPIP$FTP_SERVER;1        "V5.7-ECO5F"      30-SEP-2016  SYS$COMMON:[SYSEXE]

      x86_64 (servers) (not _my_ excessive white space):

V87 $ pipe tcpip show vers /all | search sys$pipe "p services", $ftp
  VSI TCP/IP Services for OpenVMS x86_64 Version V6.0
  TCPIP$FTP_CHILD;1         "V6.0-24"         23-JAN-2024  SYS$COMMON:[SYSEXE]  
         
  TCPIP$FTP_CLIENT;1        "V6.0-24"         23-JAN-2024  SYS$COMMON:[SYSEXE]  
         
  TCPIP$FTP_SERVER;1        "V6.0-24"         23-JAN-2024  SYS$COMMON:[SYSEXE]  
         
V90 $ pipe tcpip show vers /all | search sys$pipe "p services", $ftp
  VSI TCP/IP Services for OpenVMS x86_64 Version V6.0
  TCPIP$FTP_CHILD;1         "V6.0-23"         17-OCT-2023  SYS$COMMON:[SYSEXE]  
         
  TCPIP$FTP_CLIENT;1        "V6.0-23"         17-OCT-2023  SYS$COMMON:[SYSEXE]  
         
  TCPIP$FTP_SERVER;1        "V6.0-23"         17-OCT-2023  SYS$COMMON:[SYSEXE]  
         

(Apparently, I haven't yet applied the latest TCPIP update to V90, but
it seemed not to matter.)


   Typical session:

      ITS (IA64) Client, V87 (x86_64) Server
      --------------------------------------

ITS $ show default
  ITS$DKA0:[junk.v87]

ITS $ ftp v87
220-   Antinode V87 FTP Server.  Please be nice.
220 v87.antinode.info FTP Server (Version 6.0) Ready.
Connected to v87.antinode.info. 
Name (v87.antinode.info:system): 
331 Username system requires a Password
Password: 
230 User logged in.
FTP> get ftp_test_fix-512.dat
200 TYPE set to IMAGE.
200 PORT command successful.
150 Opening data connection for SYS$SYSROOT:[SYSMGR]ftp_test_fix-512.dat; (10.0.
0.140,55738) (352 bytes)
226 Transfer complete.
local: ITS$DKA0:[junk.v87]ftp_test_fix-512.dat;1  remote: ftp_test_fix-512.dat
352 bytes received in 00:00:00.00 seconds (49.11 Kbytes/s)
FTP> get ftp_test_stm.txt
200 PORT command successful.
150 Opening data connection for SYS$SYSROOT:[SYSMGR]ftp_test_stm.txt; (10.0.0.14
0,55751) (18 bytes)
226 Transfer complete.
local: ITS$DKA0:[junk.v87]ftp_test_stm.txt;1  remote: ftp_test_stm.txt
18 bytes received in 00:00:00.00 seconds (3.52 Kbytes/s)
FTP> get ftp_test_stmlf.txt
200 PORT command successful.
150 Opening data connection for SYS$SYSROOT:[SYSMGR]ftp_test_stmlf.txt; (10.0.0.
140,55764) (19 bytes)
226 Transfer complete.
local: ITS$DKA0:[junk.v87]ftp_test_stmlf.txt;1  remote: ftp_test_stmlf.txt
19 bytes received in 00:00:00.00 seconds (2.65 Kbytes/s)
FTP> get ftp_test_var.txt;1
200 PORT command successful.
150 Opening data connection for SYS$SYSROOT:[SYSMGR]ftp_test_var.txt;1 (10.0.0.1
40,55771) (18 bytes)
226 Transfer complete.
local: ITS$DKA0:[junk.v87]ftp_test_var.txt;1  remote: ftp_test_var.txt;1
18 bytes received in 00:00:00.00 seconds (2.51 Kbytes/s)
FTP> 
FTP> get debug_v87-m_2deb_len_256_srv.log
200 PORT command successful.
150 Opening data connection for SYS$SYSROOT:[SYSMGR]debug_v87-m_2deb_len_256_srv
.log; (10.0.0.140,55790) (21576 bytes)
<Ctrl/T>
ITS::_FTA2: 21:21:18 TCPIP$FTP CPU=00:02:08.68 PF=73894 IO=351787 MEM=848
GET (VMS+)                  0 bytes 00:00:11.22 elapsed (0.00 B/S)
      Local: ITS$DKA0:[junk.v87]debug_v87-m_2deb_len_256_srv.log;
     Remote: debug_v87-m_2deb_len_256_srv.log
<Ctrl/T>
ITS::_FTA2: 21:21:59 TCPIP$FTP CPU=00:02:08.68 PF=73894 IO=351791 MEM=848
GET (VMS+)                  0 bytes 00:00:51.89 elapsed (0.00 B/S)
      Local: ITS$DKA0:[junk.v87]debug_v87-m_2deb_len_256_srv.log;
     Remote: debug_v87-m_2deb_len_256_srv.log
<Ctrl/C>
 Cancel 

%TCPIP-E-FTP_FDLFAIL, cannot process input FDL file
-SYSTEM-W-CANCEL, operation canceled
FTP> get debug_v87-m_2deb_short_srv.log
200 PORT command successful.
150 Opening data connection for SYS$SYSROOT:[SYSMGR]debug_v87-m_2deb_short_srv.l
og; (10.0.0.140,55881) (31328 bytes)
<Ctrl/T>
ITS::_FTA2: 21:22:54 TCPIP$FTP CPU=00:02:08.68 PF=73917 IO=351834 MEM=867
GET (VMS+)                  0 bytes 00:00:34.81 elapsed (0.00 B/S)
      Local: ITS$DKA0:[junk.v87]debug_v87-m_2deb_short_srv.log;
     Remote: debug_v87-m_2deb_short_srv.log
<Ctrl/T>
ITS::_FTA2: 21:23:58 TCPIP$FTP CPU=00:02:08.68 PF=73917 IO=351838 MEM=867
GET (VMS+)                  0 bytes 00:01:38.76 elapsed (0.00 B/S)
      Local: ITS$DKA0:[junk.v87]debug_v87-m_2deb_short_srv.log;
     Remote: debug_v87-m_2deb_short_srv.log
<Ctrl/C>
 Cancel 

%TCPIP-E-FTP_FDLFAIL, cannot process input FDL file
-SYSTEM-W-CANCEL, operation canceled
FTP> [...]

   Similar failures for debug_v87_2deb_len_256_srv.log and
debug_v87_2deb_short_srv.log.

   The only files in the destination were "ftp_test_*.*".  (Failures
left no residue.)

   The results were the same using a client on REX, or the server on
V90, or both, so it doesn't look to me like one defective system.

   As always, questions are welcome.

[sms]

Code: Select all

> [...] All the recent trouble has been with an IA64 client and x86_64
> server. [...]

   Tests in the opposite direction, that is, with an x86_64 client
(either V87 or V90) and an IA64 server (either ITS or REX), still doing
GET (as in the original SPS-632 complaint), work and fail in exactly the
same ways.

Added in 7 hours 42 minutes 20 seconds:

Code: Select all

   Hmmm.  The original SPS-632 complaint claimed that this worked with
an x86_64 client (V86) and an x86_64 server (also V86), but tests today
between different x86_64 systems (V87, V90) show the same
success-failure mix as with mixed IA64-x86_64 tests.

[sms]

Code: Select all

   Converting the failing files from Record format: Variable length to
Record format: Stream_LF had no obvious effect on the results.

   For your convenience:

ITS $ type utility:cnvstmlf.com
$!                                              12 December 1999.  SMS.
$!
$!    CONVERT a file to StreamLF record format.
$!
$ convert 'p1' 'p2' /fdl = sys$input:
RECORD
        FORMAT stream_lf
$!
ITS $