Synchronization Upload quota issue on "bigger" files

Here you can discuss the universal Integrated Development Environment for OpenVMS.

Topic author
cgoodwin
Contributor
Posts: 15
Joined: Wed Aug 21, 2019 8:43 am
Reputation: 0
Status: Offline

Synchronization Upload quota issue on "bigger" files

Post by cgoodwin » Wed Oct 16, 2019 4:34 pm

I have been synchronizing, building and debugging files using the VMS IDE for the past couple weeks and things have been working well. Recently however, I synchronized one of our larger Fortran files (over 3000 lines). Synchronizing it locally seemed to work ok, I made the edits, then synchronized it back to VMS. The output log said that the upload of that file was successful. When I went to build, it failed. VS Code also gave me a waarning saying that it got truncated after 7200 characters. Upon further investigation, it appeared that the build was only trying to build code from the first third of the file. I went into VMS and did a type command on the code file, and after outputting the first third of the code file, it errored out with the following message:

Code: Select all

%TYPE-F-WRITEERR, error writing SYS$OUTPUT:.;
-RMS-F-SYS, QIO system service request failed
-SYSTEM-F-EXQUOTA, process quota exceeded
Doing a type command on the previous version of the file (before any synchronizations), successfully outputs the full file. I deleted the most recent version on both VMS and locally then added some text to the end of the file, synchronized again, did my type command and had the same issue. I did this same test, except I removed text from the code file instead of adding text, and still got the same results. What could cause this issue?


sergey_vorfolomeev
VSI Expert
Valued Contributor
Posts: 98
Joined: Thu Aug 22, 2019 12:17 am
Reputation: 0
Status: Offline

Re: Synchronization Upload quota issue on "bigger" files

Post by sergey_vorfolomeev » Fri Oct 18, 2019 4:26 am

We are unable to reproduce the problem using considerably larger files. The problem could be related to a quota issue (BYTLM for example). What happens if you try sftp’ing the file manually?


Topic author
cgoodwin
Contributor
Posts: 15
Joined: Wed Aug 21, 2019 8:43 am
Reputation: 0
Status: Offline

Re: Synchronization Upload quota issue on "bigger" files

Post by cgoodwin » Fri Oct 18, 2019 12:34 pm

The file is ok when I sftp it, just when I use the VMS IDE Sync functionalities is when the issue occurs.


brianreiter
Active Contributor
Posts: 26
Joined: Fri Jun 14, 2019 4:17 pm
Reputation: 0
Status: Offline

Re: Synchronization Upload quota issue on "bigger" files

Post by brianreiter » Mon Oct 21, 2019 4:40 am

Have you a look at the file? DIR/FULL or a DUMP/REC?

I've had that error in the past when FTP/SFTP have done something to the file format. Something akin to what we get when text files are uploaded as binary using FTP.

Might be worthwhile doing a DIFF between what gets uploaded and the previous version,


Topic author
cgoodwin
Contributor
Posts: 15
Joined: Wed Aug 21, 2019 8:43 am
Reputation: 0
Status: Offline

Re: Synchronization Upload quota issue on "bigger" files

Post by cgoodwin » Mon Oct 21, 2019 11:56 am

I did have a look at the both the previous version compared to the uploaded version of a dir/full for the file and there is no difference. Doing a dump/rec and viewing the results, it ends prematurely with a bunch of '.'s.

When I did a DIFF it fails prematurely with the following error (at the same place it fails on the type, dump and edit):
%DIFF-F-READERR, error reading file_name
-RMS-W-RTB, 25711 byte record too large for user's buffer

Added in 1 hour 20 minutes 5 seconds:
As a small update, I synced the original file locally using Filezilla, renamed the file, added it to the project, then synced it back up with VMS. This did work, so the problem must be with the initial sync down.

Added in 22 minutes :
An even stranger result just came from a second test I just performed. After my previous test, I deleted my local copies, and resynced with VMS. This pulled down:
- The original
- The version I renamed from my previous test.
I added a new line to both, resynchded them both to VMS, then checked them on VMS. The renamed file works!
As another interesting bit, since this uploads a new version of the original, I decided to wipe my local files again and sync with VMS an additional time. The new version of the original that synced had a bunch of scrambled characters where it would error out on VMS (it looks like a binary encoding). Is there a possibility that part of the file could be getting a binary encoding when being synced for any reason?
Last edited by cgoodwin on Mon Oct 21, 2019 1:43 pm, edited 2 times in total.


sergey_vorfolomeev
VSI Expert
Valued Contributor
Posts: 98
Joined: Thu Aug 22, 2019 12:17 am
Reputation: 0
Status: Offline

Re: Synchronization Upload quota issue on "bigger" files

Post by sergey_vorfolomeev » Tue Oct 22, 2019 12:48 am

What type of synchronization did you use? "downloadNewFiles":
https://wiki.vmssoftware.com/Synchronize_Configuration


Topic author
cgoodwin
Contributor
Posts: 15
Joined: Wed Aug 21, 2019 8:43 am
Reputation: 0
Status: Offline

Re: Synchronization Upload quota issue on "bigger" files

Post by cgoodwin » Tue Oct 22, 2019 9:24 am

sergey_vorfolomeev wrote:
Tue Oct 22, 2019 12:48 am
What type of synchronization did you use? "downloadNewFiles":
https://wiki.vmssoftware.com/Synchronize_Configuration
I always used overwrite.


dante123
Member
Posts: 7
Joined: Thu Jan 07, 2021 10:55 am
Reputation: 0
Status: Offline

Re: Synchronization Upload quota issue on "bigger" files

Post by dante123 » Mon Jan 11, 2021 8:17 am

I have a similar problem, "upload" transfer files with corruption.
I have 2 files that were corruppted on VMS after upload/synchronization
- a dcl (make procedure) file which was corrupted once, about 10 empty rows were inserted at a random place and of course the DCL syntax was broken. Only once!

- a fortran file about 50KB which is always corrupted, not always at the same place and in the same way. But never at the beginning of the file. (additional characters added, end of line removed at some rows, ...)

The IDE says: Synchronization completed successfully. [projectname]

Corruption so far occurs only at upload, It's not the size of file because another fortran file which is bigger has never been corrupted.
The file had swedish character in comments so I removed then, I also tried to copy paste with notepad and other editor (in case there was still invisible character)
ftp up/down does not alter the file

Is there a part that try to translate the encoding and is not working ?
Now the only things that is working for this file is, after upload, copy the text in the IDE, telnet vms open the file delete everyting and paste the text.

This program has only 5 source files so I detect the error, but we have process with hundred files ...

I suspect that some encoding translation does not understand the special fortran code.

Is there some hidden configuration switch I can test ?
How can I trace which commands are issued during transfer?


sergey_vorfolomeev
VSI Expert
Valued Contributor
Posts: 98
Joined: Thu Aug 22, 2019 12:17 am
Reputation: 0
Status: Offline

Re: Synchronization Upload quota issue on "bigger" files

Post by sergey_vorfolomeev » Fri Jan 15, 2021 12:28 am

Sorry for late answer.
There is no special encoding or hidden configuration switch. All files uploaded via the same way - code, images, zip e.t.c.

Can You test if such files successfully uploaded via SFTP instead of FTP?

Also, You can use "VMS-IDE: Upload all using ZIP" command for selected project. But in this case file dates will be lost (it will be the date when file is unzipped).


dante123
Member
Posts: 7
Joined: Thu Jan 07, 2021 10:55 am
Reputation: 0
Status: Offline

Re: Synchronization Upload quota issue on "bigger" files

Post by dante123 » Fri Jan 15, 2021 11:08 am

I do not have corruption when transfering the file with sftp, only from the IDE.
At corruption position I have nl and other invisible character mixed with the rest of the code (see attachment)

Typically:

On Windows:
C:\Users\adm037349\Documents\IPSERVER\SRC>sftp
server : prv003
user name : dantev
Welcome to HP OpenVMS Industry Standard 64 Operating System, Version V8.4-2L1
Enter password for dantev@prv003:
/dsa1/USER/DANTEV>put IPSERVER_NET.F90
Transferring IPSERVER_NET.F90
Uploaded C:\Users\adm037349\Documents\IPSERVER\SRC\IPSERVER_NET.F90 to /dsa1/USER/DANTEV/IPSERVER_NET.F90 (47.46kB 0.00 B/s 00:00:01)
/dsa1/USER/DANTEV>

On VMS
DV_PRV003> diff ipserver_net.f90; USER_ROOT:[DANTEV.VSCODE.IPS.SRC]IPSERVER_NET.F90;74
Number of difference sections found: 0
Number of difference records found: 0

DIFFERENCES /MERGED=1-
USER_ROOT:[DANTEV]IPSERVER_NET.F90;1-
USER_ROOT:[DANTEV.VSCODE.IPS.SRC]IPSERVER_NET.F90;74



After synchronisering ( version 75 created, no change in code just saved on windows to force synch)
IDE:
upload succeeded: SRC/IPSERVER_NET.F90
Synchronization completed successfully. [IPSERVER]

On VMS
DV_PRV003> diff ipserver_net.f90; USER_ROOT:[DANTEV.VSCODE.IPS.SRC]IPSERVER_NET.F90;75
************
File USER_ROOT:[DANTEV]IPSERVER_NET.F90;1
1449 deallocate(bytBuf13, stat=intStat)
1450 end if
1451
1452 case (14)
...
2160
2161 ! - end dante
******
File USER_ROOT:[DANTEV.VSCODE.IPS.SRC]IPSERVER_NET.F90;75
1449 deall
1450 shtParTy3 & /TCPIP$C_OOBINLINE/š integer*4 intParAdr3" integer*ocate(bytBuf13, stat=intStat)e
en
************

Number of difference sections found: 1
Number of difference records found: 713

DIFFERENCES /MERGED=1-
USER_ROOT:[DANTEV]IPSERVER_NET.F90;1-
USER_ROOT:[DANTEV.VSCODE.IPS.SRC]IPSERVER_NET.F90;75
DV_PRV003>
Attachments
editor.png
editor.png (3.86 KiB) Viewed 12890 times

Post Reply