OpenVMS x86-64 V9.2-1: Significant TOY Slippage over time

[bobgezelter]

I have observed multiple occurrences of accumulating TOY skew running OpenVMS x86-64 under Virtual Box. The underlying Windows host is not particularly busy. In the most recent episode, the time skew is over 1 hour 50 minutes(!) in approximately 24 hours. The skew appears not to be uniform, but episodic.

The hardware/software configuration is:

Host CPU: Dell Latitude E6420 (i5-2520M@2.5GHz; Turbo to 3.26GHz; Typical usage: 30% @ .90GHz); 16GB; 2TB Seagate Barracuda

Host OS: Windows 10 Professional, all patches to date
Virtual Machine: Oracle VirtualBox 7.0.6r1555176 (Qt5.15.2)
Basic OpenVMS x86-64 install; only VSI-supplied basic products (OVMS Community License; TCPIP; DECnet IV)

Examples:
Host Time (EDT) VMS Time (UTC)

Bootstrap 1:
09:38 1338Z
15:25 1857Z
16:26 1958Z
16:47 2019Z
18:46 2218Z
22:49 0219Z

Bootstrap 2:
10:25 1425Z
10:31 1431Z
23:22 0309Z
09:23 1211Z

This is a serious flaw for multiple reasons:
- The slippage is not at a constant ratio. Processes that sequence events based on time can not be reliably run if the clock reference is unpredictably varying. At best it triggers timeouts and correctable errors. At worst, it can compromise the safety of equipment operation.

- Time comparisons between OpenVMS-recorded events and other logs are not reliable.

- Unreliable time stamps compromise the ability to use OpenVMS log time stamps in legal proceedings. Having testified in court involving software-generated logs, unreliable Time of Year recording significantly undermines the reliability of such logs.

- SET TIME from DCL has no effect

It should be noted that guest Ubuntu LTS 20 virtual instances do not encounter corresponding difficulties.

- Bob Gezelter, http://www.rlgsc.com <gezelter@rlgsc.com>

[mjvms27]

Here is a link to a browse location for the VirtualBox Guest Additions code related to synchronizing time:
https://www.virtualbox.org/browser/vbox ... meSync.cpp

The crux of that service thread is calls to a function named VbglR3GetHostTime, which probalby goes down at least another 2 or 3 levels before you actually get to the real communication with the host.

VbglR3GetHostTime is in VBoxGuestR3LibTime.cpp in a subfolder beneath the second link that I included:
https://www.virtualbox.org/browser/vbox ... xGuest/lib

[bobgezelter]

mjvms27,

Resynchronizing the OpenVMS TOY with the host system TOY corrects the cosmetic problem of having diverging time stamps.

However, while helpful, it does not address the deeper problem. Many processes have timing requirements. Significant timing variations can cause bedeviling problems with communicating processes, both within OpenVMS (think OpenVMSclusters) and between OpenVMS-hosted processes and corresponding processes, either on premises or cloud-based.

Resynching the OpenVMS TOY is palliative. It does not address the underlying divergence. That underlying divergence is likely to appear elsewhere under far less benign circumstances. I understand Clair's comment about laptop v server; but I also recognize that the a small seemingly inconsequential change in server environment could uncover the problem on a far broader scale. Been there; seen that.

[cgrant]

As suspected, setting my laptop to "When plugged in, put my device to sleep after = never" eliminates the VBox guest losing time.

Added in 15 minutes 38 seconds:
This thread illustrates the point we have always known, that is in order to truly manage time the guest has to interact with the host. We are already looking into this for VMware via the VMTools interface. We have poked at KVM some but it is behind VMware in priority. We'll put VBox on the list but it is clearly behind the other two in priority.

I know, not a great answer, but that is the best we can do at this time.

Clair

[jonesd]

As suspected, setting my laptop to "When plugged in, put my device to sleep after = never" eliminates the VBox guest losing time.

I started a thread on this topic back in April: [url]viewtopic.php?f=39&t=8659/url]

It's still unclear to me whether just supporting ACPI will provide the necessary communication path between guest and host.

I make sure my laptop never sleeps while running the OpenVMS VM by starting it using this .BAT file

Code: Select all

cd "c:\users\jones\VirtualBox VMs"
"C:\Program Files\Oracle\VirtualBox\VboxManage.exe" startvm "Hobby"
c:\users\jones\nosleep.exe putty -load hobby-com1

This script combines starting the VM with starting the telnet session (saved in Putty as hobby-com1) that acts as its console. The nosleep program is a wrapper that calls the Windows function SetThreadExecutionState to tell Windows not to sleep. When I shut down the VM and close the putty session, nosleep returns the thread execution state to its original value prior to exiting (i.e. allowing sleep again).

[jreagan]

I did a quick Google search for "Does Virtual Box handle Windows sleep mode?" and found several discussions on the Virtual Box support forums for such time slipping with other OSs. The response seemed to be "turn off sleeping".

[bobgezelter]

Clair,

My Latitude is effectively permanently mounted in its docking station, with "Sleep never when in line mode" set.

Sleeping is only enabled for when on battery power.

When I get a chance to have all programs idle. I will reset sleep on battery power to NEVER and reboot. The cord has slipped out once or twice over the weeks, putting the system on batteries for short periods. I want to see if the problem disappears. If it does, then Sleep mode is stickier than one would think.

If that is the case, perhaps a cautionary note in release notes.

- Bob

[mjvms27]

It should be noted that guest Ubuntu LTS 20 virtual instances do not encounter corresponding difficulties.

Are the Ubuntu LTS 20 virtual instances running the VirtualBox Guest Additions for Linux?

Yes, the Ubuntu LTS20 instances have the latest Guest Additions installed.

Resynching the OpenVMS TOY is palliative. It does not address the underlying divergence.

If you're running the Guest Additions on linux, then the time resynching is likely occurring on those Linux systems, and in the absence of comparative test results from a Linux VM that is not running with the Guest Additions, I wonder if the comparison with the OpenVMS situation is really comparing apples-to-apples?

Perhaps the cause of the issue actually resides in VirtualBox behaviour. Has the issue been observed on any hypervisor other than VirtualBox to this point? If I recall correctly, at least one person stated that after switching from VirtualBox to VMware workstation on the same machine, the time drift was no longer observed. I don't think they detailed any alterations to the power management on the host to coincide with that switch, but perhaps they did.

[bobgezelter]

mjvms27,

Separate issues. Linux requires "Guest Additions" due to the fact that the VirtualBox-specific additions are, for whatever reason, not part of the base distribution.

OpenVMS x86-64 at present ONLY runs under hypervisors. No add-in kit should be necessary. Whatever particular hypervisor-specific components should be loaded at system initialization, as has always been the case for processor-specific modules.

- Bob

[bobgezelter]

Clair,

Had a chance to reset the Windows sleep options to "Never" for both battery and line power on the Latitude E6420 (screen shot of power settings attached).

Did a shutdown, power down to cold. Then powered up and full cold boot of Windows 10 Professional. @ 1213 EDT yesterday. When Windows came up logged on and started VirtualBox Manager, then started the OpenVMS instance. At the beginning, the OpenVMS time was correctly 1613Z.

Checked time at 230600 EDT. OpenVMS time then showed as 024740Z. Redid the time check at 053800EDT, OpenVMS showed 091635Z.

Host sleep setting is not doing it. From previous runs, the drift is monotonic, but far from uniform.

I also concur somewhat with mjvms27, this problem could well be a VirtualBox "feature", or at least an underappreciated subtlety of the client/VirtualBox interface. The sources of the linux Guest Additions may be yield insight.

I would prefer a running this to ground, as it may reveal other issues. In the immediate case, some palliative temporary remediations:
- resynch the OpenVMS TOY from the host TOY every 5-10 seconds.
- if the hypervisors support notification of suspend/resume, generate an error log event to that effect
- log if the TOY resynch yields the lost of more than a small number of ticks
- document the above in Release Notes
- consider a documentation note explaining the variability of time on a virtualized instance stemming from the hupervisor and host operating systems' time-slicing between different activities.

[jonesd]

I would prefer a running this to ground, as it may reveal other issues. In the immediate case, some palliative temporary remediations:
- resynch the OpenVMS TOY from the host TOY every 5-10 seconds.
- if the hypervisors support notification of suspend/resume, generate an error log event to that effect
- log if the TOY resynch yields the lost of more than a small number of ticks

Is your VM instance not on a network? Create an app that connects to a host with a reliable clock and send heartbeat messages with a timestamp once per second, getting a reply with the remote's current time.

My VMS machines run NTP daemon that comes with TCP/IP Services, configured to get time from ntp.org.