E9.2-3 and SECURITY_SERVER issue

[vkorkko]

We had last nigth apparently SECURITY_SERVER gone "mad"/"confused"

It was looping, at priority 0 and login reportedly only possible from console.
After initial investigation an $ SET SERVER SECURITY_SERVER/RESTART was attempted but no help.
Then a crash was forced and system recovered usable


Attached is the CLUE$JUHANI_151024_0046.LIS named as .txt

_veli

Added in 3 minutes 21 seconds:
Crash dump is available but it is somewhat big, 3GB.

Juhani> prod show prod/fu
------------------------------------ ----------- --------- ------------------------------------ ------------------------------------
PRODUCT KIT TYPE STATE MAINTENANCE REFERENCED BY
------------------------------------ ----------- --------- ------------------------------------ ------------------------------------
SCI VMS DFU V3.5-100 Full LP Installed
VMSPORTS X86VMS PERL534 T5.34-0 Full LP Installed VSI X86VMS OPENVMS E9.2-3
VSI X86VMS AVAIL_MAN_BASE E9.2-3 Full LP Installed VSI X86VMS OPENVMS E9.2-3
VSI X86VMS DECNET_PLUS V9.2-G Full LP Installed VSI X86VMS OPENVMS E9.2-3
VSI X86VMS DWMOTIF V1.8-1 Full LP Installed VSI X86VMS OPENVMS E9.2-3
VSI X86VMS DWMOTIF_SUPPORT E9.2-3 Full LP Installed VSI X86VMS VMS E9.2-3
VSI X86VMS KERBEROS V3.3-3 Full LP Installed VSI X86VMS OPENVMS E9.2-3
VSI X86VMS OPENSSH V8.9-1I01 Full LP Installed VSI X86VMS OPENVMS E9.2-3
VSI X86VMS OPENVMS E9.2-3 Platform Installed
VSI X86VMS SSL111 V1.1-1W Full LP Installed VSI X86VMS OPENVMS E9.2-3
VSI X86VMS SSL3 V3.0-13 Full LP Installed VSI X86VMS OPENVMS E9.2-3
VSI X86VMS TCPIP V6.0-25
VSI X86VMS T4 V4.4-E Full LP Installed
VSI X86VMS TCPIP V6.0-25 Full LP Installed VSI X86VMS OPENVMS E9.2-3
VSI X86VMS VMS E9.2-3 Oper System Installed VSI X86VMS DECNET_PLUS V9.2-G
VSI X86VMS DWMOTIF V1.8-1
VSI X86VMS KERBEROS V3.3-3
VSI X86VMS OPENVMS E9.2-3
VSI X86VMS TCPIP V6.0-25
------------------------------------ ----------- --------- ------------------------------------ ------------------------------------
14 items found
Juhani> dir/siz=All 2024*

Directory DEV$SUPPORT:[KORKKO]

20241015_SECURITY_SERVER_ISSUE.DMP;1
6490585/6490592

Total of 1 file, 6490585/6490592 blocks.
Juhani> write sys$output 6490592/2048
3169

[volkerhalle]

Veli,

a CLUE file does not help analyze a forced crash. All you can see is that SECURITY_SERVER is the current process.

You need someone with access to the SECURITY_SERVER source listings AND access to the forced crash, to try and determine, why SECURITY_SERVER was looping by looking at the stack, registers etc.

Raise a call to VSI, if you can.

Volker.

[dougf]

If you bring up SDA, set process as the security server, and post the results from show call/summary, I could take a stab at it.

Doug.

[goathunter]

I've experienced this same problem on EISNER, which is an Alpha system, three times now.

The next time it happens, I'll get the CALL/SUMMARY....

[dougf]

Question: Are these non-x86 systems under relatively constant attack?

I ask, because the security_server has a thread that cleans ups up the intrusion database periodically; perhaps after 30 minutes of idle processing. A system that is under constant attack may not hit the conditions for running cleanup. The end result can be a very large database in the logical tables that increases the workload of the server for every transaction and results in much slower response. If the system reaches the resource limit for the logical table, things could even stall.

A "$ show intrusion" command will trigger the cleanup. The speed of the response to this command corresponds to the size of the database. Larger database, slower response.

A workaround for this issue is to create a batch job that runs periodically that issues the above command.

This problem has been addressed in the new server written for X86. The intent is to make the new server available to all VSI versions of OpenVMS in a future update.

I'm very curious about any issues with the new server on X86. Please report any issues to VSI so we can attend to them promptly.

[goathunter]

Yes, they come and go, but each time it's happened on EISNER, it had been under constant attack for quite a while. I assumed the database had gotten very large, but I had not considered that the cleanup code may not have run, leading to the problem. I'll implement your workaround on EISNER until the new server is rolled out to Alpha.

Thanks!

Hunter

Added in 6 minutes 13 seconds:
Sounds like you don't need it anymore, but after posting the above, I did a SHOW INTRUSION on EISNER and the server process took over. I did finally manage to get the SHOW CALL/SUMMARY on Alpha:

Call Frame Summary
------------------

Frame Type Frame Address Return PC Procedure Entry
-------------------- ----------------- ----------------- --------------
---
Stack Frame 00000000.7FF87D80 00000000.00000000 FFFFFFFF.8080E020 SECURESHRP+3E020
Stack Frame 00000000.005542A0 00000000.00039A40 00000000.000360F8 SECURITY_SERVER+360F8
Stack Frame 00000000.00559380 00000000.000318CC 00000000.00039340 SECURITY_SERVER+39340
Stack Frame 00000000.00565980 00000000.00031C38 00000000.000317A8 SECURITY_SERVER+317A8
Stack Frame 00000000.00571B30 00000000.00032080 00000000.00031A48 SECURITY_SERVER+31A48
Stack Frame 00000000.00575C10 00000000.7C0BD694 00000000.00032050 SECURITY_SERVER+32050
Stack Frame 00000000.00575C30 FFFFFFFF.80A4E74C 00000000.7C0BD290 ADARTL+3D290
Stack Frame 00000000.7AFD9760 00000000.0008DB0C 00000000.00030000 SYS$K_VERSION_03
Stack Frame 00000000.7AFD97C0 FFFFFFFF.80A4E728 00000000.0008DE50 SECURITY_SERVER+8DE50
Stack Frame 00000000.7AFD9820 FFFFFFFF.80A28444 FFFFFFFF.80A4E480 PTHREAD$RTL+56480
Stack Frame 00000000.7AFD9A20 FFFFFFFF.8034C964 FFFFFFFF.80A28260 PTHREAD$RTL+30260
Stack Frame 00000000.7AFDDB00 FFFFFFFF.80148104 FFFFFFFF.8034C7D0 IMAGE_MANAGEMENT+147D0

[vkorkko]

yes, this system is being attacked all the time. And to keep defences up I have rather nasty LGI settings. We remember each failed attempt for something liek say three hours, we start penalty from third attempt and suspension is somewhere from 48 upwards

So the intrusion db tends to quite a few entries. Actually I had to enlarge paged dynamic memory (?)
because the intrusion database logical names caused me to run out of pool..

_veli

[sms]

Code: Select all

> [...] on EISNER, it had been under constant attack for quite a while.
> [...]

> [...] this system is being attacked all the time. [...]

   What _kind_ of attacks?

   Years ago, I configured port forwarding on my router to move SSH
away from (external) port 22, and that solved the worst intrusion
problem I ever had.

[imiller]

Eisner does not use port 22 for SSH

[vkorkko]

Well, somebody clever enough found my alternate SSH port already to bombard me therein also.

I have now put in place some processing of $SHOW INTRUSION and putting worst offenders onto mandatory pause using $TCPIP SET SERVICE SSHD22 /reject=host and sometimes using also /reject=network as well.

These are rather restrictive though since /REJECT=HOST allows max 32 hosts listed and /REJECT=NETWORK allows mere 16 networks

_veli