Dovetailed Technologies

1. Basic Installation and Configuration

1.1 Introduction

This guide is designed to help systems programmers quickly configure z/OS - OpenSSH. This guide assumes that you are running either z/OS V2R4, or that you have installed APAR OA54299 with z/OS V2R2 or V2R3.

With this support, IBM z/OS OpenSSH will directly use CPACF instructions, when present, to implement common symmetric ciphers and MAC algorithms. This configuration is preferred over our prior recommendation to use ICSF for ciphers and MACs.

While the procedures in this document will work in most environments, users should reference the appropriate IBM documentation as appropriate. The primary reference is the z/OS OpenSSH User's Guide. This guide will call out specific sections of the User's Guide or other documents for additional information. Please reference the specific manual for your installed version, and note that chapter numbers and section names may vary somewhat between releases.

[Note]Updated for OpenSSH running on z/OS V2R4, or with APAR OA54299 on V2R2 or V2R3

This version of the quick install guide has been updated specifically for the the new functionality added to OpenSSH with this APAR for CPACF support.

In addition to the IBM documentation above, the following SHARE presentation outlines the new features in z/OS V2R4: OpenSSH for z/OS V2R4: New features and functions.

Topics covered in this guide:

  • Prerequisites, service planning

  • Language Environment tuning considerations

  • Setup for secure random numbers via /dev/random

  • Configuration files, started task, etc.

  • z/OS Communications Server TCP/IP, Resolver and syslogd considerations

  • CPACF support for hardware accelerated ciphers and MACs

  • Managing the /tmp filesystem

Note: The included examples assume that you are running RACF as your system security product. z/OS OpenSSH will also work with CA-ACF2 and CA-TSS, but you will be required to translate RACF commands as shown to those products. If you have one of those products and would like to contribute tested examples, please contact us.

1.2 Prerequisites

This guide assumes that you are running OpenSSH on z/OS V2V2 or later. Using this product and exploiting these features requires:

  • z/OS V2R2, V2R3, or V2R4. (APAR OA54299 is required for V2R2 or V2R3)

  • CPACF: processor feature 3863 (free and enabled by default in most countries)

  • /dev/random support: either a z14 with APAR OA55437 installed prior to V2R4, or ICSF installed and running (a co-processor card is not required)

1.3 Install / Service Planning

  • Review and install as appropriate any service for OpenSSH (HOS2220, HOS2230, or HOS2240). See the appropriate upgrade bucket (Subset ZOSOSSH)

  • Be sure to install the PTF for APAR OA54299 prior to z/OS V2R4.

  • If required (see above), install ICSF and its required service.

1.4 Check file attributes and ownership

From a z/OS Unix shell, check the permissions and owner of the following directories:

$ ls -ld /etc/ssh /var/empty /var/run
drwxrwxrwx1  2 STC1  2 SYS1        ...   /etc/ssh
drwxr-xr-x   3 STC1     SYS1        ...   /var/empty
drwxr-xr-x   2 STC1     SYS1        ...   /var/run
	

Check the permissions, extended attributes, and owner of the following files:

$ ls -El /usr/sbin/sshd
-rwxr--r--1 ap--  2 STC1  2  SYS1     ...   /usr/sbin/sshd

$ ls -El /bin/ssh* /bin/scp /bin/sftp                                                                                               
-rwxr-xr-x  -p--3 2 STC1     SYS1     ...   /bin/scp
-rwxr-xr-x  -p--  2 STC1     SYS1     ...   /bin/sftp
-rwxr-xr-x  -p--  2 STC1     SYS1     ...   /bin/ssh
-rwxr-xr-x  --s-  2 STC1     SYS1     ...   /bin/ssh-add
-rwxr-xr-x  --s-  2 STC1     SYS1     ...   /bin/ssh-agent
-rwxr-xr-x  --s-  2 STC1     SYS1     ...   /bin/ssh-keygen
-rwxr-xr-x  --s-  2 STC1     SYS1     ...   /bin/ssh-keyscan
-rwxr-xr-x  --s-4 2 STC1     SYS1     ...   /bin/ssh-proxyc

$ ls -El /usr/lib/ssh 
drwxr-xr-x        2 STC1     SYS1     ...   IBM
-rwxr-xr-x  -p--  2 STC1     SYS1     ...   sftp-server
-rwxr-xr-x  --s-  2 STC1     SYS1     ...   ssh-askpass
-rwsr-xr-x  ----  2 STC1     SYS1     ...   ssh-keysign
-rwxr-xr-x  aps-  2 STC1     SYS1     ...   zsshgss.so 
1

The permissions bits should match this column.

2

The owner must be UID=0; one of your UID=0 userids should be displayed.

3

The extended attributes should match this column. a="APF authorized" p="Program Controlled" s="allow shared address space"

4

Only for z/OS V2R4 or later

Reference: z/OS OpenSSH User's Guide: "Steps for verifying the prerequisites for using OpenSSH"

1.5 Language Environment Tuning

OpenSSH uses the LE XPLINK libraries, and IBM recommends the following:

  • Add SCEELPA to LPALST

  • Add SCEERUN and SCEERUN2 to LNKLST

  • Add SCEERUN and SCEERUN2 to LLA

  • SCEERUN and SCEERUN2 must be program controlled

  • Implement samples SCEESAMP(CEEWLPA) and SCEESAMP(EDCWLPA). We recommend implementing both of these as shipped.

Note: OpenSSH will still run if recommended XPLINK modules are not placed in LPA. This is something that you can defer for your next system maintenance window.

References:

1.6 Using ICSF and /dev/random

Generation of secure random numbers is key to using OpenSSH (or any cryptographic tool). OpenSSH requires a working /dev/random device in order to run. On z/OS Unix, /dev/random is provided by either:

  • ICSF's CSFRNG service, or

  • a z14 processor and either z/OSV2R4 or the PTF for APAR OA55437.

Without /dev/random support, OpenSSH will fail to start with the following message:

FOTS1949 PRNG is not seeded. Please activate the 
Integrated Cryptographic Service Facility (ICSF)

If you are running on a pre-z14 processor and using ICSF to provide this support, then you must permit your users to use the CSFRNG service. For most environments, it will be acceptable to permit all users to the CSFRNG service:

 RDEFINE CSFSERV CSFRNG UACC(NONE)
 PERMIT CSFRNG CLASS(CSFSERV) ID(*) ACCESS(READ)
 SETROPTS RACLIST(CSFSERV) REFRESH	

To verify that /dev/random is working, issue this command from a z/OS UNIX shell and userid with normal priviledges (and CSFRNG access). This should display some random data in hex:

$ head /dev/random | od -x

Reference: z/OS OpenSSH User's Guide: "Using hardware support to generate random numbers"

1.7 Creating configuration files

Copy the sample configuration files to the /etc/ssh directory.

[Note]Note:
If you are running a previous version of OpenSSH you will want to review "Migration" items in the z/OS OpenSSH User's guide. Before copying the IBM sample configuration, backup your existing configuration files and compare the differences between your current configuration files and these samples to see if any site-specific configuration options should be migrated to the new release. In particular, if you have updated zos_ssh_config or zos_sshd_config to add CiphersSource and MACsSource keywords, you will probably want to remove these. See section Chapter 2, Exploiting crypto hardware acceleration for more information.

Note: You must use a UID=0 userid for this:

$ cd /samples
$ cp -p moduli          /etc/ssh
$ cp -p ssh_config      /etc/ssh
$ cp -p sshd_config     /etc/ssh
$ cp -p zos_ssh_config  /etc/ssh
$ cp -p zos_sshd_config /etc/ssh	

Note: All of the above files in /etc/ssh should be owned by a UID=0 userid and have permissions 644:

-rw-r--r--   1 STC1     SYS1       ...  moduli
-rw-r--r--   1 STC1     SYS1       ...  ssh_config
-rw-r--r--   1 STC1     SYS1       ...  sshd_config
-rw-r--r--   1 STC1     SYS1       ...  zos_ssh_config
-rw-r--r--   1 STC1     SYS1       ...  zos_sshd_config

Reference: z/OS OpenSSH User's Guide: "Steps for creating or editing configuration files"

Special considerations for migrating to V2R4 or later

We recommend that users follow the recommendations in the z/OS OpenSSH User's Guide: "Migrating to z/OS OpenSSH Version 2 Release 4". Following is one exception regarding the subsection: "DSA (ssh-dss) key support":

It is best to migrate from using weaker "ssh-dss" keys rather the enabling them as shown in this section. If you wish to enable ssh-dsa support, we do not advise that you uncomment the following line in /etc/ssh/ssh_config as suggested by IBM in this chapter, since this would change the way that the ssh client negotiates Host Key algorithms in a way that can cause "Remote Host Identification Has Changed" failures even if ssh-dss host keys are not being used.

# in /etc/ssh/ssh_config, do *not* uncomment this:
#HostKeyAlgorithms +ssh-dss

Instead, we recommend that users scan their /etc/ssh/ssh_known_hosts and user $HOME/.ssh/known_hosts files for "ssh-dss" key types and replace these with supported key types for the affected hosts. Co:Z Enterprise Licence and Support customers are advised to open a support ticket and we will assist you.

1.8 Creating SSHD server keys

You must generate one or more public/private key pairs that are used for authentication of your SSHD server. Each client that connects to the server will either already have one of the public keys (aka "host fingerprint") or will be required to accept your server's public key as proof of the server's identity.

For more information on SSH key authentication, see the recording of our webinar: IBM Ported Tools for z/OS: Key Authentication

Server keys can be stored either in protected UNIX files or in SAF/RACF keyrings. Most installations will choose to use files, which is covered below. For information on how to use SAF/RACF keyrings, see our webinar: IBM Ported Tools for z/OS: Using Key Rings

The following commands can be executed by a UID=0 userid to create all variants of the server key pairs. It will not overwrite existing keys of a given type, so it can be used safely:

$ cd /etc/ssh
$ ssh-keygen -A
ssh-keygen: generating new host keys: RSA1 RSA DSA ECDSA

This should result in four pairs of private/public key files with the following permissions, all owned by a UID=0 userid:

$ ls -al *key*
-rw-------   1 STC1     SYS1       ...  ssh_host_dsa_key
-rw-r--r--   1 STC1     SYS1       ...  ssh_host_dsa_key.pub
-rw-------   1 STC1     SYS1       ...  ssh_host_ecdsa_key
-rw-r--r--   1 STC1     SYS1       ...  ssh_host_ecdsa_key.pub
-rw-------   1 STC1     SYS1       ...  ssh_host_key
-rw-r--r--   1 STC1     SYS1       ...  ssh_host_key.pub
-rw-------   1 STC1     SYS1       ...  ssh_host_rsa_key
-rw-r--r--   1 STC1     SYS1       ...  ssh_host_rsa_key.pub

[Note]Note

We recommend against creating new DSA (ssh-dss) type host keys. If new files ssh_host_dsa_key and ssh_host_dsa_key.pub are created by the ssh-keygen -A command above, we recommend that you remove them. See section the section called “Special considerations for migrating to V2R4 or later” for more information.

1.9 Set up SSHD server userids

Your SSHD server will use two SAF/RACF userids (besides the actual userids that clients sign on with):

  1. The privileged UID=0 userid used to start the started task (here we use "SSHDAEM")

  2. The unprivileged "privilege separation" userid (this must be "SSHD", or have an alias of "SSHD")

The privileged started task userid can be an existing UID=0 userid, like OMVSKERN, but we recommend creating a new userid, defined like OMVSKERN:

  ADDUSER SSHDAEM DFLTGRP(OMVSGRP)
    OMVS(UID(0) HOME('/') PROGRAM('/bin/sh'))
    NOPASSWORD  

The privileged started task userid must have read access to BPX.DAEMON -

  RDEFINE FACILITY BPX.DAEMON UACC(NONE)
  PERMIT BPX.DAEMON CLASS(FACILITY) ID(SSHDAEM) ACCESS(READ)
  SETROPTS RACLIST(FACILITY) REFRESH 

Note: If your system has the FACILITY/BPX.POE defined, then the SSHDAEM userid will require READ access. If the SERVAUTH class is active, then the SSHDAEM userid will require authorization to access incoming socket connections.

Create the unprivileged "privilege separation" userid, where ggg is an unused groupid and uuu is an unused non-zero uid (you may alternatively use the AUTOGID and AUTOUID keywords if you have enabled the BPX.NEXT.USER profile) -

  ADDGROUP SSHDG OMVS(GID(ggg))
  ADDUSER SSHD DFLTGRP(SSHDG) OMVS(UID(uuu) HOME(’/var/empty’)
    PROGRAM(’/bin/false’)) NOPASSWORD  	

Reference: z/OS OpenSSH User's Guide: "Step for creating the sshd privilege separation user" and "Starting sshd as a stand-alone daemon"

1.10 Create SSHD server started task

The best way to start your SSHD server is by using a started task proc:

//SSHD PROC                                                   
//SSHD EXEC PGM=BPXBATCH,REGION=0M,TIME=NOLIMIT,              
//     PARM='PGM /bin/sh -c /etc/ssh/sshd.sh'                 
//STDERR DD SYSOUT=*   
//  

This proc executes a shell script /etc/ssh/sshd.sh that you must create:

#!/bin/sh
export NLSPATH=$NLSPATH:/usr/lib/nls/msg/%L/%N.cat
export _EDC_ADD_ERRNO2=1
export _CEE_RUNOPTS="HEAP(4M,1M,ANYWHERE,FREE)"
export _CEE_REALLOC_CONTROL=32K,25
nohup /usr/sbin/sshd -f /etc/ssh/sshd_config & 
sleep 1

This script should have permissions "700" and be owned by a UID=0 userid.

-rwx------   1 STC1     SYS1         141 Feb 26  2013 sshd.sh

The SSHD started task must be configured to start with the privileged userid "SSHDAEM" that you setup in the prior section:

SETROPTS GENERIC(STARTED)
RDEFINE STARTED SSHD.* 
  STDATA(USER(SSHDAEM) GROUP(OMVSGRP) TRUSTED(NO))
SETROPTS RACLIST(STARTED) REFRESH 

To start sshd, issue the following MVS command:

S SSHD

Verify that the proper userid and group were assigned to the SSHD started task by examining the system log:

S SSHD                                                            
$HASP100 SSHD     ON STCINRDR                                     
IEF695I START SSHD     WITH JOBNAME SSHD     IS ASSIGNED TO USER  
SSHDAEM, GROUP OMVSGRP                                           
$HASP373 SSHD     STARTED                                         
$HASP395 SSHD     ENDED 

Note: like FTPD, this started task will quickly terminate, but it will spin off an OMVS address space with jobname "SSHDn".

References:

1.11 TCP configuration

Using the default sshd_config settings, SSHD listens on port 22 on all stacks. Since this is a privileged port number, only programs running as superuser are allowed to listen on this port. We recommend that for your next IPL that you also reserve this port in PROFILE.TCPIP, since it also serves to document usage. In the following example template, we also cause the SSHD started task to be started after TCPIP has started:

AUTOLOG 
...
    SSHD                 ; SSHD Server (STC SSHDn)   
...
ENDAUTOLOG
...
PORT
...
    22 TCP SSHD* NOAUTOLOG  ; OpenSSH SSHD server
... 	

Reference: z/OS OpenSSH User's Guide: "Steps for creating or editing configuration files" Step 4

1.12 Verify z/OS DNS / Resolver operation

The ssh client will perform DNS lookups on target host names. Also, by default, the sshd server will lookup remote host names and check that the resolved name maps back to the IP address (warnings will be logged otherwise) . If the z/OS DNS client (the "Resolver") is not working properly, these requests might hang before timing out.

A full z/OS DNS server is not required to run OpenSSH, but most shops will want to run a "caching-only server" connected to their corporate DNS servers. At minimum you should at least configure the z/OS resolver so that DNS requests do not hang.

To verify that your z/OS resolver is working properly, issue the following command from a z/OS UNIX shell. Try known and unknown host names to verify that neither hang:

$ host www.ibm.com
EZZ8321I e3062.x.akamaiedge.net has addresses 23.67.232.41
EZZ8322I aliases: www.ibm.com, www.ibm.com.cs186.net, www.ibm.com.edgekey.net

$ host h42444.not-a-domain.com                                                                                     
EZZ8342I h12345.not-a-domain.com: Unknown host 

References:

1.13 Configuring the syslogd daemon

SSHD server will log messages to the z/OS Communications Server syslogd facility using a local UNIX datagram socket (/dev/log).

IBM FTPD, TELNETD, IDS, and other z/OS Comm Server applications use syslogd for logging and tracing as well. Although some of these z/OS Comm Server applications will log to the z/OS console if syslogd is not running, the OpenSSH SSHD server will not. Therefore, it is important to have syslogd running in local mode (-i) so that SSHD server log messages are available.

Starting with z/OS 1.11, significant enhancements were made to syslogd -

  • performance improvements

  • improved operator interfaces

  • automatic archival to MVS data sets

  • ISPF syslogd viewer

The OpenSSH SSHD server by default will log all messages with INFO severity or higher to the local syslogd "AUTH" facility.

You should verify that syslogd is configured and started on your z/OS system and that your /etc/syslog.conf file is configured so that messages to the AUTH facility will be logged to some file. For example:

# log any messages to AUTH facility (sshd)
auth.* /tmp/syslogd.auth.log -X 

References:

1.14 Verify basic functionality

Before we move on, lets verify the basic functionality of your SSHD server. To do this, you will need to install an SSH client, like PuTTY on your workstation on a network that can connect to your z/OS system on port 22.

Note: You will want to test an ssh session to an unprivileged z/OS userid that has an OMVS segment and home directory. If you have a non UID=0, unprivileged TSO userid that can get into the TSO OMVS shell, then use that.

C:> putty zosuser@my.zos.host 
 - or -
C:> putty zosuser@10.2.3.4 
Using username "zosuser".
zosuser@my.zos.host's password: ******
/u/home/zosuser>    

If successful, the above will place you in an SSH session with an interactive z/OS UNIX login shell. Once you have a non-TSO UNIX shell, you can use the ssh client command to connect to other hosts. For example, you can also connect to the same z/OS system using the loopback address:

/u/home/zosuser> ssh zosuser@127.0.0.1
The authenticity of host '127.0.0.1 (127.0.0.1)' can't be established.
RSA key fingerprint is 22:cb:9c:30:9d:98:c8:4f:45:a8:ac:00:e5:8e:62:af.
Are you sure you want to continue connecting (yes/no)? yes
zosuser@my.zos.host's password: ******
/u/home/zosuser>
/u/home/zosuser> exit
/u/home/zosuser> (back to the first connection) 

Now is also a good time to verify your syslogd configuration. Look at the end of the logfile that you configured in /etc/syslog.conf:

$ tail /tmp/syslogd.auth.log
...
Feb 14 21:11:23 S0W1 sshd[67174502]: Port of Entry information retained for ... 
Feb 14 21:11:24 S0W1 sshd[67174502]: Accepted password for zosuser from ...

Copyright© 2009-2021 Dovetailed Technologies, LLC. All rights reserved.
Co:Z® is a registered trademark of Dovetailed Technologies, LLC.