Dovetailed Technologies

1. Basic Installation and Configuration

1.1 Introduction

This guide is designed to help systems programmers quickly install and configure "IBM Ported Tools for z/OS - OpenSSH". Steps are taken to ensure smooth operation and efficiency, including exploitation of ICSF and CPACF. Proper configuration of these features has been shown to reduce CPU consumption for Ported Tools SFTP or Co:Z SFTP by over 50%.

While the procedures in this document will work in most environments, users should reference the appropriate IBM documentation as appropriate. The primary reference is the IBM Ported Tools for z/OS: OpenSSH User's Guide, ("PTUG"). This guide will call out specific sections of the PTUG or other documents for additional information.

[Note]Updated for z/OS Ported Tools - OpenSSH v1.3

This version of the quick install guide has been updated for z/OS Ported Tools - OpenSSH v1.3, which became generally available in February 2015. Please refer to version 1.0.x of this guide (available on our website) if you are using z/OS Ported Tools - OpenSSH v1.2

In addition to the IBM documentation above, the following SHARE presentation outlines the new features in z/OS Ported Tools - OpenSSH v1.3 and covers many of the topics in this guide OpenSSH for z/OS: New features and functions.

Topics covered in this guide:

  • Prerequisites, service planning

  • Language Environment tuning considerations

  • ICSF support for secure random numbers via /dev/random

  • Configuration files, started task, etc.

  • z/OS Communications Server TCP/IP, Resolver and syslogd considerations

  • ICSF support for hardware accelerated ciphers and MACs

  • Managing the /tmp filesystem

Note: The included examples assume that you are running RACF as your system security product. IBM Ported Tools OpenSSH will also work with CA-ACF2 and CA-TSS, but you will be required to translate RACF commands as shown to those products. If you have one of those products and would like to contribute tested examples, please contact us.

1.2 Prerequisites

This guide assumes that you have or will be installing IBM Ported Tools OpenSSH release 1.3. Using this product and exploiting these features requires:

  • z/OS 1.13 or later

  • CPACF - processor feature 3863 (free and enabled by default in most countries)

  • ICSF installed and running (even if you don't have a co-processor card)

    • CPACF instructions are used by ICSF for Ciphers and MACS

    • ISCF level HCR7780 or later. HCR77A0 ("A0" level) and later has support for /dev/random without crypto card.

    • PTF for APAR 0A45548 ("AES CTR MODE ENCRYPTION SUPPORT")

1.3 Install / Service Planning

  • Order/Install latest release: "IBM Ported Tools for z/OS" 5665-M23 01.03.00

  • Review and install as appropriate ICSF and its required service.

1.4 Check file attributes and ownership

From a z/OS Unix shell, check the permissions and owner of the following directories:

$ ls -ld /etc/ssh /var/empty /var/run
drwxr-xr-x1  2 STC1  2  SYS1        8192 Feb 25 14:30 /etc/ssh
drwxr-xr-x   3 STC1     SYS1        8192 Feb 21  2013 /var/empty
drwxr-xr-x   2 STC1     SYS1        8192 Jan 29 15:09 /var/run
	

Check the permissions, extended attributes, and owner of the following files:

$ ls -El /usr/sbin/sshd
-rwxr--r--1 ap--  2 STC1  2  SYS1     7823360 Feb 25 14:30 /usr/sbin/sshd

$ ls -El /bin/ssh* /bin/scp /bin/sftp                                                                                               
-rwxr-xr-x  a-s-3 2 STC1     SYS1     6041600 Feb 25 14:30 /bin/scp
-rwxr-xr-x  a-s-  2 STC1     SYS1     6180864 Feb 25 14:30 /bin/sftp
-rwxr-xr-x  ----  2 STC1     SYS1     7536640 Feb 25 14:30 /bin/ssh
-rwxr-xr-x  --s-  2 STC1     SYS1     5693440 Feb 25 14:30 /bin/ssh-add
-rwxr-xr-x  --s-  2 STC1     SYS1     5476352 Feb 25 14:30 /bin/ssh-agent
-rwxr-xr-x  --s-  2 STC1     SYS1     5918720 Feb 25 14:30 /bin/ssh-keygen
-rwxr-xr-x  --s-  2 STC1     SYS1     6070272 Feb 25 14:30 /bin/ssh-keyscan

$ ls -El /usr/lib/ssh 
drwxr-xr-x        2 STC1     SYS1        8192 Oct 22  2011 IBM
-rwxr-xr-x  a-s-  2 STC1     SYS1     1122304 Feb 25 14:30 sftp-server
-rwxr-xr-x  --s-  2 STC1     SYS1     3866624 Feb 25 14:30 ssh-askpass
-rwsr-xr-x  ----  2 STC1     SYS1     6418432 Feb 25 14:30 ssh-keysign 
1

The permissions bits should match this column.

2

The owner must be UID=0; one of your UID=0 userids should be displayed.

3

The extended attributes should match this column. a="APF authorized" p="Program Controlled" s="allow shared address space"

Reference: PTUG: "Steps for verifying the prerequisites for using OpenSSH"

1.5 Language Environment Tuning

IBM Ported Tools z/OS uses the LE XPLINK libraries, and IBM recommends the following:

  • Add SCEELPA to LPALST

  • Add SCEERUN and SCEERUN2 to LNKLST

  • SCEERUN and SCEERUN2 must be program controlled

  • Implement samples SCEESAMP(CEEWLPA) and SCEESAMP(EDCWLPA). We recommend implementing both of these as shipped.

Note: IBM Ported Tools OpenSSH will still run if recommended XPLINK modules are not placed in LPA. This is something that you can defer for your next system maintenance window.

References:

  • PTUG: "Setting up the XPLINK environment for use by IBM Ported Tools for z/OS: OpenSSH" (Chapter 4)

  • Language Environment Customization "Placing Language Environment modules in link pack and LIBPACK"

1.6 Using ICSF and /dev/random

Generation of secure random numbers is key to using OpenSSH (or any cryptographic tool). IBM Ported Tools OpenSSH requires a working /dev/random device in order to run (the obsolete alternative ssh-rand-helper has been removed from OpenSSH). On z/OS Unix, /dev/random is provided by ICSF's CSFRNG service. In the past this required that you have a co-processor card, but with the "A0" or later level of ICSF (HCR77A0/A1) you don't need a co-processor card - ICSF will generate a cache of secure random numbers using CPACF instructions as appropriate.

Without /dev/random support, OpenSSH will fail to start with the following message:

FOTS1949 PRNG is not seeded. Please activate the 
Integrated Cryptographic Service Facility (ICSF)

Assuming that ICSF is running and supports the CSFRNG service, all you need to do is to authorize your users to this service. For most environments, it will be acceptable to permit all users to the CSFRNG service:

 RDEFINE CSFSERV CSFRNG UACC(NONE)
 PERMIT CSFRNG CLASS(CSFSERV) ID(*) ACCESS(READ)
 SETROPTS RACLIST(CSFSERV) REFRESH	

To verify that /dev/random is working, issue this command from a z/OS UNIX shell and userid with normal priviledges (and CSFRNG access). This should display some random data in hex:

$ head /dev/random | od -x
	

Reference: PTUG: "Using hardware support to generate random numbers"

1.7 Creating configuration files

Copy the sample configuration files to the /etc/ssh directory. You must use a UID=0 userid for this:

$ cd /samples
$ cp -p moduli          /etc/ssh
$ cp -p ssh_config      /etc/ssh
$ cp -p sshd_config     /etc/ssh
$ cp -p zos_ssh_config  /etc/ssh
$ cp -p zos_sshd_config /etc/ssh	

Note: All of the above files in /etc/ssh should be owned by a UID=0 userid and have permissions 644:

-rw-r--r--   1 STC1     SYS1      242153 Jan 15 17:08 moduli
-rw-r--r--   1 STC1     SYS1          3483 Jan 15 17:08 ssh_config
-rw-r--r--   1 STC1     SYS1          4685 Jan 15 17:08 sshd_config
-rw-r--r--   1 STC1     SYS1          1158 Jan 15 17:08 zos_ssh_config
-rw-r--r--   1 STC1     SYS1          1209 Jan 15 17:08 zos_sshd_config

Reference: PTUG: "Steps for creating or editing configuration files"

1.8 Creating SSHD server keys

You must generate one or more public/private key pairs that are used for authentication of your SSHD server. Each client that connects to the server will either already have one of the public keys (aka "host fingerprint") or will be required to accept your server's public key as proof of the server's identity.

For more information on SSH key authentication, see the recording of our webinar: IBM Ported Tools for z/OS: Key Authentication

Server keys can be stored either in protected UNIX files or in SAF/RACF keyrings. Most installations will choose to use files, which is covered below. For information on how to use SAF/RACF keyrings, see our webinar: IBM Ported Tools for z/OS: Using Key Rings

The following commands can be executed by a UID=0 userid to create all variants of the server key pairs. It will not overwrite a key of a given type, so it can be used safely:

$ cd /etc/ssh
$ ssh-keygen -A
ssh-keygen: generating new host keys: RSA1 RSA DSA ECDSA

This should result in four pairs of private/public key files with the following permissions, all owned by a UID=0 userid:

$ ls -al *key*
-rw-------   1 STC1     SYS1         668 Feb 25 15:00 ssh_host_dsa_key
-rw-r--r--   1 STC1     SYS1         606 Feb 25 15:00 ssh_host_dsa_key.pub
-rw-------   1 STC1     SYS1         227 Feb 25 15:00 ssh_host_ecdsa_key
-rw-r--r--   1 STC1     SYS1         178 Feb 25 15:00 ssh_host_ecdsa_key.pub
-rw-------   1 STC1     SYS1         981 Feb 25 15:00 ssh_host_key
-rw-r--r--   1 STC1     SYS1         646 Feb 25 15:00 ssh_host_key.pub
-rw-------   1 STC1     SYS1        1679 Feb 25 15:00 ssh_host_rsa_key
-rw-r--r--   1 STC1     SYS1         398 Feb 25 15:00 ssh_host_rsa_key.pub

Reference: PTUG: "Steps for setting up server authentication when keys are stored in UNIX files"

1.9 Set up SSHD server userids

Your SSHD server will use two SAF/RACF userids (besides the actual userids that clients sign on with):

  1. The privileged UID=0 userid used to start the started task (here we use "SSHDAEM")

  2. The unprivileged "privilege separation" userid (this must be "SSHD", or have an alias of "SSHD")

The privileged started task userid can be an existing UID=0 userid, like OMVSKERN, but we recommend creating a new userid, defined like OMVSKERN:

  ADDUSER SSHDAEM DFLTGRP(OMVSGRP)
    OMVS(UID(0) HOME('/') PROGRAM('/bin/sh'))
    NOPASSWORD  

The privileged started task userid must have read access to BPX.DAEMON -

  RDEFINE FACILITY BPX.DAEMON UACC(NONE)
  PERMIT BPX.DAEMON CLASS(FACILITY) ID(SSHDAEM) ACCESS(READ)
  SETROPTS RACLIST(FACILITY) REFRESH 

Note: If your system has the FACILITY/BPX.POE defined, then the SSHDAEM userid will require READ access. If the SERVAUTH class is active, then the SSHDAEM userid will require authorization to access incoming socket connections.

Create the unprivileged "privilege separation" userid, where ggg is an unused groupid and uuu is an unused non-zero uid (you may alternatively use the AUTOGID and AUTOUID keywords if you have enabled the BPX.NEXT.USER profile) -

  ADDGROUP SSHDG OMVS(GID(ggg))
  ADDUSER SSHD DFLTGRP(SSHDG) OMVS(UID(uuu) HOME(’/var/empty’)
    PROGRAM(’/bin/false’)) NOPASSWORD  	

Reference: PTUG: "Step for creating the sshd privilege separation user" and "Starting sshd as a stand-alone daemon"

1.10 Create SSHD server started task

The best way to start your SSHD server is by using a started task proc:

//SSHD PROC                                                   
//SSHD EXEC PGM=BPXBATCH,REGION=0M,TIME=NOLIMIT,              
//     PARM='PGM /bin/sh -c /etc/ssh/sshd.sh'                 
//STDERR DD SYSOUT=*   
//  

This proc executes a shell script /etc/ssh/sshd.sh that you must create:

#!/bin/sh
export NLSPATH=$NLSPATH:/usr/lib/nls/msg/%L/%N.cat
export _EDC_ADD_ERRNO2=1
nohup /usr/sbin/sshd -f /etc/ssh/sshd_config & 
sleep 1

This script should have permissions "700" and be owned by a UID=0 userid.

-rwx------   1 STC1     SYS1         141 Feb 26  2013 sshd.sh

The SSHD started task must be configured to start with the privileged userid "SSHDAEM" that you setup in the prior section:

SETROPTS GENERIC(STARTED)
RDEFINE STARTED SSHD.* 
  STDATA(USER(SSHDAEM) GROUP(OMVSGRP) TRUSTED(NO))
SETROPTS RACLIST(STARTED) REFRESH 

To start sshd, issue the following MVS command:

S SSHD

Verify that the proper userid and group were assigned to the SSHD started task by examining the system log:

S SSHD                                                            
$HASP100 SSHD     ON STCINRDR                                     
IEF695I START SSHD     WITH JOBNAME SSHD     IS ASSIGNED TO USER  
SSHDAEM, GROUP OMVSGRP                                           
$HASP373 SSHD     STARTED                                         
$HASP395 SSHD     ENDED 

Note: like FTPD, this started task will quickly terminate, but it will spin off an OMVS address space with jobname "SSHDn".

References:

  • PTUG: "Starting sshd as a stand-alone daemon"

  • PTUG: "Ways to start sshd as a stand-alone daemon" / "Using BPXBATCH"

1.11 TCP configuration

Using the default sshd_config settings, SSHD listens on port 22 on all stacks. Since this is a privileged port number, only programs running as superuser are allowed to listen on this port. We recommend that for your next IPL that you also reserve this port in PROFILE.TCPIP, since it also serves to document usage. In the following example template, we also cause the SSHD started task to be started after TCPIP has started:

AUTOLOG 
...
    SSHD                 ; SSHD Server (STC SSHDn)   
...
ENDAUTOLOG
...
PORT
...
    22 TCP SSHD* NOAUTOLOG  ; Ported Tools SSHD server
... 	

Reference: PTUG: "Steps for creating or editing configuration files" Step 4

1.12 Verify z/OS DNS / Resolver operation

The ssh client will perform DNS lookups on target host names. Also, by default, the sshd server will lookup remote host names and check that the resolved name maps back to the IP address (warnings will be logged otherwise) . If the z/OS DNS client (the "Resolver") is not working properly, these requests might hang before timing out.

A full z/OS DNS server is not required to run IBM Ported Tools OpenSSH, but most shops will want to run a "caching-only server" connected to their corporate DNS servers. At minimum you should at least configure the z/OS resolver so that DNS requests do not hang.

To verify that your z/OS resolver is working properly, issue the following command from a z/OS UNIX shell. Try known and unknown host names to verify that neither hang:

$ host www.ibm.com
EZZ8321I e3062.x.akamaiedge.net has addresses 23.67.232.41
EZZ8322I aliases: www.ibm.com, www.ibm.com.cs186.net, www.ibm.com.edgekey.net

$ host h42444.not-a-domain.com                                                                                     
EZZ8342I h12345.not-a-domain.com: Unknown host 

References:

1.13 Configuring the syslogd daemon

The IBM Ported Tools SSHD server will log messages to the z/OS Communications Server syslogd facility using a local UNIX datagram socket (/dev/log).

IBM FTPD, TELNETD, IDS, and other z/OS Comm Server applications use syslogd for logging and tracing as well. Although some of these z/OS Comm Server applications will log to the z/OS console if syslogd is not running, IBM Ported Tools SSHD server will not. Therefore, it is important to have syslogd running in local mode (-i) so that SSHD server log messages are available.

Starting with z/OS 1.11, significant enhancements were made to syslogd -

  • performance improvements

  • improved operator interfaces

  • automatic archival to MVS data sets

  • ISPF syslogd viewer

IBM Ported Tools SSHD server by default will log all messages with INFO severity or higher to the local syslogd "AUTH" facility.

You should verify that syslogd is configured and started on your z/OS system and that your /etc/syslog.conf file is configured so that messages to the AUTH facility will be logged to some file. For example:

# log any messages to AUTH facility (sshd)
auth.* /tmp/syslogd.auth.log -X 

References:

1.14 Verify basic functionality

Before we move on, lets verify the basic functionality of your SSHD server. To do this, you will need to install an SSH client, like PuTTY on your workstation on a network that can connect to your z/OS system on port 22.

Note: You will want to test an ssh session to an unprivileged z/OS userid that has an OMVS segment and home directory. If you have a non UID=0, unprivileged TSO userid that can get into the TSO OMVS shell, then use that.

C:> putty zosuser@my.zos.host 
 - or -
C:> putty zosuser@10.2.3.4 
Using username "zosuser".
zosuser@my.zos.host's password: ******
/u/home/zosuser>    

If successful, the above will place you in an SSH session with an interactive z/OS UNIX login shell. Once you have a non-TSO UNIX shell, you can use the ssh client command to connect to other hosts. For example, you can also connect to the same z/OS system using the loopback address:

/u/home/zosuser> ssh zosuser@127.0.0.1
The authenticity of host '127.0.0.1 (127.0.0.1)' can't be established.
RSA key fingerprint is 22:cb:9c:30:9d:98:c8:4f:45:a8:ac:00:e5:8e:62:af.
Are you sure you want to continue connecting (yes/no)? yes
zosuser@my.zos.host's password: ******
/u/home/zosuser>
/u/home/zosuser> exit
/u/home/zosuser> (back to the first connection) 

Note: Starting with the v1.3 release of IBM Ported Tools for z/OS - OpenSSH, the ssh client will now run under a TSO OMVS shell. When run this way, password authetication is not allowed, but other non-interactive authentication methods can be used (e.g. public_key).

Now is also a good time to verify your syslogd configuration. Look at the end of the logfile that you configured in /etc/syslog.conf:

$ tail /tmp/syslogd.auth.log
...
Feb 14 21:11:23 S0W1 sshd[67174502]: Port of Entry information retained for ... 
Feb 14 21:11:24 S0W1 sshd[67174502]: Accepted password for zosuser from ...

Copyright© 2009-2017 Dovetailed Technologies, LLC. All rights reserved.
Co:Z® is a registered trademark of Dovetailed Technologies, LLC.