Dovetailed Technologies

4. Using the Co:Z SFTP client

An enhanced sftp client (cozsftp) for z/OS is also included in the Co:Z toolkit. This client can be used to initiate transfers with a remote host and supports the same set of file transfer options as the Co:Z SFTP server. The cozsftp command is installed in the $COZ_HOME/bin directory.

4.1 Starting the Co:Z SFTP client on z/OS

$ export PATH=/opt/dovetail/coz/bin:$PATH 1
$ cozsftp user@host
Co:Z sftp version: 1.1.0 (5.0p1) 2008-10-20
Copyright (C) Dovetailed Technologies, LLC. 2008. All rights reserved.
Connecting to host...
user@host's password: ***** 
cozsftp>
1

Add the Co:Z binaries directory to your PATH. This is not necessary if symbolic links from /bin were created during installation.

4.2 Co:Z SFTP client logging

OpenSSH SFTP logging is enabled by setting the -v option on the command line. The logging level can be raised by increasing the number of v options from -v to -vvv. Additionally, the COZ_LOG environment variable can be used to set logging options for the Co:Z extension library used to add z/OS support to the cozsftp command. The default setting for COZ_LOG is I. To change this default for all users, modify /etc/ssh/cozsftp_client.rc as needed. Individual users can override this setting by exporting COZ_LOG on the command line or in the copy of cozsftp_client.rc in their individual .ssh directory.

Client logging information is sent to stderr. In order to capture logging information in a file, use the following:

$ export COZ_LOG=D; cozsftp -vvv user@hostname  2>&1  | tee ~/cozsftp-output.log

The Co:Z client loglevel can also be set using the lzopts command described in the next section. See Section B.2, “Miscellaneous options” for additional information.

For information on setting these logging options in batch, see the section called “Logging in batch”.

4.3 Setting, displaying and clearing file transfer options

The enhanced client introduces two new commands:

lzopts [-a] [option=value,...]

The lzopts command is used to set local (client) file transfer options. These options are set prior to initiating file/dataset transfers from z/OS to a remote host.

zopts [-a] [option=value,...]

The zopts command is used to set server file transfer options -- if the server is a Co:Z SFTP server. The zopts command is functionally equivalent to the ls /+<option_list> command used by existing clients to set Co:Z sftp-server file transfer options.

Multiple options can be set by separating the option=value pairs with commas. An error is returned if one or more of the options was incorrectly specified, but the remaining options are set as requested.

The active options and their settings can be displayed by issuing the commands without arguments. The -a option can be specified to list all available options, even those that are not active.

The client performs some shell-like processing of its commands. In general, this is not an issue for the setting of options, but if the supplied option value contains a hash symbol (#), the option=value pair must be quoted, either with single or double quotes. For example:

cozsftp> lzopts "dataclas=#MYCLASS"
[Note]Note

For compatibility with z/OS OpenSSH SFTP, the cozsftp command recognizes the following additional subcommands: ascii and binary. these subcommands are treated as synonyms for lzopts mode=text and lzopts mode=binary respectively.

Client session options are determined in the following priority order:

  1. The fixed: section of /etc/ssh/cozsftp_config (highest priority and non-modifiable)

  2. The first matching pattern (if any) from $HOME/.ssh/cozsftp_config

  3. The first matching pattern (if any) from /etc/ssh/cozsftp_config

  4. Previous interactive commands: lzopts (described below) in the same session

  5. The environment variable SFTP_ZOS_OPTIONS

  6. The default: section of /etc/ssh/cozsftp_config (lowest priority)

For a list of available options, see Appendix B, Co:Z SFTP options.

For a description of the cozsftp_config file format, including how to specify file name patterns, see Appendix C, Session config files.

Example: Setting and displaying local (client) transfer options

cozsftp> lzopts mode=text     1
 mode=text
cozsftp> lzopts     2
clientcp=IBM-1047    loglevel=I           mode=text     
servercp=IBM-1047
1

The local option command lzopts mode=text is used to set the transfer mode to text. mode=binary is the default.

2

The local option list command lzopts shows the options currently in effect. In this case, the codepages clientcp and servercp are set to the defaults.

Example: Setting multiple local options

cozsftp> lzopts lrecl=80,recfm=fb,space=trk.3.2     1
 lrecl=80,recfm=fb,space=trk.3.2
1

Multiple options can be specified, separated by commas. Note that the SPACE parameter uses periods for commas to avoid ambiguity.

Example: Showing all local options

cozsftp> lzopts -a      1
clientcp=IBM-1047  linerule=flexible  loglevel=I        lrecl=80            
mode=text          overflow=wrap      recfm=fb          servercp=IBM-1047   
space=trk.3.2      NOallowmount       NOblksize         NObufno             
NOcopies           NOdataclas         NOdest            NOdir               
NOdisp             NOdsorg            NOforms           NOgdgnt             
NOhold             NOlabel            NOlike            NOmaxvol            
NOmgmtclas         NOnorecall         NOoutdes          NOrelease           
NOretpd            NOsequence         NOshowall         NOspin              
NOstorclas         NOsysout           NOtrim            NOtrtch             
NOucount           NOunit             NOvol             NOwriter
1

The option command lzopts -a is used to show all of the available options, even those that are not currently active.

4.4 Coordinating Transfer Options with a Co:Z SFTP Server

The enhanced Co:Z SFTP client can connect to any sftp server, including a Co:Z SFTP server. In this case, there are two sets of transfer options in effect; the enhanced client's and the server's. Client side (local) options are controlled via the lzopts command. Server side (remote) options are controlled via the zopts command.

When transferring POSIX files between a z/OS server and z/OS client, using the default mode=binary transfer option both locally and remotely will usually yield the desired results. If codepage translations need to take place, the desired clientcp, servercp and mode=text can be set either locally (via lzopts command) or remotely (via the zopts command). The other side can be left in mode=binary.

When transferring datasets between a z/OS server and z/OS client, it is generally recommended that linerule=rdw be used for binary transfers so that record mode boundaries are preserved.

When converting from dataset to POSIX file between a z/OS server and z/OS client, the transfer options should be set where the dataset resides.

4.5 Working with Datasets

The Co:Z implementation of sftp accepts two prefix strings to identify MVS datasets as absolute paths. The first (//) is consistent with IBM's common usage. A secondary form (/-/) is also available.

Navigating Datasets

The sftp lcd command can be used to navigate around the z/OS dataset space. Using the dataset prefix // or /-/, the dataset space can be entered. Once there, traversal up and down various dataset levels can be performed similarly to hierarchical file systems.

Partitioned datasets are treated as directories as well. Once a PDS is made the current working directory, its members can be listed and retrieved like normal files.

Just as listing the entire catalog from the root is not allowed, it is not possible to make the catalog root the current working directory. As such, the command lcd // will fail.

Example: Navigating the dataset space

cozsftp> lcd //user     1
cozsftp> lpwd     2
Local working directory: //USER
cozsftp> lcd coz.testjcl     3
cozsftp> lpwd
Local working directory: //USER.COZ.TESTJCL
cozsftp> lcd ..     4
cozsftp> lpwd
Local working directory: //USER.COZ
1

Using the dataset prefix //, the high level qualifier user is specified. For lcd commands, the dataset name is case insensitive.

2

The lpwd command will list the current working dataset level. Note that the name is properly displayed in uppercase

3

Multiple levels can be traversed at a time. Instead of using the normal separator (.), a slash can be used: lcd coz/testjcl.

4

The lcd .. command will move up a level, as expected.

Transferring Datasets

The get and put commands are used to transfer datasets and PDS members.

Any options previously set via the lzopts are in effect for any given transfer.

Example: Get a file to a text sequential dataset

$ cozsftp user@linux.com     1
Connecting to linux.com...
user@linux.com's password:
cozsftp> lzopts mode=text     2
 mode=text
cozsftp> lzopts    
clientcp=IBM-1047    loglevel=I           mode=text            
servercp=ISO8859-1
cozsftp> get /tmp/GPGDSN //USER.GPGDSN     3
Fetching /tmp/GPGDSN to //USER.GPGDSN
ZosDataset[I]: Opening dataset USER.GPGDSN for write with options: new catalog
/tmp/GPGDSN                              100% 1215     1.2KB/s   00:00    
ZosDataset[I]: Closing dataset //USER.GPGDSN - 1215 bytes received, 15 records written
ZosSmf119Record[I]: SMF Type119 recording not enabled; SMF recording disabled
1

This example shows the full connection process, using keyboard-interactive password authentication to a remote linux system.

2

The default transfer mode of binary is overridden and set to text.

3

The get command uses the dataset path prefix // (or, optionally /-/) to specify that a dataset is to be written. At the default log level of I (INFO), information is emitted about the transfer process. Note also that in this case, SMF recording is disabled because the FTP SMF records (type 119) are not currently configured for recording.

Example: Get a text file to a PDS member

cozsftp> lzopts     1
clientcp=IBM-1047     loglevel=I     mode=text     
servercp=ISO8859-1
cozsftp> lcd //user.coz.testjcl 
cozsftp> lpwd
Local working directory: //USER.COZ.TESTJCL
cozsftp> get /tmp/GPGDSN     2
Fetching /tmp/GPGDSN to //USER.COZ.TESTJCL/GPGDSN
ZosDataset[I]: Opening dataset USER.COZ.TESTJCL(GPGDSN) for write with options: old
/tmp/GPGDSN                                             100% 1215     1.2KB/s   00:00    
ZosDataset[I]: Closing dataset //USER.COZ.TESTJCL(GPGDSN) - 1215 bytes received, 15 records written
1

If this transfer is performed after the prior example, the transfer mode will still be text. Using the lzopts command quickly confirms the active options.

2

The get command uses the dataset path prefix // and pds member name in parentheses to identify the member to create.

Example: Get multiple files using a wild-card pattern to a GDG

In release 2.4.0, support was added to allow multiple files to be downloaded to new generations of a GDG.

cozsftp> lzopts gdgnt     1
gdgnt
cozsftp> get /tmp/*.data //USER.COZ.GDG(+1)     2
Fetching /tmp/file1.data to //USER.COZ.GDG(+1)
ZosDataset[I]: Opening dataset //USER.COZ.GDG(+1) for write with options: new catalog
/tmp/file1.data                                             100% 523     20.2KB/s   00:01    
ZosDataset[I]: Closing dataset //USER.COZ.GDG.G0001V00 - 523 bytes received, 10 records written 3
Fetching /tmp/test2.data to //USER.COZ.GDG(+1)
ZosDataset[I]: Opening dataset //USER.COZ.GDG(+1) for write with options: new catalog
/tmp/test2.data                                             100% 886     18.5KB/s   00:01    
ZosDataset[I]: Closing dataset //USER.COZ.GDG.G0002V00 - 886 bytes received, 12 records written 
1

Wild-card downloading of remote files to new GDG generations is only supported if the gdgnt option is enabled. Sites should consider adding this option to the default section of their /etc/ssh/cozsftp_server_config file.

2

This get command uses a wild-card (*) pattern to select any file in the /tmp directory that ends in ".data". Each file will be downloaded to a new generation of the target GDG: USER.COZ.GDG.

3

Each file that matches the pattern is transferred separately. The generation name that was used is printed when the data set is closed.

Example: Put PDS members

cozsftp> lpwd
Local working directory: //USER.COZ.TESTJCL
cozsftp> put ONETEST /tmp/ONETEST     1
Uploading ////USER.COZ.TESTJCL(ONETEST) to /tmp/ONETEST
ZosDataset[I]: Opening dataset USER.COZ.TESTJCL(ONETEST) for read with options: shr

ZosDataset[I]: Closing dataset //USER.COZ.TESTJCL(ONETEST) - 38 records read, 3078 bytes sent

cozsftp> put //USER.coz.testjcl(*)     2
Uploading //USER.COZ.TESTJCL(@@README) to /tmp/@@README
ZosDataset[I]: Opening dataset USER.COZ.TESTJCL(@@README) for read with options: shr

ZosDataset[I]: Closing dataset //USER.COZ.TESTJCL(@@README) - 34 records read, 2754 bytes sent
Uploading //USER.COZ.TESTJCL(ALLOCDS) to /tmp/ALLOCDS
ZosDataset[I]: Opening dataset USER.COZ.TESTJCL(ALLOCDS) for read with options: shr

ZosDataset[I]: Closing dataset //USER.COZ.TESTJCL(ALLOCDS) - 6 records read, 486 bytes sent
Uploading //USER.COZ.TESTJCL(CHKENVD) to /tmp/CHKENVD
ZosDataset[I]: Opening dataset USER.COZ.TESTJCL(CHKENVD) for read with options: shr

ZosDataset[I]: Closing dataset //USER.COZ.TESTJCL(CHKENVD) - 1 records read, 81 bytes sent
Uploading //USER.COZ.TESTJCL(CHKPOST) to /tmp/CHKPOST
ZosDataset[I]: Opening dataset USER.COZ.TESTJCL(CHKPOST) for read with options: shr

ZosDataset[I]: Closing dataset //USER.COZ.TESTJCL(CHKPOST) - 6 records read, 486 bytes sent
Uploading //USER.COZ.TESTJCL(CHKPRE) to /tmp/CHKPRE
ZosDataset[I]: Opening dataset USER.COZ.TESTJCL(CHKPRE) for read with options: shr

ZosDataset[I]: Closing dataset //USER.COZ.TESTJCL(CHKPRE) - 72 records read, 5832 bytes sent
Uploading //USER.COZ.TESTJCL(COZCFGO) to /tmp/COZCFGO
ZosDataset[I]: Opening dataset USER.COZ.TESTJCL(COZCFGO) for read with options: shr

ZosDataset[I]: Closing dataset //USER.COZ.TESTJCL(COZCFGO) - 1 records read, 81 bytes sent
Uploading //USER.COZ.TESTJCL(GPGDSN) to /tmp/GPGDSN
ZosDataset[I]: Opening dataset USER.COZ.TESTJCL(GPGDSN) for read with options: shr

ZosDataset[I]: Closing dataset //USER.COZ.TESTJCL(GPGDSN) - 15 records read, 1215 bytes sent
Uploading //USER.COZ.TESTJCL(ONETEST) to /tmp/ONETEST
ZosDataset[I]: Opening dataset USER.COZ.TESTJCL(ONETEST) for read with options: shr

ZosDataset[I]: Closing dataset //USER.COZ.TESTJCL(ONETEST) - 38 records read, 3078 bytes sent
Uploading //USER.COZ.TESTJCL(TESTPROC) to /tmp/TESTPROC
ZosDataset[I]: Opening dataset USER.COZ.TESTJCL(TESTPROC) for read with options: shr

ZosDataset[I]: Closing dataset //USER.COZ.TESTJCL(TESTPROC) - 111 records read, 8991 bytes sent
Uploading //USER.COZ.TESTJCL(USERTEST) to /tmp/USERTEST
ZosDataset[I]: Opening dataset USER.COZ.TESTJCL(USERTEST) for read with options: shr

ZosDataset[I]: Closing dataset //USER.COZ.TESTJCL(USERTEST) - 187 records read, 15147 bytes sent
1

In this case, the current local directory is the PDS. This put command will transfer a specific member from a fully qualified dataset. Alternatively, the command: put //USER.COZ.TESTJCL(ONETEST) /target

could be used without regard to the current local directory.
2

When the put command is used on a PDS with "*" specified as the member, all of the members are uploaded. Note that the ability to specify a mask, like (AB*) is not currently supported.

Example: Put all generations of a GDG

In release 2.4.0, support was added to allow all generations of a GDG to be uploaded in one put commmand.

cozsftp> ls -alf //coz.test.gdg
Volume   Referred  Ext  Tracks    Used Recfm Lrecl BlkSz Dsorg  Dsname
                                                          GDG   COZ.TEST.GDG
VPWRKA  2013/06/04   1       1       1  U        0  6144  PS    COZ.TEST.GDG.G0003V00
VPWRKA  2013/06/04   1       1       1  U        0  6144  PS    COZ.TEST.GDG.G0004V00
VPWRKC  2013/06/04   1       1       1  U        0  6144  PS    COZ.TEST.GDG.G0005V00
VPWRKB  2013/06/04   1       1       1  U        0  6144  PS    COZ.TEST.GDG.G0006V00

cozsftp> put //coz.test.gdg(*) /tmp  1
Uploading //COZ.TEST.GDG.G0003V00 to /tmp/G0003V00
ZosDataset[I]: Opening dataset COZ.TEST.GDG.G0003V00 for read with options: shr
//COZ.TEST.GDG.G0003V00                                     20%   10KB  10.0KB/s   00:03 ETA
ZosDataset[I]: Closing dataset //COZ.TEST.GDG.G0003V00 - 2 records read, 10248 bytes sent

Uploading //COZ.TEST.GDG.G0004V00 to /tmp/G0004V00
ZosDataset[I]: Opening dataset COZ.TEST.GDG.G0004V00 for read with options: shr
//COZ.TEST.GDG.G0004V00                                     20%   10KB  10.0KB/s   00:03 ETA
ZosDataset[I]: Closing dataset //COZ.TEST.GDG.G0004V00 - 2 records read, 10248 bytes sent

Uploading //COZ.TEST.GDG.G0005V00 to /tmp/G0005V00
ZosDataset[I]: Opening dataset COZ.TEST.GDG.G0005V00 for read with options: shr
//COZ.TEST.GDG.G0005V00                                      0%    5     0.0KB/s 2:43:49 ETA
ZosDataset[I]: Closing dataset //COZ.TEST.GDG.G0005V00 - 1 records read, 5 bytes sent

Uploading //COZ.TEST.GDG.G0006V00 to /tmp/G0006V00
ZosDataset[I]: Opening dataset COZ.TEST.GDG.G0006V00 for read with options: shr
//COZ.TEST.GDG.G0006V00                                      0%    5     0.0KB/s 2:43:49 ETA
ZosDataset[I]: Closing dataset //COZ.TEST.GDG.G0006V00 - 1 records read, 5 bytes sent
1

In this example, all generations of a GDG are uploaded to the /tmp directory on the target system. As with all wild-card put commands, if the target directory is not specified it defaults to the current remote working directory.

Listing datasets and PDS directories

MVS datasets can be listed using the sftp lls command. Partitioned datasets are treated as directories with their members as entries.

When listing z/OS datasets locally with the lls command, catalog search filter keys are in effect for any wildcard requests. The catalog search wildcards *, **, and % used in the examples below are described in the IBM manual DFSMS: Managing Catalogs - SC26-7409. Note that this is different behavior from sftp clients that connect to the Co:Z sftp-server and list datasets with the ls. In that case, regular file globbing rules are in effect.

Example: Listing datasets

cozsftp> lcd //USER
cozsftp> lls -al     1
Volume  Referred  Ext  Tracks    Used Recfm Lrecl BlkSz Dsorg  Dsname
WORK84 2008/09/05   1       1       1  FB      80 27920  PS    USER.AFILE.TXT                              
WORK81 2008/09/08   1      30       ?  U        0  6144  PO-E  USER.COZ.LOADLIB                            
WORK81 2008/09/11   1      15       4  FB      80 27920  PO    USER.COZ.SAMPJCL                            
WORK84 2008/09/11   1       1       1  U        0  6144  PS    USER.COZ.TEST.SEQ                           
WORK81 2008/09/09   1      15       3  FB      80 27920  PO    USER.COZ.TESTJCL

cozsftp> lls -al //user.coz.t*     2
Volume  Referred  Ext  Tracks    Used Recfm Lrecl BlkSz Dsorg  Dsname
WORK81 2008/10/20   1      15       4  FB      80 27920  PO    USER.COZ.TESTJCL 

cozsftp> lls -al //user.c*.**     3
Volume  Referred  Ext  Tracks    Used Recfm Lrecl BlkSz Dsorg  Dsname
WORK81 2008/10/20   1      30       ?  U        0  6144  PO-E  USER.COZ.LOADLIB                            
WORK81 2008/10/20   1      15       4  FB      80 27920  PO    USER.COZ.SAMPJCL                            
WORK84 2008/09/25   1       1       1  U        0  6144  PS    USER.COZ.TEST.SEQ                           
WORK81 2008/10/20   1      15       4  FB      80 27920  PO    USER.COZ.TESTJCL
1

The long form of the list command ls -al will list detailed information from the catalog about each dataset.

2

Using the catalog search filter key syntax, a single asterisk can be used to as a wildcard for a single dataset level.

3

Using the catalog search filter key syntax, a double asterisk can be used to perform a deep listing. In this example, the single and double asterisk syntax is combined to list all of the datasets beginning with the prefix USER.C.

Example: Listing a PDS directory

...
cozsftp> lcd //user.coz.sampjcl     1
cozsftp> lls -al     2
Name           Size  Created         Changed          ID
@@README 
BPXBATCH         13 2008/04/04 2008/04/04 17:18:09  USER   
BPXBATSL         16 2008/04/03 2008/04/03 10:36:52  USER   
COZCFGD          65 2008/03/27 2008/05/12 14:28:54  USER   
COZPROC          30 2008/03/27 2008/03/27 11:54:48  USER   
DTLSPAWN         40 2008/05/05 2008/05/05 09:31:08  USER   
GPGDSN           15 2008/05/05 2008/05/05 10:40:05  USER   
GREPDSN  
GREPSED          12 2008/05/05 2008/05/05 09:30:51  USER   
OFFLDSMF 
RUNCOZ           20 2008/03/27 2008/05/12 14:08:02  USER   
RUNCOZ2          15 2008/05/05 2008/05/05 10:02:51  USER   
RUNCOZ3           8 2008/05/05 2008/05/06 08:50:37  USER   
RUNSPAWN         54 2008/05/12 2008/05/12 14:25:37  USER   
RUNSPWN2         20 2008/05/12 2008/05/12 13:19:05  USER   
TDIRK            18 2008/04/03 2008/04/03 10:19:20  USER   
WGET2DSN
1

The lcd command is used to make a PDS the current working local "directory".

2

The lls -al command (long list form) is used to display the members of the PDS, including available statistics.

4.6 Working with POSIX files

This section describes how to use the enhanced client with POSIX files (HFS, zFS) on z/OS. Standard sftp implementations (including z/OS OpenSSH) support only binary mode file transfers. The Co:Z implementation provides binary transfer mode by default, but also supports text mode transfers. Text mode transfers are controlled via the following options:

  • mode: when set to text causes file transfers to be text based.

  • clientcp and servercp: When text mode is active, these settings determine the codepage translation that will take place. The default client code page is ISO8859-1. The default server code page is the current z/OS locale.

  • linerule: When text mode is active, this setting determines how line separators are converted between the client and server.

Transferring Files

The get and put commands are used to transfer POSIX files (either on HFS or zFS filesystems).

The options (listed above) that have been previously set via the lzopts are in effect for any given transfer. All other options (used for dataset support) are ignored for POSIX file transfers.

Example: Get a text POSIX file

$ cozsftp user@linux.com     1
Connecting to linux.com...
user@linux.com's password: *****
cozsftp> lzopts mode=text,servercp=UTF-8     2
 mode=text         servercp=UTF-8  
cozsftp> lzopts     3
clientcp=IBM-1047    loglevel=I           mode=text            
servercp=UTF-8
cozsftp> pwd
Remote working directory: /tmp
cozsftp> get msgs.txt     4
Fetching /tmp/msgs.txt to msgs.txt
/tmp/msgs.txt                                        100%   19KB  19.0KB/s   00:00    
ZosPosixFile[I]: Closing file msgs.txt - 19488 bytes received, 19488 bytes written5    
cozsftp>
1

This example shows the full connection process, using keyboard-interactive password authentication to a remote linux system.

2

The default transfer mode of binary is overridden and set to text. Additionally, the server (linux) code page is explicitly set to UTF-8.

3

Displays the active options. Note that the client code page, if not explicitly set, defaults to the current z/OS locale.

4

The get command requests the transfer of the POSIX file using the options in effect.

5

Upon completion, an informational message is written that describes the number of bytes received from the server and the number of bytes written to the local file. These counts are commonly the same, but changes in line separators and codepages can result in different counts.

Example: Put a text POSIX file

cozsftp> put sftp-server.log /tmp     1
Uploading sftp-server.log to /tmp/sftp-server.log
sftp-server.log                                  100%  127     0.1KB/s   00:00    
ZosPosixFile[I]: Closing file sftp-server.log - 127 bytes read, 127 bytes sent
ZosSmf119Record[I]: SMF Type119 recording not enabled; SMF recording disabled
1

The client text file sftp-server.log is put to the remote directory /tmp. The active file transfer options are used.

4.7 Using the Co:Z SFTP client in batch

The cozsftp client command can be conveniently used in a z/OS batch job without user interaction. The COZBATCH batch utility, also installed as part of the Co:Z toolkit, makes it easy to run cozsftp (or other Unix shell scripts) directly as z/OS batch jobs.

The authentication with the remote system must be set up so as not to require any user interaction. There are three ways to do this with OpenSSH:

  • Use the SSH_ASKPASS environment variable to point to a program that will read a password.

  • Use an OpenSSH public/private keypair.

  • Use a RACF Digital Certificate.

For details on these three authentication options, see Appendix F, Client Authentication Mechanisms. Note that instructions in this appendix must be followed in order to run the examples described below.

Notes for running batch mode SFTP

When sftp is run in batch mode, it is important to know that sftp will abort if any of the supplied commands fail (i.e. complete with a non-zero return code). This behavior is different from an interactive sftp session, where a failed command will report an error, but the session will continue. In cases where a failed command is expected or acceptable (e.g. rm old_file, where old_file may not exist) it is useful to direct batch mode sftp to continue processing. To do this, prefix the command with a dash (-):

-rm old_file

Sample SFTPPROC and batch scripts

A sample SFTPPROC and batch scripts (installed in <COZ_HOME>/samples/sftp_batch) are distributed with the Co:Z toolkit to simplify maintenance and support of batch jobs using the Co:Z SFTP client. Using these samples achieves the following:

  • COZBATCH customized for running CO:Z SFTP

  • installation default options separated from individual JCL members

  • standards defined for a set of variables controlling connection, authentication, options and filenames

  • unix shell script logic separated into separate reusable script files

The reusable script files that are distributed with the Co:Z toolkit are the following:

  • sftp_get.sh - Get a file from a remote system to a local (z/OS) file using the cozsftp command. This script connects a cozsftp client to a remote system running sshd and issues a get command to move a file from the remote system to a local file on z/OS. cozsftp transfer options (lzopts) can be specified to customize the transfer.

  • sftp_put.sh - Put a local (z/OS) file to a remote system using the cozsftp command. This script connects a cozsftp client to a remote system running sshd and issues a put command to move a local file to the remote system. cozsftp transfer options (lzopts) can be specified to customize the transfer.

  • sftp_connect.sh - Connect to a remote system using the Co:Z toolkit cozsftp command. This script connects a cozsftp client to a remote system running sshd and prepares it to accept batch commands.

  • sftp_cat.sh - Get multiple files from a remote system and concatenate them to a local (z/OS) file using the cozsftp command. This script connects a cozsftp client to a remote system running sshd and issues an ls command to get a list of files to get. Each of these files is then retrieved and written to the specified local file on z/OS. cozsftp transfer options (lzopts) can be specified to customize the transfer.

The standard set of variables to control connection, authentication, options and filenames are defined in the table below. Variables used by all scripts are required for sftp_connect.sh. sftp_cat.sh, sftp_get.sh, and sftp_put.sh invoke sftp_connect.sh to establish a connection with the remote host. Some variables are used only for specific scripts as noted in the Script column.

Table 4.1. Script Variables

VariableScriptRequiredDescription
userallrequiredSet to the remote userid
hostallrequiredSet to the remote host
portalloptionalSet to the sshd port on the remote host. Port 22 is used by default
pwdsnalloptionalSet to a fully qualified dataset name (or fully qualified dataset member) containing the user's remote system password. If so, SSH_ASKPASS authentication will be used.

Note: If neither pwdsn or cert is set, z/OS OpenSSH defaults will be used for public/private key authentication.

certalloptionalSet to the name of a SAF digital certificate using one of the following formats:

RING_NAME (no whitespace allowed; the default label will be used)

RING_NAME:LABEL_NAME (no whitespace allowed)

"RING_NAME LABEL_NAME" (whitespace between ring and label)

If the first or second format is used, the connection will be authenticated using the Co:Z Toolkit saf-ssh-agent, which is the recommended approach as hardware private keys are supported. If the third format is used, the z/OS OpenSSH identityKeyRingLabel option will be used for authentication.

Note: If neither pwdsn or cert is set, z/OS OpenSSH defaults will be used for public/private key authentication.

sftp_optsalloptionalSet to any desired SFTP options, including any ssh specific options (designated via the -o switch). Set using contatenation unless the prior variable setting is being overwritten.
cozbin_diralloptionalSet to the absolute path of the installed Co:Z Toolkit "bin" directory. If not set, this directory must be present in the current PATH environment variable.
script_diralloptionalSet to the absolute path of the directory containing the sftp_batch scripts (including this file). All scripts are assumed to be in this directory. If not set, this directory must be present in the current PATH environment variable.
lfilesftp_get.sh and sftp_put.shrequiredSet to the local file to be created or transferred.
rfilesftp_get.sh and sftp_put.shrequiredSet to the remote file to get or put.
lzoptssftp_get.sh, sftp_put.sh, and sftp_cat.shoptionalSet to cozsftp transfer options required for the transfer (e.g. mode=text,replace=no)
ldsnsftp_cat.shrequiredSet to the local dataset or DD to be written
rpatsftp_cat.shrequiredSet to the remote file pattern to get

Using the sample SFTPPROC and scripts, a batch job that gets a file from a remote host and stores it in a data set can be as simple as the following:

//PROCLIB JCLLIB ORDER='COZUSER.COZ.SAMPJCL'
//* 
//*********************************************************************
//* Use the sftp_get.sh script to retrieve a remote file to a local
//* dataset. 
//*********************************************************************
//SFTPGET EXEC PROC=SFTPPROC 
//SFTPIN DD * 
cert="MY-RING:RSA-CERT" 
user=myuser
host=myhost 
lzopts="mode=text" 
lfile=//DD:MYDD 
rfile=/etc/profile 
                                                            
. $script_dir/sftp_get.sh 

//MYDD  DD DSN=COZUSER.SFTPGET.DATA,DISP=(MOD,KEEP), 
//        DCB=(LRECL=80,RECFM=FB),SPACE=(CYL,(3,1)) 
//* 

The sections below describe the SFTPPROC, installation default options, and a few more examples.

PROC for executing the Co:Z SFTP client (cozsftp) in batch

The SFTPPROC sample JCL distributed with the Co:Z toolkit can be used as a tailorable model for customizing COZBATCH for using CO:Z SFTP.

//*********************************************************************
//* 
//* PROC for executing the Co:Z SFTP client (cozsftp) in batch 
//* 
//* Tailor the proc for your installation: 
//* 1.) Tailor LIBRARY with the PDSE that contains the 
//*     COZ load module. 
//* 2.) Tailor SFTPIND= to point to SAMPJCL member that contains
//*     site specific shell variable settings for running the
//*     Co:Z SFTP batch scripts
//* 3.) Review the Co:Z SFTP batch scripts (located in
//*     $COZHOME/samples/sftp_batch) for additional shell variables
//*     to set for individual jobs to get, put, connect, etc...
//*********************************************************************
//EXSFTP    PROC ARGS=,                      < [-L<log_opt>] 1
//   LIBRARY='COZUSER.COZ.LOADLIB',          < STEPLIB FOR COZBATCH
//   SFTPIND='COZUSER.COZ.SAMPJCL(SFTPIND)', < Installation defaults 2
//   REGSIZE='64M',                          < Execution region size
//   LEPARM=''
//RUNSFTP  EXEC PGM=COZBATCH,REGION=&REGSIZE, 3
//   PARM='&LEPARM/&ARGS'
//STEPLIB  DD DSN=&LIBRARY,DISP=SHR
//STDIN    DD DSN=&SFTPIND,DISP=SHR	4	
//         DD DDNAME=SFTPIN
//SFTPIN   DD DUMMY                          < Customized stdin to SFTP 5
//*
// PEND
  
1

COZBATCH logging may be added to ARGS for problem diagnosis.

2

Defines the member that contains the installation CO:Z SFTP defaults. These defaults can be overriden in individual jobs as necessary.

3

Defines the program to execute as COZBATCH, a utility similar to IBM's BPXBATCH. COZBATCH runs a Unix login shell in the original address space.

4

Ensures that the site specific installation defaults are included first in STDIN, before any job specific commands.

5

Defines a name for STDIN allowing jobs using this proc to include commands in STDIN.

Co:Z SFTP Batch Script Settings

The SFTPIND sample JCL member distributed with the Co:Z toolkit can be used as a tailorable model for CO:Z SFTP installation defaults.

###############################################################################
# Co:Z SFTP Batch Script Settings
# The shell variables below can be set to site specific values, but may
# be overridden in individual jobs.
###############################################################################

#
# CONFIGURATION VARIABLES:
#
# cozbin_dir - May be set to the absolute path of the installed Co:Z Toolkit
#              "bin" directory.  If not set, this directory must be present in
#              the current user's PATH environment variable.
# script_dir - May be set to the absolute path of the directory containing the
#              sftp_batch scripts (including this file).  All scripts are 
#              assumed to be in this directory.  If not set, this directory must
#              be present in the current user's PATH environment variable. 
#
cozbin_dir="/usr/local/coz/bin"   1
script_dir="/usr/local/coz/samples/sftp_batch" 2

#
# SFTP OPTIONS VARIABLE:
#
# sftp_opts  - May be set to any site specific SFTP options, including any ssh
#              options (designated via the -o switch).
#
sftp_opts=""
sftp_opts="$sftp_opts -oConnectTimeout=60" 
sftp_opts="$sftp_opts -oServerAliveInterval=60" 3
#
# Set the following option to "no" if you would like to 
# automatically accept host keys for new servers. 
sftp_opts="$sftp_opts -oStrictHostKeyChecking=yes"
  
1

Defines a variable for the location of the cozsftp executable. This variable is used by the sftp batch scripts to execute the cozsftp command.

2

Defines a variable for the location of the sample or customized version of the sftp batch scripts. This variable is used in all jobs executing the sftp batch scripts.

3

Sets global installation options as necessary. Note that the sftp_opts variable is appended as each option is added. Jobs using SFTPPROC can reset or append to these options using this variable.

Logging in batch

Co:Z SFTP Client logging is described earlier in this chapter (Section 4.2, “Co:Z SFTP client logging”). This section describes how to set these options in batch.

OpenSSH SFTP logging is enabled by passing the -v option on the sftp command. This can be done in batch by appending the desired -v option to the sftp_opts variable in the STDIN section of the job:

//SFTPIN DD *
sftp_opts="$sftp_opts -vvv"

Co:Z SFTP client logging is enabled by exporting the COZ_LOG variable in the STDIN section of the job.

//SFTPIN DD * 
export COZ_LOG=T 

COZBATCH logging is enabled by adding a parameter to the EXEC statement for the SFTPRPOC. The following example uses the -LD command switch to set the default logging level to "Debug" for COZBATCH. The t option is also used to prefix all messages with a timestamp.

//SFTPGET EXEC PROC=SFTPPROC,
        ARGS='-LD,t'
//SFTPIN DD *

Finally, it is often very helpful to see details about the Unix System Service shell script processing of the input to COZBATCH. This can be controlled by explicitly requesting a login shell (/bin/sh -L) along with the trace option (-x). The following example sets both the COZBATCH logging level to "Debug" and these shell options:

//SFTPGET EXEC PROC=SFTPPROC,
        ARGS='-LD,t /bin/sh -L -x'
//SFTPIN DD *

Batch job containing examples of running cozsftp in batch

The SFTPSAMP sample JCL distributed with the Co:Z toolkit can be used as a tailorable model for writing batch jobs using CO:Z SFTP.

//SFTPSAMP   JOB (),'DOVETAIL',MSGCLASS=H,NOTIFY=&SYSUID 
//PROCLIB JCLLIB ORDER='COZUSER.COZ.SAMPJCL'
//* 
//*********************************************************************
//* 
//* Batch job containing examples of running cozsftp in batch
//* 
//* Tailor the proc and job for your installation: 
//* 1.) Modify the Job card per your installation's requirements 
//* 2.) Modify the PROCLIB card to point to this PDS, or wherever 
//*     the SFTPPROC procedure has been installed. 
//* 
//*********************************************************************
//* 
//*********************************************************************
//* Use the sftp_get.sh script to retrieve a remote file to a local
//* dataset.  This example uses a user ssh key stored in a SAF
//* digital certificate
//*********************************************************************
//SFTPGET EXEC PROC=SFTPPROC 1
//SFTPIN DD * 2  
cert="MY-RING:RSA-CERT" 3 
user=myuser
host=myhost 
lzopts="mode=text" 
lfile=//DD:MYDD 
rfile=/etc/profile 
                                                            
. $script_dir/sftp_get.sh 4 

//MYDD  DD DSN=COZUSER.SFTPGET.DATA,DISP=(MOD,KEEP), 
//        DCB=(LRECL=80,RECFM=FB),SPACE=(CYL,(3,1)) 
//* 
//*********************************************************************
//* Use the sftp_put.sh script to send a local file to a remote
//* file.  This example uses a password (via the SSH_ASKPASS protocol)
//* to connect to the remote system
//*********************************************************************
//SFTPPUT EXEC PROC=SFTPPROC 
//SFTPIN DD * 
pwdsn="COZUSER.COZ.SAMPJCL(PW)" 
user=myuser
host=myhost 
lzopts="mode=text" 
lfile=/etc/profile 
rfile=/home/myuser/zprofile.txt 

# Don't try to use our public key even if we have a default one
# This would not normally be a required setting
sftp_opts="$sftp_opts -oPubkeyAuthentication=no"
                                                            
. $script_dir/sftp_put.sh

//* 
//*********************************************************************
//* Use the sftp_cat.sh script to retrieve multiple files from a remote
//* system and concatenate them to a local dataset.  This example uses
//* z/OS OpenSSH defaults for public/private key authentication
//* (because neither the "cert" or "pwdsn" variables are defined)
//*********************************************************************
//SFTPCAT EXEC PROC=SFTPPROC 
//SFTPIN DD * 
user=myuser
host=myhost 
lzopts="mode=text" 
ldsn="//DD:MYDD" 
rpat=/home/myuser/doc/*.txt 
                                                          
. $script_dir/sftp_cat.sh 

/* 
//MYDD  DD DSN=COZUSER.SFTPCAT.DATA,DISP=(MOD,KEEP), 
//        DCB=(LRECL=80,RECFM=FB),SPACE=(CYL,(3,1)) 
//* 
//*********************************************************************
//* Use the sftp_connect.sh script to connect to a remote system
//* and send customized sftp commands.
//*********************************************************************
//SFTPCONN EXEC PROC=SFTPPROC 
//SFTPIN DD * 
cert="MY-RING:RSA-CERT" 
user=myuser 
host=myhost 
                                                          
. $script_dir/sftp_connect.sh << EOB 5
ls -al
EOB

// 
  
  
1

Each step in this sample job uses the SFTPPROC.

2

Each step uses SFTPIN defined by the SFTPPROC.

3

Sets all variables required for sftp_get.sh. Note that some required variables are in the installation global defaults. Global variables may be optionally overriden here before calling the shell script.

4

Executes sftp_get.sh to get a file from the remote system saving it in the data set defined by //MYDD.

5

The symbols '<< EOB' followed by an ending 'EOB' define a Here document which, in this example, is an inline string containing a SFTP command.

Wild-card downloading using a DD

In release 2.4.0, support was added to allow multiple files to be downloaded using a wild-card pattern (*) to a single DD if the DD was allocated with DISP=MOD.

//SFTPGET EXEC PROC=SFTPPROC
//SFTPIN DD *
user=myuser
host=myhost
pwdsn="COZUSER.COZ.SAMPJCL(PW)"
lzopts="mode=text"
lfile=//DD:MYDD       1
rfile=/somedir/*.trn  2
. $script_dir/sftp_get.sh  3

//MYDD DD DSN=COZUSER.SFTP.MULTIGET.DATA,DISP=(MOD,CATLG,DELETE), 1
//        DCB=(LRECL=2052,RECFM=FB),SPACE=(CYL,(3,1))
1

The lfile variable references a DD in the job step that is allocated with DISP=MOD. This allows multiple files to be downloaded, in succession, to the end of the same target dataset. Each matching file will be downloaded separately, in alphabetical order.

2

The get command uses a wild-card (*) pattern to select any file in the /somedir directory that ends in ".trn".

3

The underlying get subcommand generated by the sftp_get.sh script will be:
get /somedir/*.trn //DD:MYDD

Copyright© 2009-2017 Dovetailed Technologies, LLC. All rights reserved.
Co:Z® is a registered trademark of Dovetailed Technologies, LLC.