Whole file not being sent.

A discussion of Co:Z sftp, a port of OpenSSH sftp for z/OS
Post Reply
giltjr
Posts: 18
Joined: Wed Jul 07, 2010 9:06 pm

Whole file not being sent.

Post by giltjr » Mon Aug 30, 2021 4:24 pm

We are running a very OLD version of CoZ:SFTP, V1.7.2 (yes I know) but I just installed and tested with V6.2.3 and it shows the same problem.

I can't remember why but we have one customer where we need to use the buffer and number of requests non-standard values: -B 1448 and -R 2. If we use the default for -B and -R, the complete file is sent. But, again, for some reason we have other problems when we use the default with this customer, so we have to use -B 1448 and -R 2.


When we use those options for some files the complete file is not sent if we send it as a "MVS" file (put //DD:DDNAME remotefile). We compress the data with PKZip. If I copy the file to OMVS and sent it, it works.

The file compressed is 39 tracks and the DCB is RECFM=U,BLKSIZE=27990,LRECL=0.

With -vvv option specific here is the last part of it working incorrectly:

%17:00:00.797920% debug3: Sent message SSH2_FXP_WRITE I:1495 O:2156072 S:1448
%17:00:00.797926% debug3: SSH2_FXP_STATUS 0
%17:00:00.797928% debug3: In write loop, ack for 1493 1448 bytes at 2153176
%17:00:00.797933% debug3: SSH2_FXP_STATUS 0
%17:00:00.797936% debug3: In write loop, ack for 1494 1448 bytes at 2154624
%17:00:00.843737% debug3: SSH2_FXP_STATUS 0
%17:00:00.843768% debug3: In write loop, ack for 1495 1448 bytes at 2156072
ZosDataset%I%: Closing dataset //ATP1JSG.TEST.SVPMPMPF.Z2 - 78 records read, 2157520 bytes sent
%17:00:00.845026% debug3: Sent message SSH2_FXP_CLOSE I:6
%17:00:00.899046% debug3: SSH2_FXP_STATUS 0


Here is the last part of it working correctly:

%17:13:30.125695% debug3: Sent message SSH2_FXP_WRITE I:1499 O:2161864 S:607
%17:13:30.125701% debug3: SSH2_FXP_STATUS 0
%17:13:30.125704% debug3: In write loop, ack for 1497 1448 bytes at 2158968
%17:13:30.172299% debug3: SSH2_FXP_STATUS 0
%17:13:30.172335% debug3: In write loop, ack for 1498 1448 bytes at 2160416
%17:13:30.172351% debug3: SSH2_FXP_STATUS 0
%17:13:30.172355% debug3: In write loop, ack for 1499 607 bytes at 2161864
PosixFile%I%: Closing file /SYSTEM/tmp/ATP1JSG.TEST.SVPMPMPF.Z2 - 2162471 bytes read, 2162471 bytes sent
%17:13:30.173272% debug3: Sent message SSH2_FXP_CLOSE I:6
%17:13:30.219514% debug3: SSH2_FXP_STATUS 0

What can we do to figure out why it's not sending the whole file.

giltjr
Posts: 18
Joined: Wed Jul 07, 2010 9:06 pm

Re: Whole file not being sent.

Post by giltjr » Mon Aug 30, 2021 4:42 pm

Quick update. It looks like it may have something to do with the BLKSIZE. If I compress the file and make the block size 2800, less than B*R (1448*2), it sent the complete file.

Seems like maybe if B*R is smaller than the last block of the file, it may drop part of the last block.

dovetail
Site Admin
Posts: 1991
Joined: Thu Jul 29, 2004 12:12 pm

Re: Whole file not being sent.

Post by dovetail » Wed Sep 08, 2021 3:46 pm

-B on the sftp (and cozsftp) commands are used to override the buffer size that is used on FXP_WRITE packets to write the stream of data. The only reason to override this would be because some SFTP server product can't handle the default buffer size (32K). Since the SSH2 RFC spec for filexfer (sftp) says that all should support this, you might be dealing with a non-conforming implementation.

-R has to do with how many writes can be outstanding. Two is a low number and could cause throughput issues if the end-end delay is very long.

It is extremely rare to need to change either one. And this shouldn't have anything to do with how much data is being read from your input data set.

In your second example that worked correctly, you are sending a Unix file, so there are many reasons that the bytes might be different.

In your examples, your are using an obsolete version of Co:Z SFTP, so I can't see the options that are being used and it is impossible to say what is happening. For example, in the first example when you are reading a data set, the options may be set to mode=text so that a line terminator character(s) would be added to each of the 78 records.

I would suggest that you print the data set with IDCAMS and check to see that the number of records match and that the number of records read. Then total up the lengths of the 78 records (from IDCAMS). If you use a current version of Co:Z SFTP and use "mode=binary", then the number of bytes sent will match this exactly.

giltjr
Posts: 18
Joined: Wed Jul 07, 2010 9:06 pm

Re: Whole file not being sent.

Post by giltjr » Tue Sep 21, 2021 1:44 pm

I did check and the number of records read were correct. It appeared that when the last block was read that instead of sending the whole last block it was only sending "B*R" bytes and dropping the rest of the block.


I went back and found some notes. The customer we are/were having the problem with was new and when using the defaults for -B and -R the transfer and the files got larger than 2-3MB the transfer would just hang on our side and eventually timeout with no response from the remote side.

They were not interested and seem to lack the skill of trouble shooting their side. So I assumed we were sending data faster than something could handle it on their side. Just started playing with -B and -R and I got it down to 1448 and 2 before the hangs stopped. That is why they are sooo small.

That being said, I have found out that the remote side has done some software upgrades with both their firewall and sftp server. I have testing with them using the defaults for -R and -B with files sizes of 2MB - 50MB and have had no hangs and the complete file is being sent.

Since we seem to be able to use the defaults for -R and -B we aren't having the problem and so I don't need to pursue this any more.

Thank you for the time and effort you have put in.

Post Reply