A large commit.

[pdp8.git] / sw / kermit / k12 / k12mit.not

From: Charles Lasner <lasner@watsun.cc.columbia.edu>
To: PDP-8 Lovers Everywhere <pdp8-lovers@ai.mit.edu>
Subject: Announcement of additional KERMIT-12 utilities.

    While no changes have been made to the body of KERMIT-12 itself,
several things have been changed/added.

    At the request of the KERMIT distribution service (KERMSRV)
certain files have been slightly modified so they are acceptable to
that bitnet, etc. facility.  (Seems to be a problem with LRECL>80.)
All files are now 80 or less.  Except for the .DOC file, all it took
was a little "cosmetic surgery" on a few lines.  FTP'd copies are
mostly unaffected.  Most of the problems have to do with
interpretation of the inter-page FF character being treated as the
first character of the "record" in this non-stream-oriented system.

    At this time there is no actual doc file, as the file K12MIT.DOC
is merely a truncation of the listing of K12MIT.PAL as passed through
PAL8 and CREF.  Anyone with a system big enough to support a 200K+
long source file can create this file themselves.  In addition, due
to certain quirks within PAL8 and CREF "beating" against unix line
conventions, the file K12MIT.DOC at watsun.cc.columbia.edu was
slightly different from the precise output of the assembly process,
but again, only a cosmetic change.

    Since this file greatly exceeded the KERMSRV restriction, it has
been withdrawn in favor of the source fragment equivalent to it taken
directly from K12MIT.PAL.  This source fragment is short enough that
even an RX01-based OS/8 system can create the listing file from it
thus recreating the original K12MIT.DOC locally.  All this will
disappear in the future when a "proper" doc file appears.  In the
meantime, K12MIT.DOC in whatever form it is available contains
hardware hints and kinks, assembly options, and other info useful to
users and anyone interested in the "innards" of the program, as well
as an edit history of how K12MIT got to be where it is now starting
from its "grandfather" K08MIT.  It ends at the first line of the code
in K12MIT.PAL, but includes all of the special purpose definitions
particular to the various devices supported, such as DECmate I,
DECmate II, etc.  Any changes to customize KERMIT-12 are still
accomplished using the separate patch file K12PCH.PAL which is
unchanged.

    New files cover two areas: 1) direct loading without KERMIT-12,
and 2) .BOO format support.

1)  Many users have the hardware for running KERMIT-12, but don't
already have it or another suitable program to acquire it yet, a real
"catch-22" situation.  Towards that end, a set of utilities has been
provided to directly load KERMIT-12 without already having it.

    Most PDP-8 sites do have access to some other machine.
Hopefully, the serial connection to be used is fairly "clean" and
error-free, or at least some of the time.  These programs depend on
this fact.  This could either be a connection to a remote multi-user
system or something like a null-modem connection to a nearby IBM-PC.
The programs assume only a few things:

    a)  The connection is error free.

    b)  The other end doesn't absolutely require anything be sent to
it to send data to the PDP-8 end.  (The -8 end will not send ^S/^Q or
anything like that because this is unnecessary; all data goes only
into PDP-8 memory directly.)

    c)  The other end will send the data at a time controlled from
its end, or after at most one character sent from the PDP-8 end of
the link.

    The first situation is illustrated by the example of a PC
connected to the -8.  The -8 program is started, and it waits
indefinitely after the -8 user presses any one key.  (The
corresponding character is sent to the PC where it is ignored.)  The
PC end is initiated with a command such as COPY K12FL0.IPL AUX: and
the data goes to the -8.

    The second situation is illustrated by a remote system where a
command such as TYPE K12FL0.IPL is available.  The delimiting CR is
not typed at this time, and will be finished later by the loading
program.  The initial connection up until the TYPE command is not
covered by the loading program itself, so the user must supply a
basic comm program, which is possible to accomplish in about 10 words
or less if the rates are "favorable", or worst-case, a terminal can
be used and the line switched over to the -8 at the appropriate time.
In any case, CR or other appropriate character is hit on the -8 and
the loading program echoes it down the line (and on the console) to
initiate the data down-load.

    d)  The other end is assumed to send the file verbatim without
insertion of <del> characters (octal 177) and upper-case/lower-case
is preserved.

    If all of these assumptions are met, then the down-load
accomplishs a partial acquisition of K12MIT.SV, the primary binary
file of KERMIT-12.  The process must be repeated several times to
acquire all portions.  If a local compare utility is available that
can compare absolute binary files, perhaps the process can be totally
repeated to assure reliable results by comparing runs.

    The method used is borrowed from the field-service use of a
medium-speed serial port reader on the -8 for diagnostic read-in.
This reader is *almost* compatible with the device 01 reader such as
the PC8E.  The difference is that the *real* PC8E is fully
asynchronous, whereas the portable reader just spews out the
characters without any protocol.  The PC8E can't drop any characters
in theory, although there are reports of misadjusted readers that
drop characters at certain crucial data rates.  (The PC8E runs at
full speed if possible, and failing this falls back to a much slower
speed.  All operations depend on the use of the hardware handshakes
of the IOTs etc., so nothing should be lost but throughput.
Misadjusted readers may drop characters when switching over to the
slower mode.)

    The reason the field reader is acceptable is that it is used only
to load diagnostics directly into memory using the RIM and BIN
loaders.  These minimal applications can't possibly fall behind the
reader running at full speed.  This is the same principle used here
to down-load KERMIT-12.

    The loading program is a 46 word long program suitable to be
toggled into ODT and saved as a small core-image program.  The user
starts the program and then (at the appropriate time) presses one key
(usually CR if it matters) and the loader waits for remote input.  As
the other end sends the data, it is directly loaded into memory.
There is a leader/trailer convention, just like paper-tape binary, so
at end-of-load the program exits to OS/8 at 07600.  At this time the
user issues a SAVE command.  This completes the down-load of a single
field of K12MIT.SV.

    At the current time, there are actually two fields of K12MIT.SV,
namely 00000-07577 and 10000-17577, and there are two such loaders.
There is no check for proper field, so the proper loader must be used
with the proper data, else the fields will get cross-loaded and will
certainly fail.

    Once the two fields are obtained as separate .SV files (named
FIELD0.SV and FIELD1.SV) they can be combined using ABSLDR.SV with
the /I switch (image mode) set.  The resultant can be saved as
K12MIT.SV.  This, if all went well, is identical in every way to the
distributed K12MIT.SV (which is only distributed in encoded form; see
below).  Actual file differences will only exist in the extraneous
portions of the file representing the header block past all useful
information and the artifacts of loading which represent 07600-07777
and 17600-17777 which are not used.  This is the normal case for any
OS/8 system when any file is saved.  Merely saving an image twice
will cause this to happen.  At this point, K12MIT.SV can be used as
intended, namely to acquire, via KERMIT protocol, the entire release.
It is recommended that the provisional copy of K12MIT.SV be abandoned
as soon as the encoded copy is decoded since the encoding process
provides some assurances of valid data (using checksumming, etc.).

    This process can be accomplished on any KL-style -8 interface
including PT08, etc., or on the printer port of VT-78 and all
DECmates.  When used on the DECmates, there may be some minor
problems associated with the down-load which may have to be done as
the first use of the printer port after power-on, or some other
restriction.  The loader includes a suggested instruction for DECmate
use if problematic (and raises the program length to 47 words).
Also, due to observed bugs in the operating system (OS/278 only),
there are restrictions on the use of ABSLDR.SV that cause certain
command forms to fail while other seemingly equivalent forms succeed!
This is documented in the latest K12MIT.BWR file in the distribution.
The command form stated in the K12IPL.PAL file is the only known form
that works correctly on these flawed systems.

    The format for down-load files is known as .IPL or Initial
Program Load format.  It consists of a leader containing only
lower-case letters (code 141-177 only) followed by "printable" data
in the range 041 (!) through 140 (`).  Each of the characters
represents six bits of data, to be read left to right as pairs, which
load into PDP-8 12-bit memory.  The implied loading address is
always to start at 0000 of the implied field.  The leader comment
contains documentation of which field of data from K12MIT.SV it is.
The trailer consists of one lower-case character followed by anything
at all.  This is why it is crucial that DEL (177) not appear anywhere
in the body of the file.

    Throughout the file, all codes 040 or less are ignored.  This
allows for spaces in the lower-case leader for better readability,
and for CR/LF throughout the entire file.  CR/LF is added every 32
words (64 characters) to satisfy cetain other systems' requirements.
The trailer contains documentation on a suggested SAVE command for
the particular data just obtained.

2)  PDP-8 ENCODE format is the format of choice to obtain binary OS/8
image files because of the validation techniques employed, etc.  This
is the standard method of distributing K12MIT.SV as well as other
"critical" files such as TECO macros and other image files.  In the
MS-DOS world there exists another very popular format known as .BOO
encoding.  It would be useful to support this format on the PDP-8 as
well.

    .BOO format files are smaller because they use six-bit encoding
instead of five-bit encoding, or at least in theory.  Both ENCODE and
.BOO use repeat compression techniques, but ENCODE can compress
12-bit words of any value, while .BOO only compresses zeroes and that
itself is based on a byte-order view of the data.  PDP-8 programs
often include large regions of non-zero words such as 7402 (HLT)
which would not compress when looked at as bytes.  Such files would
show compression rations quite different from the norm.

    In any case, .BOO format is useful on the PDP-8 because it allows
inter-change with .BOO files created on other systems, such as PCs.
This allows the exchange of unusually formatted files, such as TECO
macros between PDP-8s and PCs.  (Both systems support a viable
version of TECO.)

    The new KERMIT-12 utilities include a .BOO encoder and .BOO
decoder, known as K12ENB.PAL (or ENBOO.PAL) and K12DEB.PAL (or
DEBOO.PAL) respectively.  They use .BOO encoded files unpacked in the
standard OS/8 "3 for 2" order to preserve the original byte contents
when the files originate from other systems.  (Technically, .BOO
format doesn't require this, but the obvious advantages dictate it.
Anything encoded into .BOO format must merely have a 24-bit data
structure encoded into four six-bit characters, so in theory any
encoding of two adjacent PDP-8 12-bit words would be acceptable.  By
additionally supplying the bits in OS/8 pack/unpack order guarantees
the inter-system compatibility as well.)

    There is an inherent weakness in the original .BOO format which
must be addressed.  .BOO format files always end on one of two data
fields: either a repeat-zero compression field, or on a 24-bit field
expressed as four characters.  Should the data in a 24-bit field
consist of only two or even one bytes, there are one or two
extraneous null bytes encoded into the field to complete it.

    Presumably the need to add the extra bytes is to allow validation
of the format.  In any case, only the encoder knows just how many (0,
1, 2) bytes are extraneous.  We can presume that if the last byte is
non-zero, it is significant.  If the last two are both zero, then the
last or possibly both are extraneous with no way to tell.

    On PC systems, the general trend is to ignore these one or two
extra bytes because so far there haven't been any complaints of
failure.  I have personally discovered that a widely used PC .BOO
encoding program (written in C) erroneously adds two null bytes as
a short compression field beyond the data!  This is not a .BOO format
issue, but rather a genuine program bug.  Apparently few PC users are
concerned that encoding their files prevents transparent delivery to
the other end.

    In the OS/8 world, the situation is quite different.  Each OS/8
record is 256 words or 384 bytes.  If even a single byte is added,
this creates an additional all-zeroes record.  Besides wasting space,
it is conceivable that such a file could be dangerous to use under
OS/8 depending on content.  (Certain files, such as .HN files are
partially identified by their length.  File damage, such as
lengthening a file from two to three records will confuse the SET
utility, etc.)  Many files cannot be identified as having been
artifically lengthened (and may be hard to shorten!), so this must be
avoided.

    I have invented a fix for the problem: repeat compression fields
are expressed as ~ followed by a count.  2 means two null bytes and
is thus the smallest "useful" field to be found.  (It takes two
characters to express what would take 2-2/3 characters in encoded
format.  One null would only take 1-1/3 characters, not two, so this
case is vestigial, but must be supported for the benefit of
brain-dead encoders.) The value of 0 means a count of literally zero,
thus ~0 is a "NOP" to a decoder.  I have successfully tested MS-DOS
programs written in BASIC and C that decode .BOO files successfully
even if ~0 is appended to the end with no ill effects.  (They
correctly ignored the appended fields.)

    In my encoding scheme, ~0 at the end of a data field containing
trailing zeroes means to "take back" a null byte. ~0~0 means to take
back two null bytes.  Thus files encoded with ENBOO.PAL either end in
a repeat-compression field as before, or in a data encoding field
possibly followed by ~0 or ~0~0 if necessary.  The corresponding
DEBOO.PAL correctly decodes such files perfectly.

    Should files encoded with ENBOO reach "foreign" systems, they
will do what they always do, i.e., make files one or two bytes too
long occasionally, with no other ill effects.  Files originating from
such systems will certainly be lacking any trailing correction fields
and will cause DEBOO to perform as foolishly as MSBPCT.  Extraneous
null bytes will appear at the end of the file in OS/8 just as in
MS-DOS in this case.  (Note that if the file length is not a multiple
of 384 bytes, additional bytes are added by DEBOO as well, but this
is not a design weakness of .BOO format.  It is caused by the clash
of fixed record size and a variable size format.)

    Hopefully, files originating on OS/8 will be decoded on OS/8 as
well, thus preserving file lengths.  Most "foreign" files will
probably be ASCII, so the ^Z convention will allow removal of
trailing null bytes at either end.  It is hoped that MS-DOS and other
systems "upgrade" their .BOO format files to be compatible with the
PDP-8 version.

    All KERMIT-12 files are available via the normal distribution
"paths" of anonymous FTP and/or KERMSRV.  The user is directed to the
file /ftp/pub/kermit/d/k12mit.dsk as a "roadmap" to the entire
distribution.  Each .PAL file includes assembly instructions.  Most
use non-default option switches and non-default loading and saving
instructions, so each must be carefully read.  The development
support files (TECO macro, .IPL generator, recent copies of PAL8,
CREF, etc.) are included in the total collection.  Development is not
possible on RX01 systems due to inadequate disk space, but RX02's are
barely adequate with a lot of disk exchanges.  (Future versions may
require larger disks for development.)

Charles Lasner (lasner@watsun.cc.columbia.edu)