A large commit.

[pdp8.git] / sw / kermit / k12 / k12mit.bwr

Date: Fri, 1 May 1992 17:00:00 EDT
From: Charles Lasner <lasner@watsun.cc.columbia.edu>
Subject: DECmate I problems and more patching problems

DECmate I problems.

    Attempts to use the distributed Kermit-12 Version 10g on a
DECmate (I) system will certainly fail.  The coding specific to the
DECmate I was never tested until recently.  Two key routines wait for
status flags that never raise because the affected registers do not
generate flag changes/interrupts.  This is unrelated to general
serial data handling which works as originally coded.

    There is a simple patch to the program to alleviate the problem:

.LOAD SYS:KERMIT.SV/I$*$        [load the file in image mode and then
                                 ask for more input.  The $ which is
                                 printed signifies the use of <ESC>
                                 as the command terminator.  The * is
                                 printed by the system command
                                 decoder requesting further input.
                                 The second $ signifies the use of
                                 <ESC> to end input to the command
                                 decoder.  The loader program
                                 terminates and control returns to
                                 the keyboard monitor.]

.ODT                            [call in ODT to patch the program]

7/ 0007 0012                    [change default baud rate to 2400;
                                 this is optional]

353/ 5352 7000                  [make a JMP .-1 into a NOP]
10302/ 5301 7000                [make a JMP .-1 into a NOP]
12243/ 3607 3610                [bump the version number]

^C                              [^C to exit to monitor]

.SAVE SYS KERMIT                [save patched file]

This also updates the release revision from 10g to 10h.  Future
versions will eliminate the overhead of the now defunct routines.

    The only DECmate versions remaining to be tested are:

    DECmate I with DP278B (the system used for testing has DP278A)
    DECmate III without internal modem
    DECmate III with internal modem
    DECmate III+

Patching problems.

    The restrictions placed on patching apparently stem from a bug
going back at least as far as OS/8 V3D (likely further).  Apparently,
when a JSW value of 1 is used (as Kermit-12 does), the GET command
doesn't work.  Apparently, the system confuses the need to save the
contents of 10000-11777 with the need to load it in the first place.
Kermit-12 operates by first placing once-only code in the affected
area, then discarding it in favor of a locked-in copy of the USR
routine.  To avoid overhead, the JSW value of 0001 is set to indicate
there is no need to save this dead code when the USR is swapped in
over it.  Apparently, the GET command sets the =1 too early in the
load process, so the code that uses the USR to carry out the actions
of the GET operation doesn't properly load the Kermit-12 code.

Consequently, the warnings documented in previous chapters of the
.BWR file (below) apply in all cases.  The interaction with CCL sited
below may not apply in all versions, but the GET command problem is
apparently universal.

cjl

------------------------------

Date: Mon, 28 October 1991 20:00:00 EST
From: Charles Lasner <lasner@watsun.cc.columbia.edu>
Subject: Kermit-12 patching restrictions revisited yet again

    Even more operating system ills (will it ever end?):

    Still further investigation into operating system bugs in OS/278
V2 on DECmates reveals that the problem in even worse that realized
two weeks ago (see previous .BWR article):

    When a SAVE command is executed from OS/278 involving a loaded
handler, the SAVE operation fails!  The contents of the files will be
corrupted in general and will likely become (at least partially) all
zeroes!  The exact scope of the problem has not been ascertained, but
certain loading tests reveal that the command fails even when
accessing additional memory beyond field zero and one.

    All operations to SYS: or any device co-resident with SYS: (or
when DSK:=SYS: which is typically the case in many systems but not a
rule) are unaffected beyond the restrictions reported previously.

    Until recently, SAVE commands were of little interest to the
casual user of OS/278, since program execution and ordinary file
creation are unaffected.  Since there are now several programs to be
loaded and saved by users, the problem is more significent.  Users of
the direct loading method of acquiring  KERMIT-12 are also in the
affected category.

    Clearly all developers and anyone assembling any part of the
KERMIT-12 package should be aware of this problem.  As a precaution,
all persons using the SAVE command for any reason are advised to use
the form involving SYS: only to avoid this problem.  (Advanced users
can determine which handlers are possibly co-resident and are thus
acceptable as well.) The resultant file can always be copied to any
device as required after the fact.

cjl

------------------------------

Date: Thu, 10 October 1991 05:00:00 EDT
From: Charles Lasner <lasner@watsun.cc.columbia.edu>
Subject: Kermit-12 patching restrictions revisited and .BOO problems

    More operating system ills regarding file loading:

    Further investigation into operating system bugs in OS/278 V2 on
DECmates reveals that the problem is worse than first realized:

    When using GET or LOAD (ABSLDR) commands, especially when loading
image files such as FIELD0.SV, FIELD1.SV (the partial load files from
direct memory loading of K12MIT), or K12MIT.SV with /I, the JSW and
starting field/address can become "mangled" into unusable values.
One particular case achieved the impossible value of 6303 for a
starting field change instruction (legal values are 6203 through 6273
by 10s).

    Consequently, the general recommendation for SAVE commands as
used in various utilities throughout KERMIT-12 configuration etc., is
to use explicit starting address and loading locations and JSW
values.  In short, always give a complete description of the SAVE
operation under OS/278.  For example, when direct-loading K12MIT
through the printer port into the DECmate, the following commands
should be used:

}LOAD FIELD0.SV/I,FIELD1.SV/I$*$

}SAVE SYS K12MIT.SV 00000-07577,10000-17577;00200=0001

    As discussed earlier, the CCL form of the ABSLDR LOAD command
works even though other seemingly equivalent forms don't.  The
complete SAVE command forces all parameters to be taken explicitly
from the command; no reliance on system "assumptions" or loading
artifacts.  Always use the complete values for loading taken from the
relevant program documentation.

    Most users of KERMIT-12 are running OS/8 V3D, etc., where this
sort of system bug isn't seen.  In the future, all KERMIT-12
documentation will give the "verbose" form of the command to contain
this OS/278 V2-specific problem.

Regarding .BOO format encoding:

    The newest release of KERMIT-12 includes .BOO format encoding of
all binary files and TECO macros as an alternative to ENCODE format.
ENCODE format is still the preferred method of distribution, but .BOO
format allows for use with other systems, such as MS-DOS.  For
example, TECO macros used with OS/8 TECO can be interchanged in .BOO
format with similar files used with MS-DOS TECO.  Intermediary sites,
such as unix systems will not destroy the "delicate" nature of such
files, etc.

    The KERMIT-12 .BOO utilities are NOT totally compatible with
existing .BOO utilities on other systems! Just like OS/8 ENCODE and
DECODE, ENBOO and DEBOO do a perfect encoding/decoding of OS/8 files
into their original form.  When used with "foreign" .BOO decoders,
some unpredictable things might occur.

    Certain other .BOO encoders are known to throw in extraneous null
bytes at the end of the file.  Further, there is a design weakness in
the original .BOO format that causes more null bytes to possibly
appear.  The KERMIT-12 programs utilize a superset of the original
format to ensure correct encoding/decoding.  When passing these files
which now contain "correction bytes" to older decoders, the files are
decoded with inflated lengths because the older decoders don't
recognize the length correction.  When passing files created by older
encoders to the PDP-8, the resultant decoded files will also have
inflated lengths because the older encoders failed to place
correction bytes into the file.

    The general rule for dealing with .BOO files originating from
other systems is that they may have incorrect lengths.  The resultant
files may be (falsely) padded out with extraneous null bytes.  In any
case, since the files generally have no blocking structure, the files
will be padded by OS/8 up to the nearest whole record or multiple of
384 bytes anyway.  Unless the file is ASCII and has a ^Z at the end,
there is no way to determine the original intended file length.
Files may  be padded by null bytes introduced by other systems' bugs,
the inherent weakness of the original .BOO format, or ultimately by
OS/8 padding requirements.

    ASCII files from other systems may be adjusted by using an editor
such as TECO which stops at the ^Z.  A second generation of the
transferred file may be somewhat shorter when processed this way.

    Should a file originating in OS/8 be intended for OS/8 use only
(such as an encoding of a .SV file), it should not be decoded on an
intermediate system, because a re-encoded version may differ from the
encoded original because of ignored correction bytes, bugs, or the
inability to insert correction bytes.  Violating any of these rules
could lead to OS/8 files corrupted into being too long.  It is
conceivable that these altered files are even dangerous to use under
OS/8 because of their inflated lengths. (Certain files are validated
by their restricted size, such as .HN files which must be exactly two
or three blocks long depending on whether they are for one or two
page handlers.  If a one-page handler became three pages in file
length, it could conceivably be confused with a two-page handler,
etc.)

cjl

------------------------------

Date: Sun, 7 October 1990 12:00:00 EDT
From: Charles Lasner <lasner@watsun.cc.columbia.edu>
Subject: Kermit-12 patching restrictions

    All Kermit-12 configuration done according to the documentation
works "as advertised." Users are tempted to patch the distributed
image file K12MIT.SV as a "quick and dirty" method to make small
modifications such as changing the default baud rate, etc.  There is
"conventional wisdom" that this can be accomplished using GET, SAVE
commands to allow the use of ODT; this method is ordinarily used with
other OS/8 family programs.  It has been reported that this does NOT
work on OS/278, the usual operating system for the DECmates.  The
following method should be avoided (a work-around is offered later):

.GET SYS KERMIT                 [setup current image for patching]

.ODT                            [call in ODT to patch the program]

7/ 0007 0012                    [change default baud rate to 2400]

^C                              [^C to exit to monitor]

.SAVE SYS KERMIT                [save patched file]

This method follows the exact procedure described in virtually every
OS/8 document regarding patching of image files.  The cited example
changes the default baud rate from 1200 Baud to 2400 Baud by
replacing the value chosen from the DEC standard table for 1200 Baud
with the applicable value for 2400 Baud.  This value is stored within
Kermit-12 as the corresponding twelve-bit word with all high-order
bits zeroed.  (The location used is 000007; this is valid for Version
10g, but could change in later versions.)

    This attempt to make changes the "conventional" way produces a
corrupted image file of K12MIT.SV (renamed to KERMIT.SV in the above
example) when using OS/278 Version 2, the usual operating system on
the DECmate II, etc.  This method probably works in earlier (OS/8
V3D, etc.) systems, however no attempt has been made to trace this
bug in prior systems.  A "fool-proof" method is required that works
in spite of bugs in the operating system.

    A work-around was attempted using OS/278 V2 on a DECmate II hard
disk system:

.LOAD SYS:KERMIT.SV/I           [load the file in image mode]

.ODT                            [call in ODT to patch the program]

7/ 0007 0012                    [change default baud rate to 2400]

^C                              [^C to exit to monitor]

.SAVE SYS KERMIT                [save patched file]

This also fails!

    For reasons not understood yet, the following seemingly
equivalent command DOES work:

.LOAD SYS:KERMIT.SV/I$*$        [load the file in image mode and then
                                 ask for more input.  The $ which is
                                 printed signifies the use of <ESC>
                                 as the command terminator.  The * is
                                 printed by the system command
                                 decoder requesting further input.
                                 The second $ signifies the use of
                                 <ESC> to end input to the command
                                 decoder.  The loader program
                                 terminates and control returns to
                                 the keyboard monitor.]

.ODT                            [call in ODT to patch the program]

7/ 0007 0012                    [change default baud rate to 2400]

^C                              [^C to exit to monitor]

.SAVE SYS KERMIT                [save patched file]

This allows ODT commands to patch the file as intended, and also
causes the subsequent SAVE command to work properly.  All OS/8 family
systems support this command (as long as CCL is enabled), so it will
"always" work.

    For those users who run with CCL turned off, the following
sequence will also work:

.R ABSLDR                       [run the loading program directly]
*KERMIT.SV/I                    [load Kermit in image mode]
*$                              [<ESC> is typed to terminate the
                                 loading process.]

.ODT                            [call in ODT to patch the program]

7/ 0007 0012                    [change default baud rate to 2400]

^C                              [^C to exit to monitor]

.SAVE SYS KERMIT                [save patched file]

    The newer OS/8 family systems generally can't turn off the CCL
mechanism.  Since the R and RU commands are typically disabled on
newer releases, only the CCL command work-around applies.  Users
opting to disable CCL are likely running "older" systems, such as
OS/8 V3D on DECtapes.  On these systems, ANY of the above methods
should work, because the problematic bug didn't exist on those
systems.  Had DEC not gone "backwards" we could have avoided this
entire discussion!

    It is assumed the user will make "correct" patches to KERMIT-12;
at least there is a "safe and proper" mechanism available to
accomplish it!

cjl

------------------------------

Date: Thu, 6 September 1990 12:00:00 EDT
From: Charles Lasner <lasner@watsun.cc.columbia.edu>
Subject: Kermit-12 potential problems

    A newly implemented ENCODE/DECODE method should eliminate the
reported problems with regard to passing encoded binary files through
problematic "paths." The method chosen is a variant on the 5-bit
encoding algorithm suggested.  Encoded files now pass right through
all of the WPS-related utilities.  It is necessary to acquire
virtually all files of this re-release of KERMIT-12 since all ENCODed
files are different, as well as the source programs for the
ENCODing/DECODing utilities themselves.  Due to the file being
"bare", the TECO macro K12GLB.TEC is possibly defective when it
arrives at a user site; it will now be ENCODed as K12GLB.ENC to avoid
this problem.

    The KERMIT-12 source files are different due to maintenance work,
requiring the user to obtain the re-released files.  The sources now
include a file to "pre-clear" memory.  This aids in reducing the size
of the ENCODed binary file K12MIT.ENC since undefined areas are no
longer "relics" of random values, rather they are all set to 0000
octal.  The long strings of identical words will be eliminated since
the new encoding format does repeat compression.

    KERMIT-12 has still not been tested on any DECmates other than
the DECmate II, as no volunteers have come forward with the proper
hardware:

    DECmate I with DP278A
    DECmate I with DP278B
    DECmate III without internal modem
    DECmate III with internal modem
    DECmate III+

    A tentative volunteer for the DECmate I with DP278A configuration
has been contacted, but testing has not yet started.

cjl

------------------------------

26-Jul-90  1:15:43-GMT,15259;000000000001
Return-Path: <lasner@cunixf.cc.columbia.edu>
Received: from cunixf.cc.columbia.edu by watsun.cc.columbia.edu (5.59/FCB)
	id AA26223; Wed, 25 Jul 90 21:15:41 EDT
Received: by cunixf.cc.columbia.edu (5.59/FCB)
	id AA11871; Wed, 25 Jul 90 21:16:19 EDT
Date: Wed, 25 Jul 90 21:16:18 EDT
From: Charles Lasner <lasner@cunixf.cc.columbia.edu>
To: fdc@cunixf.cc.columbia.edu
Subject: This was sent out to PDP8-LOVERS
Message-Id: <CMM.0.88.648954978.lasner@cunixf.cc.columbia.edu>

    I thought you might want to see this; it refers to the encoding
problem I reported for a user with the problem (he has no net
capability) in the programs using that encoding scheme we
discussed...

From:	Charles Lasner (cjl)
To:	PDP8-LOVERS
Subj:	Feedback on encoding issues regarding archived files.

    I have written a pair of OS/8 programs to ENCODE and DECODE
binary files into an "ASCII-fied printable" format.  Those of you
familiar with either uuencode/uudecode or .BOO format will understand
my intentions.  They were originally written for the purpose of
distribution of binary (.SV) files of KERMIT-12 by Columbia
University in NY as part of the standard KERMIT collection (K12*.*).
Columbia imposes a restriction on all files: they must be distributed
in ASCII only.  This is to ensure proper distribution regardless of
the "path" taken between Columbia and the end user.  Be advised that
various problematic E-mailers, ASCII-EBCDIC EBCDIC-ASCII
translations, filters for reserved codes, known problematic character
substitutions, etc. are lurking out there! Consider yourself lucky if
you get your sender's copy intact without some form of "cosmetic"
reformatting.  By encoding the binary files into an appropriate
subset of ASCII, these problems hopefully are avoided.  While we
can't prevent ALL problems, we can usually tackle the most likely
ones.

    My original design was based on a discussion I had with Frank da
Cruz of Columbia University (of KERMIT fame) regarding what to
restrict ourselves to in a robust format.  He was "unhappy" with some
of the vulnerabilities of the uuencode and .BOO formats, which while
popular, are not impervious to some "real" problems that have come
up.  We essentially designed an archiving format that was PDP-8
oriented, but not limited to -8s only.  Some of the highlights of the
format are:

a)  File format restricted to "printable" six-bit subset of ASCII
only.  All else ignored; this was to minimize the "garble" factor,
yet maintain a fairly high rate of efficiency: two ASCII characters
equal one PDP-8 12-bit word. (This has proved to be problematic and
is why we are here!)

b)  The archive file contains imbedded commands, not implied ones.
By validating the commands, you can "trust" the contents.  Commands
are available for whatever purpose arises.  Already implemented are
commands to start ("(FILE filename.ext)") and end ("(END
filename.ext)") the imbedded file, and an official comment command
("(REMARK anything)") to help document the contents of the rest of
the file.  This is of course expandable.  My OS/8 programs create all
three types of commands.  The start and end commands also
theoretically allow multiple files in an archive, but I ignore the
end command in the decoder and only allow one file per archive.  I do
support the start command completely, which includes a suggested name
for the file.  This name can be used at the user's option, or can be
locally overridden.  The encoding program inserts the original file
name in this field, as this is of course the most likely name for the
file at the other end.

c)  The archive contains a checksum for its contents to ensure the
validity of the file.

d)  All "white space" character considerations are ignored; imbedded
extraneous space characters, formfeeds, extra CR/LF, etc. are
harmless.  The CR/LF must be present at appropriate intervals, but
this is only a problem with files passed through unix systems that
delete the CR.  Since OS/8 requires the CR and LF to be considered
"printable", this is not a problem;  the use of programs such as
c-KERMIT will insert the CR if configured properly (SET FILE TYPE
TEXT).  Programs such as Rahul Dhesi's FLIP program are available to
correct the problem easily if necessary: FLIP -m *.* or equivalent
will remedy this.

e)  There is an internal record length of 64 characters with framing
characters, to ensure the validity of each record.  This prevents the
file from getting "out of sync" with its original.  This causes each
line to be 68 characters including CR and LF, which is usually
reasonable to most systems.

    Unfortunately, this scheme has proved to be flawed in an
important way that "matters."  This format must deliver files to
OS/278 systems by the prevailing paths of existent systems connected
to DECmates containing only the normally present DEC release
software.  This could include sending the files via DEC-DX through
WPS8, or acquiring the files on either DECmate CP/M-80 or DECmate
MS-DOS, possibly using KERMIT-80, or KERMIT-MS as appropriate.  If a
file is received in the CP/M-80 environment, it can be converted to
WPS8 format using a DEC-supplied program called WPSCONV.  If a file
is received in the MS-DOS environment, it can be converted to WPS8
format using a DEC-supplied program called CONVERT.  Incidentally,
CONVERT can also convert CP/M-80 files as well, using MS-DOS format
as an intermediary;  WPSCONV is known to have bugs, which were
corrected in CONVERT (which requires the MS-DOS hardware, not just
the CP/M-80 hardware).  These CP/M-80 and  MS-DOS files can also come
to the DECmate directly from a Rainbow as well, since the
corresponding Rainbow systems are format compatible with the DECmate.
DECmate MS-DOS additionally supports IBM-PC diskettes (160K or 180K
single-sided only and read-only) as yet another source.  Thus there
are many paths to WPS8 versions of our files.

    The problem with these methods is that apparently there is a bug
in OS/278 WPFLOP, the only distributed conversion program a user
would already have on his OS/278 system. (We haven't actually
isolated the problem to WPFLOP, as the complaining user was taking
the files from MS-DOS via CONVERT then to OS/278 via WPFLOP;
conceivably the problem is in CONVERT, but in any case the problem
exists somewhere in this supported path.)

    The internal encoding used is to break the 12-bit word into two
six-bit halves.  Each half is in the range 00-77 octal.  Adding 041
to the value yields characters in the range of ! through ` or 041
through 140 octal.  The codes for 101-132 are A through Z and can be
replaced by 141-172 for a through z if desired.  This prevents
case-sensitivity which is another possible network anomaly.  We
identified the DECmate problem as an anomaly regarding @ and `.  The
character codes for 100 and 140 are not treated uniquely, so the
resulting OS/278 file is an inaccurate representation of the file.
The decoding program correctly failed the conversion on a checksum
error, so at least the user was aware of the problem!

    As the PDP8-LOVERS, we will hopefully acquire an archive site for
our files, so all of these considerations will apply.  We need a file
format that is "bullet-proof" to avoid problems like this one.  I am
soliciting suggestions for improvements on this encoding scheme (and
any other overall file format suggestions as well) to provide an
effective solution.  The resultant programs will be added to the
KERMIT-12 collection freely distributed by Columbia as well as other
sources (DECUS, etc.).

    Some suggestions have already been made:

1.  Just "quick-fix" the problem by providing an alternate character
to the ` to make it non-anomalous with @.  The available choices are
{ | } and ~ only.  The DEL character (octal 177) is unsuitable for
other reasons; all other characters are either already used, or
unprintable, or lower-case.  This has the advantage of being most
compatible with the existing programs, since the original character
code can be supported as well; the "preferred" character would be
generated by all future versions of the ENCODE program, and existing
files could be trivially edited for compatibility as needed.  This
would have to be tested  -- it is possible that the bug would
persist.  The choice is further narrowed to { and | only, since 175
and 176 are sometimes treated as alternates to ESC.  It is likely
that systems which "mangle" the case of a character which is
alphabetic could also do the same to { | } and ~ making them [ \ ]
and _ respectively.  This makes the entire suggestion unworkable.

2.  Change the format to "Hexafied-ASCII" where each PDP-8 12-bit
word becomes represented by three characters from a 16-character set
such as 0-9,A-F or A-P.  The alphabetic codes would be immune to case
conversion, and virtually every system supports this subset of ASCII.
Instead of 64 characters on a line representing 32 12-bit words, each
line would be 72 characters on a line representing 24 12-bit words
(not counting framing characters and CR/LF).  This also allows for
many additional codes if needed.  This scheme has the drawback of
making the encoded file more inefficient, as the file will generally
be 50% longer than those created by the original six-bit scheme.
This robust scheme is workable.

3.  Modify 2. to include some form of compression.  The easiest is to
incorporate repeat compression.  One simple scheme is to use an
indicator character (R was suggested) as a prefix for an encoded
count.  It could be followed by three characters encoding the value
of the 12-bit word and two characters encoding the value of the
repeat count.  Since this occupies six characters, as does two
adjacent 12-bit encoded words, this scheme saves space when used for
repeat compression lengths greater than two.  The compressed field is
the same length as two compressed "triplets", so overall file
validation techniques wouldn't require special-case checks, as long
as trailing "fill" characters were allowed for the last record before
the short checksum record (which is signalled by its length).  (T was
suggested for this trailer character to be used to pad the last line
with 0-69 characters.) This allows for compressing 3-258 repeated
12-bit words into six characters.  This would benefit files
containing large areas of zeroes or HLT instructions, etc., as this
can be the actual contents of binary files.  If a .BN file created by
PAL8, etc. is loaded and saved, then "junk" areas are created in the
.SV file.  Unfortunately, this is the norm, and the junk increases
the size of the encoded version of the file.  If the .BN file is
loaded AFTER loading an all-zeroes file such as the binary output of:

        *0

        ZBLOCK  7600

        $

or equivalent as necessary (extended memory zeroed if required,
etc.), then the file has all-zeroes gaps in it.  These would repeat
compress out using this scheme.  Incidentally, an additional
advantage of this method is that the resulting "cleaner" core-image
file is slightly easier to disassemble, in case the source is lost.
(Anyone who ever disassembled a .SV file or equivalent understands
what I mean!).  This also makes a binary papertape file (such as a
diagnostic) loaded into a .SV file a little easier to follow when
consulting the write-up, as memory is zeroed in between the locations
referenced in the listing.  The .SV file is smaller when encoded than
the .BN file due to elimination of the paper-tape encoding overhead.
OS/8 files of diagnostics could therefore be more efficiently
archived as .SV files (encoded) than .BN files.

4.  Change to a 5-bit encoding with compression.  This would use 32
codes chosen from A-Z, 0-9 to encode the file five bits at a time per
character.  Five PDP-8 12-bit words would be encoded in 12
characters.  Since PDP-8 binary files are always multiples of 128
12-bit word pages, there would need to be 4.8 "junk words" at the end
of each block to encode the implied length of 130 words/block.  Each
line would be 78 characters (plus framing characters and CR/LF) so
that four lines encodes a PDP-8 page, just as in the original six-bit
scheme (the original scheme used 64 characters per line!).  The last
line of the file would contain 0-77 padding characters as necessary
to maintain the line width as before.  Repeat compression schemes can
be expressed in any way that is a multiple of 12 characters; perhaps
one or two adjacent expressions of repeat compression similar to
above.  Expected efficiency of this scheme is similar to the original
six-bit method, or possibly slightly better; if compression is NEVER
useful, then the file is 1.2 times as large.

    There is an implementation restriction placed on the DECODE
program: it should be relatively short, since it must be distributed
in source form.  It must also be written in a subset of PAL8
compatible with the original PAL8 of the PS/8 days (ugh!) to ensure
viability on any OS/8 family system.  PAL8 Version B0 from OS/278 is
distributed in ENCODed form, so this restriction need not apply to
any other programs such as the ENCODE program or KERMIT-12, etc.  It
has been determined that PAL8 Version B0 and the companion CREF
Version B0 will correctly function on any OS/8 family system on any
PDP-8 member suitably configured to run the operating system the user
already has running. (There is a minor anomaly when using input files
from the TTY: handler; see K12MIT.DOC for a detailed explanation.)
CPU extensions such as BSW and IAC RAL are not present in these
programs, as was the original intention of OS/8 (which eventually was
lost as newer members of DEC's programming staff were ignorant of
this problem!).  It is acceptable to have a "bare-bones" subset of
the DECODE program distributed in "old" PAL8-compatible source form,
along with a "fancier" version written in a more modern PAL8
language, as the binary could then be DECODed with the subset DECODE
program, or the source could be assembled with PAL8 Version B0 to
"bootstrap" the "full" version of the DECODE program as necessary.

    For those of you who can't wait, and want these utilities as they
stand (using the fallible six-bit method), they are available via
anonymous FTP from Columbia University (watsun) as
/w/kermit/d/k12dec.pal and /w/kermit/d/k12enc.pal for the DECODer and
ENCODer respectively.  More information is available in
/w/kermit/d/k12mit.doc or /w/kermit/d/k12mit.pal regarding use of
PAL8 Version B0, other assemblers (such as PAL10 or P?S PAL) or other
KERMIT-12 issues, etc.

Charles Lasner (lasner@cunixf)
cjl

------------------------------

Date: Fri, 4 May 90 13:55:02 EDT
From: Charles Lasner <lasner@watsun.cc.columbia.edu>
Subject: Kermit-12 problems

    If the release files of KERMIT-12 are brought to DECmate MS-DOS
via any of the various paths that can be used (such as from a Rainbow
in either CP/M RX50 or MS-DOS RX50 format, etc.; in this particular
case the reporting user obtained them using IBM-PC SSDD 180k 5-1/4"
PC-DOS format.) then the files are available as DECmate II MS-DOS or
CP/M-80 files on one of its standard devices (a:,b:,c:,d: floppies or
e:,f:,g:,h: hard disk volumes).

    The ultimate goal is to get these files (un-scathed!) to DECmate
II OS/278 for KERMIT-12 installation.  The standard DEC CONVERT
program alledgedly can convert any combination of MS-DOS or CP/M-80
or WPS/8 from/to each other.  By converting the files to WPS/8
documents, the files can be translated to OS/278 later (using the
OS/278 WPFLOP utility).

    There is a problem with DEC's CONVERT.EXE: it only CORRECTLY
supports Rainbow/DECmate RX50 MS-DOS and CP/M diskettes, so the other
formats (8" CP/M-80 diskettes and one-sided PC diskettes) have to be
pre-converted with the appropriate copy commands to a supported
diskette or hard disk volume first before using CONVERT.  This is not
a big problem, as we are merely using standard procedures, but the
point is that much of this is undocumented or obscure.  (I had to
help the reporting user to copy his files to a "friendlier" device
for CONVERT's benefit which only delayed our discovery of the REAL
problem!)

    The CONVERT program alledgedly supports ASCII/WPS format
conversion from/to any of MS-DOS, CP/M-80, or WPS/8 (but only on
a:,b:,c:,d:,e:,f:,g:,h: logical drives, not on the other hardware or
media possibly hooked up to the DECmate!).  Our purpose is to move
the K12MIT files to WPS/8 format.  This can be attempted with
standard commands of CONVERT, but there apparently is a bug:

    When you boot to OS/278 and retrieve the WPS/8 documents (via
WPFLOP) which are the ENCODed files of KERMIT-12 as OS/278 files,
there is a character anomaly between two encoding characters
(specifically @ and `) that destroys the integrity of the affected
file.  This is possibly due to a bug in OS/278 WPFLOP, but more
likely is a problem with MS-DOS CONVERT.  Regardless of the
perpetrator, this path is not viable to obtain the ENCODed files of
the KERMIT-12 release.

    Fortunately, the source files are not affected, as the anomalous
characters are not part of the PDP-8 assembly language, and only
comments could be affected. (As far as I can tell, there aren't any
affected characters even in the comments!) It is therefore necessary
to assemble KERMIT-12 directly from the sources when installing it on
the DECmate II if obtaining it via any path which includes
CONVERT/WPFLOP.  The other ENCODed files are for PAL8 Version B0 and
CREF Version B0 which are already present on the DECmate II as part
of the standard release of OS/278 for the DECmate II and are thus
superfluous.  All ENCODed files can be recreated from OS/278 itself
using the sources, etc., so the intended release files can be
recreated for distribution to other OS/278 sites (bypassing the
CONVERT/WPFLOP path).  Future versions of the DECODE program will
obviate this problem when an appropriate alternative format is
supported properly which is immune to DEC's glitch.

    A related problem surrounds the GLOBAL TECO macro K12GLB.TEC (aka
GLOBAL.TEC).  Due to the "delicate" nature of TECO macros, they could
get "mangled" by the time they get to a user site.  Future releases
of KERMIT-12 will "protect" the macro by ENCODing it into K12GLB.ENC.
It has also been reported that there are problems running the macro
on certain releases of OS/8 family TECO and on other TECOs for other
machines, and also problems running certain versions of OS/8 TECO on
the DECmates.  The author will investigate this problem eventually,
but the main usage of the macro is for KERMIT-12 source maintainence
on an OS/8 V3D system using the corresponding version of TECO; it is
beyond the scope of KERMIT-12 development to investigate the myriad
releases of TECO and their hardware and operating system
dependencies; perhaps some TECO hackers can assist us!

    An obscure problem indeed!  Users give good feedback...

   Can you suggest a fix for the CONVERT/WPFLOP-induced corruption?
One is to allow the current format as a subset, but use a
substitution character for the garbled character.  Our character set
is the 64 characters from ! through `, so the anomalous occurrences
of @ are problematic.  If we change the preferred character for ` to
a lower-case letter (only octal 141 up is available, so let's assume
the use of a) we avoid the CONVERT/WPFLOP problem.  Newer released
ENCODed files would then be immune to the treachery, but would
require the newer DECODing program (or use TECO to change all
occurrences of a to ` and then use the old DECODE program).

    Should we abandon this inner format altogether?  We could use an
even more robust format like ASCII hex: 0-9 and A-F (allowing a-f as
well!) at the expense of longer files (currently 2 characters=12
bits, but would become 3 characters=12 bits).  This would also hold
up better through EBCDIC network conversion...

cjl