| 1 | From: Charles Lasner <lasner@watsun.cc.columbia.edu> |
| 2 | To: PDP-8 Lovers Everywhere <pdp8-lovers@ai.mit.edu> |
| 3 | Subject: Announcement of additional KERMIT-12 utilities. |
| 4 | |
| 5 | While no changes have been made to the body of KERMIT-12 itself, |
| 6 | several things have been changed/added. |
| 7 | |
| 8 | At the request of the KERMIT distribution service (KERMSRV) |
| 9 | certain files have been slightly modified so they are acceptable to |
| 10 | that bitnet, etc. facility. (Seems to be a problem with LRECL>80.) |
| 11 | All files are now 80 or less. Except for the .DOC file, all it took |
| 12 | was a little "cosmetic surgery" on a few lines. FTP'd copies are |
| 13 | mostly unaffected. Most of the problems have to do with |
| 14 | interpretation of the inter-page FF character being treated as the |
| 15 | first character of the "record" in this non-stream-oriented system. |
| 16 | |
| 17 | At this time there is no actual doc file, as the file K12MIT.DOC |
| 18 | is merely a truncation of the listing of K12MIT.PAL as passed through |
| 19 | PAL8 and CREF. Anyone with a system big enough to support a 200K+ |
| 20 | long source file can create this file themselves. In addition, due |
| 21 | to certain quirks within PAL8 and CREF "beating" against unix line |
| 22 | conventions, the file K12MIT.DOC at watsun.cc.columbia.edu was |
| 23 | slightly different from the precise output of the assembly process, |
| 24 | but again, only a cosmetic change. |
| 25 | |
| 26 | Since this file greatly exceeded the KERMSRV restriction, it has |
| 27 | been withdrawn in favor of the source fragment equivalent to it taken |
| 28 | directly from K12MIT.PAL. This source fragment is short enough that |
| 29 | even an RX01-based OS/8 system can create the listing file from it |
| 30 | thus recreating the original K12MIT.DOC locally. All this will |
| 31 | disappear in the future when a "proper" doc file appears. In the |
| 32 | meantime, K12MIT.DOC in whatever form it is available contains |
| 33 | hardware hints and kinks, assembly options, and other info useful to |
| 34 | users and anyone interested in the "innards" of the program, as well |
| 35 | as an edit history of how K12MIT got to be where it is now starting |
| 36 | from its "grandfather" K08MIT. It ends at the first line of the code |
| 37 | in K12MIT.PAL, but includes all of the special purpose definitions |
| 38 | particular to the various devices supported, such as DECmate I, |
| 39 | DECmate II, etc. Any changes to customize KERMIT-12 are still |
| 40 | accomplished using the separate patch file K12PCH.PAL which is |
| 41 | unchanged. |
| 42 | |
| 43 | New files cover two areas: 1) direct loading without KERMIT-12, |
| 44 | and 2) .BOO format support. |
| 45 | |
| 46 | 1) Many users have the hardware for running KERMIT-12, but don't |
| 47 | already have it or another suitable program to acquire it yet, a real |
| 48 | "catch-22" situation. Towards that end, a set of utilities has been |
| 49 | provided to directly load KERMIT-12 without already having it. |
| 50 | |
| 51 | Most PDP-8 sites do have access to some other machine. |
| 52 | Hopefully, the serial connection to be used is fairly "clean" and |
| 53 | error-free, or at least some of the time. These programs depend on |
| 54 | this fact. This could either be a connection to a remote multi-user |
| 55 | system or something like a null-modem connection to a nearby IBM-PC. |
| 56 | The programs assume only a few things: |
| 57 | |
| 58 | a) The connection is error free. |
| 59 | |
| 60 | b) The other end doesn't absolutely require anything be sent to |
| 61 | it to send data to the PDP-8 end. (The -8 end will not send ^S/^Q or |
| 62 | anything like that because this is unnecessary; all data goes only |
| 63 | into PDP-8 memory directly.) |
| 64 | |
| 65 | c) The other end will send the data at a time controlled from |
| 66 | its end, or after at most one character sent from the PDP-8 end of |
| 67 | the link. |
| 68 | |
| 69 | The first situation is illustrated by the example of a PC |
| 70 | connected to the -8. The -8 program is started, and it waits |
| 71 | indefinitely after the -8 user presses any one key. (The |
| 72 | corresponding character is sent to the PC where it is ignored.) The |
| 73 | PC end is initiated with a command such as COPY K12FL0.IPL AUX: and |
| 74 | the data goes to the -8. |
| 75 | |
| 76 | The second situation is illustrated by a remote system where a |
| 77 | command such as TYPE K12FL0.IPL is available. The delimiting CR is |
| 78 | not typed at this time, and will be finished later by the loading |
| 79 | program. The initial connection up until the TYPE command is not |
| 80 | covered by the loading program itself, so the user must supply a |
| 81 | basic comm program, which is possible to accomplish in about 10 words |
| 82 | or less if the rates are "favorable", or worst-case, a terminal can |
| 83 | be used and the line switched over to the -8 at the appropriate time. |
| 84 | In any case, CR or other appropriate character is hit on the -8 and |
| 85 | the loading program echoes it down the line (and on the console) to |
| 86 | initiate the data down-load. |
| 87 | |
| 88 | d) The other end is assumed to send the file verbatim without |
| 89 | insertion of <del> characters (octal 177) and upper-case/lower-case |
| 90 | is preserved. |
| 91 | |
| 92 | If all of these assumptions are met, then the down-load |
| 93 | accomplishs a partial acquisition of K12MIT.SV, the primary binary |
| 94 | file of KERMIT-12. The process must be repeated several times to |
| 95 | acquire all portions. If a local compare utility is available that |
| 96 | can compare absolute binary files, perhaps the process can be totally |
| 97 | repeated to assure reliable results by comparing runs. |
| 98 | |
| 99 | The method used is borrowed from the field-service use of a |
| 100 | medium-speed serial port reader on the -8 for diagnostic read-in. |
| 101 | This reader is *almost* compatible with the device 01 reader such as |
| 102 | the PC8E. The difference is that the *real* PC8E is fully |
| 103 | asynchronous, whereas the portable reader just spews out the |
| 104 | characters without any protocol. The PC8E can't drop any characters |
| 105 | in theory, although there are reports of misadjusted readers that |
| 106 | drop characters at certain crucial data rates. (The PC8E runs at |
| 107 | full speed if possible, and failing this falls back to a much slower |
| 108 | speed. All operations depend on the use of the hardware handshakes |
| 109 | of the IOTs etc., so nothing should be lost but throughput. |
| 110 | Misadjusted readers may drop characters when switching over to the |
| 111 | slower mode.) |
| 112 | |
| 113 | The reason the field reader is acceptable is that it is used only |
| 114 | to load diagnostics directly into memory using the RIM and BIN |
| 115 | loaders. These minimal applications can't possibly fall behind the |
| 116 | reader running at full speed. This is the same principle used here |
| 117 | to down-load KERMIT-12. |
| 118 | |
| 119 | The loading program is a 46 word long program suitable to be |
| 120 | toggled into ODT and saved as a small core-image program. The user |
| 121 | starts the program and then (at the appropriate time) presses one key |
| 122 | (usually CR if it matters) and the loader waits for remote input. As |
| 123 | the other end sends the data, it is directly loaded into memory. |
| 124 | There is a leader/trailer convention, just like paper-tape binary, so |
| 125 | at end-of-load the program exits to OS/8 at 07600. At this time the |
| 126 | user issues a SAVE command. This completes the down-load of a single |
| 127 | field of K12MIT.SV. |
| 128 | |
| 129 | At the current time, there are actually two fields of K12MIT.SV, |
| 130 | namely 00000-07577 and 10000-17577, and there are two such loaders. |
| 131 | There is no check for proper field, so the proper loader must be used |
| 132 | with the proper data, else the fields will get cross-loaded and will |
| 133 | certainly fail. |
| 134 | |
| 135 | Once the two fields are obtained as separate .SV files (named |
| 136 | FIELD0.SV and FIELD1.SV) they can be combined using ABSLDR.SV with |
| 137 | the /I switch (image mode) set. The resultant can be saved as |
| 138 | K12MIT.SV. This, if all went well, is identical in every way to the |
| 139 | distributed K12MIT.SV (which is only distributed in encoded form; see |
| 140 | below). Actual file differences will only exist in the extraneous |
| 141 | portions of the file representing the header block past all useful |
| 142 | information and the artifacts of loading which represent 07600-07777 |
| 143 | and 17600-17777 which are not used. This is the normal case for any |
| 144 | OS/8 system when any file is saved. Merely saving an image twice |
| 145 | will cause this to happen. At this point, K12MIT.SV can be used as |
| 146 | intended, namely to acquire, via KERMIT protocol, the entire release. |
| 147 | It is recommended that the provisional copy of K12MIT.SV be abandoned |
| 148 | as soon as the encoded copy is decoded since the encoding process |
| 149 | provides some assurances of valid data (using checksumming, etc.). |
| 150 | |
| 151 | This process can be accomplished on any KL-style -8 interface |
| 152 | including PT08, etc., or on the printer port of VT-78 and all |
| 153 | DECmates. When used on the DECmates, there may be some minor |
| 154 | problems associated with the down-load which may have to be done as |
| 155 | the first use of the printer port after power-on, or some other |
| 156 | restriction. The loader includes a suggested instruction for DECmate |
| 157 | use if problematic (and raises the program length to 47 words). |
| 158 | Also, due to observed bugs in the operating system (OS/278 only), |
| 159 | there are restrictions on the use of ABSLDR.SV that cause certain |
| 160 | command forms to fail while other seemingly equivalent forms succeed! |
| 161 | This is documented in the latest K12MIT.BWR file in the distribution. |
| 162 | The command form stated in the K12IPL.PAL file is the only known form |
| 163 | that works correctly on these flawed systems. |
| 164 | |
| 165 | The format for down-load files is known as .IPL or Initial |
| 166 | Program Load format. It consists of a leader containing only |
| 167 | lower-case letters (code 141-177 only) followed by "printable" data |
| 168 | in the range 041 (!) through 140 (`). Each of the characters |
| 169 | represents six bits of data, to be read left to right as pairs, which |
| 170 | load into PDP-8 12-bit memory. The implied loading address is |
| 171 | always to start at 0000 of the implied field. The leader comment |
| 172 | contains documentation of which field of data from K12MIT.SV it is. |
| 173 | The trailer consists of one lower-case character followed by anything |
| 174 | at all. This is why it is crucial that DEL (177) not appear anywhere |
| 175 | in the body of the file. |
| 176 | |
| 177 | Throughout the file, all codes 040 or less are ignored. This |
| 178 | allows for spaces in the lower-case leader for better readability, |
| 179 | and for CR/LF throughout the entire file. CR/LF is added every 32 |
| 180 | words (64 characters) to satisfy cetain other systems' requirements. |
| 181 | The trailer contains documentation on a suggested SAVE command for |
| 182 | the particular data just obtained. |
| 183 | |
| 184 | 2) PDP-8 ENCODE format is the format of choice to obtain binary OS/8 |
| 185 | image files because of the validation techniques employed, etc. This |
| 186 | is the standard method of distributing K12MIT.SV as well as other |
| 187 | "critical" files such as TECO macros and other image files. In the |
| 188 | MS-DOS world there exists another very popular format known as .BOO |
| 189 | encoding. It would be useful to support this format on the PDP-8 as |
| 190 | well. |
| 191 | |
| 192 | .BOO format files are smaller because they use six-bit encoding |
| 193 | instead of five-bit encoding, or at least in theory. Both ENCODE and |
| 194 | .BOO use repeat compression techniques, but ENCODE can compress |
| 195 | 12-bit words of any value, while .BOO only compresses zeroes and that |
| 196 | itself is based on a byte-order view of the data. PDP-8 programs |
| 197 | often include large regions of non-zero words such as 7402 (HLT) |
| 198 | which would not compress when looked at as bytes. Such files would |
| 199 | show compression rations quite different from the norm. |
| 200 | |
| 201 | In any case, .BOO format is useful on the PDP-8 because it allows |
| 202 | inter-change with .BOO files created on other systems, such as PCs. |
| 203 | This allows the exchange of unusually formatted files, such as TECO |
| 204 | macros between PDP-8s and PCs. (Both systems support a viable |
| 205 | version of TECO.) |
| 206 | |
| 207 | The new KERMIT-12 utilities include a .BOO encoder and .BOO |
| 208 | decoder, known as K12ENB.PAL (or ENBOO.PAL) and K12DEB.PAL (or |
| 209 | DEBOO.PAL) respectively. They use .BOO encoded files unpacked in the |
| 210 | standard OS/8 "3 for 2" order to preserve the original byte contents |
| 211 | when the files originate from other systems. (Technically, .BOO |
| 212 | format doesn't require this, but the obvious advantages dictate it. |
| 213 | Anything encoded into .BOO format must merely have a 24-bit data |
| 214 | structure encoded into four six-bit characters, so in theory any |
| 215 | encoding of two adjacent PDP-8 12-bit words would be acceptable. By |
| 216 | additionally supplying the bits in OS/8 pack/unpack order guarantees |
| 217 | the inter-system compatibility as well.) |
| 218 | |
| 219 | There is an inherent weakness in the original .BOO format which |
| 220 | must be addressed. .BOO format files always end on one of two data |
| 221 | fields: either a repeat-zero compression field, or on a 24-bit field |
| 222 | expressed as four characters. Should the data in a 24-bit field |
| 223 | consist of only two or even one bytes, there are one or two |
| 224 | extraneous null bytes encoded into the field to complete it. |
| 225 | |
| 226 | Presumably the need to add the extra bytes is to allow validation |
| 227 | of the format. In any case, only the encoder knows just how many (0, |
| 228 | 1, 2) bytes are extraneous. We can presume that if the last byte is |
| 229 | non-zero, it is significant. If the last two are both zero, then the |
| 230 | last or possibly both are extraneous with no way to tell. |
| 231 | |
| 232 | On PC systems, the general trend is to ignore these one or two |
| 233 | extra bytes because so far there haven't been any complaints of |
| 234 | failure. I have personally discovered that a widely used PC .BOO |
| 235 | encoding program (written in C) erroneously adds two null bytes as |
| 236 | a short compression field beyond the data! This is not a .BOO format |
| 237 | issue, but rather a genuine program bug. Apparently few PC users are |
| 238 | concerned that encoding their files prevents transparent delivery to |
| 239 | the other end. |
| 240 | |
| 241 | In the OS/8 world, the situation is quite different. Each OS/8 |
| 242 | record is 256 words or 384 bytes. If even a single byte is added, |
| 243 | this creates an additional all-zeroes record. Besides wasting space, |
| 244 | it is conceivable that such a file could be dangerous to use under |
| 245 | OS/8 depending on content. (Certain files, such as .HN files are |
| 246 | partially identified by their length. File damage, such as |
| 247 | lengthening a file from two to three records will confuse the SET |
| 248 | utility, etc.) Many files cannot be identified as having been |
| 249 | artifically lengthened (and may be hard to shorten!), so this must be |
| 250 | avoided. |
| 251 | |
| 252 | I have invented a fix for the problem: repeat compression fields |
| 253 | are expressed as ~ followed by a count. 2 means two null bytes and |
| 254 | is thus the smallest "useful" field to be found. (It takes two |
| 255 | characters to express what would take 2-2/3 characters in encoded |
| 256 | format. One null would only take 1-1/3 characters, not two, so this |
| 257 | case is vestigial, but must be supported for the benefit of |
| 258 | brain-dead encoders.) The value of 0 means a count of literally zero, |
| 259 | thus ~0 is a "NOP" to a decoder. I have successfully tested MS-DOS |
| 260 | programs written in BASIC and C that decode .BOO files successfully |
| 261 | even if ~0 is appended to the end with no ill effects. (They |
| 262 | correctly ignored the appended fields.) |
| 263 | |
| 264 | In my encoding scheme, ~0 at the end of a data field containing |
| 265 | trailing zeroes means to "take back" a null byte. ~0~0 means to take |
| 266 | back two null bytes. Thus files encoded with ENBOO.PAL either end in |
| 267 | a repeat-compression field as before, or in a data encoding field |
| 268 | possibly followed by ~0 or ~0~0 if necessary. The corresponding |
| 269 | DEBOO.PAL correctly decodes such files perfectly. |
| 270 | |
| 271 | Should files encoded with ENBOO reach "foreign" systems, they |
| 272 | will do what they always do, i.e., make files one or two bytes too |
| 273 | long occasionally, with no other ill effects. Files originating from |
| 274 | such systems will certainly be lacking any trailing correction fields |
| 275 | and will cause DEBOO to perform as foolishly as MSBPCT. Extraneous |
| 276 | null bytes will appear at the end of the file in OS/8 just as in |
| 277 | MS-DOS in this case. (Note that if the file length is not a multiple |
| 278 | of 384 bytes, additional bytes are added by DEBOO as well, but this |
| 279 | is not a design weakness of .BOO format. It is caused by the clash |
| 280 | of fixed record size and a variable size format.) |
| 281 | |
| 282 | Hopefully, files originating on OS/8 will be decoded on OS/8 as |
| 283 | well, thus preserving file lengths. Most "foreign" files will |
| 284 | probably be ASCII, so the ^Z convention will allow removal of |
| 285 | trailing null bytes at either end. It is hoped that MS-DOS and other |
| 286 | systems "upgrade" their .BOO format files to be compatible with the |
| 287 | PDP-8 version. |
| 288 | |
| 289 | All KERMIT-12 files are available via the normal distribution |
| 290 | "paths" of anonymous FTP and/or KERMSRV. The user is directed to the |
| 291 | file /ftp/pub/kermit/d/k12mit.dsk as a "roadmap" to the entire |
| 292 | distribution. Each .PAL file includes assembly instructions. Most |
| 293 | use non-default option switches and non-default loading and saving |
| 294 | instructions, so each must be carefully read. The development |
| 295 | support files (TECO macro, .IPL generator, recent copies of PAL8, |
| 296 | CREF, etc.) are included in the total collection. Development is not |
| 297 | possible on RX01 systems due to inadequate disk space, but RX02's are |
| 298 | barely adequate with a lot of disk exchanges. (Future versions may |
| 299 | require larger disks for development.) |
| 300 | |
| 301 | Charles Lasner (lasner@watsun.cc.columbia.edu) |