Commit | Line | Data |
---|---|---|
81e70d48 PH |
1 | Date: Fri, 1 May 1992 17:00:00 EDT |
2 | From: Charles Lasner <lasner@watsun.cc.columbia.edu> | |
3 | Subject: DECmate I problems and more patching problems | |
4 | ||
5 | DECmate I problems. | |
6 | ||
7 | Attempts to use the distributed Kermit-12 Version 10g on a | |
8 | DECmate (I) system will certainly fail. The coding specific to the | |
9 | DECmate I was never tested until recently. Two key routines wait for | |
10 | status flags that never raise because the affected registers do not | |
11 | generate flag changes/interrupts. This is unrelated to general | |
12 | serial data handling which works as originally coded. | |
13 | ||
14 | There is a simple patch to the program to alleviate the problem: | |
15 | ||
16 | .LOAD SYS:KERMIT.SV/I$*$ [load the file in image mode and then | |
17 | ask for more input. The $ which is | |
18 | printed signifies the use of <ESC> | |
19 | as the command terminator. The * is | |
20 | printed by the system command | |
21 | decoder requesting further input. | |
22 | The second $ signifies the use of | |
23 | <ESC> to end input to the command | |
24 | decoder. The loader program | |
25 | terminates and control returns to | |
26 | the keyboard monitor.] | |
27 | ||
28 | .ODT [call in ODT to patch the program] | |
29 | ||
30 | 7/ 0007 0012 [change default baud rate to 2400; | |
31 | this is optional] | |
32 | ||
33 | 353/ 5352 7000 [make a JMP .-1 into a NOP] | |
34 | 10302/ 5301 7000 [make a JMP .-1 into a NOP] | |
35 | 12243/ 3607 3610 [bump the version number] | |
36 | ||
37 | ^C [^C to exit to monitor] | |
38 | ||
39 | .SAVE SYS KERMIT [save patched file] | |
40 | ||
41 | This also updates the release revision from 10g to 10h. Future | |
42 | versions will eliminate the overhead of the now defunct routines. | |
43 | ||
44 | The only DECmate versions remaining to be tested are: | |
45 | ||
46 | DECmate I with DP278B (the system used for testing has DP278A) | |
47 | DECmate III without internal modem | |
48 | DECmate III with internal modem | |
49 | DECmate III+ | |
50 | ||
51 | Patching problems. | |
52 | ||
53 | The restrictions placed on patching apparently stem from a bug | |
54 | going back at least as far as OS/8 V3D (likely further). Apparently, | |
55 | when a JSW value of 1 is used (as Kermit-12 does), the GET command | |
56 | doesn't work. Apparently, the system confuses the need to save the | |
57 | contents of 10000-11777 with the need to load it in the first place. | |
58 | Kermit-12 operates by first placing once-only code in the affected | |
59 | area, then discarding it in favor of a locked-in copy of the USR | |
60 | routine. To avoid overhead, the JSW value of 0001 is set to indicate | |
61 | there is no need to save this dead code when the USR is swapped in | |
62 | over it. Apparently, the GET command sets the =1 too early in the | |
63 | load process, so the code that uses the USR to carry out the actions | |
64 | of the GET operation doesn't properly load the Kermit-12 code. | |
65 | ||
66 | Consequently, the warnings documented in previous chapters of the | |
67 | .BWR file (below) apply in all cases. The interaction with CCL sited | |
68 | below may not apply in all versions, but the GET command problem is | |
69 | apparently universal. | |
70 | ||
71 | cjl | |
72 | ||
73 | ------------------------------ | |
74 | ||
75 | Date: Mon, 28 October 1991 20:00:00 EST | |
76 | From: Charles Lasner <lasner@watsun.cc.columbia.edu> | |
77 | Subject: Kermit-12 patching restrictions revisited yet again | |
78 | ||
79 | Even more operating system ills (will it ever end?): | |
80 | ||
81 | Still further investigation into operating system bugs in OS/278 | |
82 | V2 on DECmates reveals that the problem in even worse that realized | |
83 | two weeks ago (see previous .BWR article): | |
84 | ||
85 | When a SAVE command is executed from OS/278 involving a loaded | |
86 | handler, the SAVE operation fails! The contents of the files will be | |
87 | corrupted in general and will likely become (at least partially) all | |
88 | zeroes! The exact scope of the problem has not been ascertained, but | |
89 | certain loading tests reveal that the command fails even when | |
90 | accessing additional memory beyond field zero and one. | |
91 | ||
92 | All operations to SYS: or any device co-resident with SYS: (or | |
93 | when DSK:=SYS: which is typically the case in many systems but not a | |
94 | rule) are unaffected beyond the restrictions reported previously. | |
95 | ||
96 | Until recently, SAVE commands were of little interest to the | |
97 | casual user of OS/278, since program execution and ordinary file | |
98 | creation are unaffected. Since there are now several programs to be | |
99 | loaded and saved by users, the problem is more significent. Users of | |
100 | the direct loading method of acquiring KERMIT-12 are also in the | |
101 | affected category. | |
102 | ||
103 | Clearly all developers and anyone assembling any part of the | |
104 | KERMIT-12 package should be aware of this problem. As a precaution, | |
105 | all persons using the SAVE command for any reason are advised to use | |
106 | the form involving SYS: only to avoid this problem. (Advanced users | |
107 | can determine which handlers are possibly co-resident and are thus | |
108 | acceptable as well.) The resultant file can always be copied to any | |
109 | device as required after the fact. | |
110 | ||
111 | cjl | |
112 | ||
113 | ------------------------------ | |
114 | ||
115 | Date: Thu, 10 October 1991 05:00:00 EDT | |
116 | From: Charles Lasner <lasner@watsun.cc.columbia.edu> | |
117 | Subject: Kermit-12 patching restrictions revisited and .BOO problems | |
118 | ||
119 | More operating system ills regarding file loading: | |
120 | ||
121 | Further investigation into operating system bugs in OS/278 V2 on | |
122 | DECmates reveals that the problem is worse than first realized: | |
123 | ||
124 | When using GET or LOAD (ABSLDR) commands, especially when loading | |
125 | image files such as FIELD0.SV, FIELD1.SV (the partial load files from | |
126 | direct memory loading of K12MIT), or K12MIT.SV with /I, the JSW and | |
127 | starting field/address can become "mangled" into unusable values. | |
128 | One particular case achieved the impossible value of 6303 for a | |
129 | starting field change instruction (legal values are 6203 through 6273 | |
130 | by 10s). | |
131 | ||
132 | Consequently, the general recommendation for SAVE commands as | |
133 | used in various utilities throughout KERMIT-12 configuration etc., is | |
134 | to use explicit starting address and loading locations and JSW | |
135 | values. In short, always give a complete description of the SAVE | |
136 | operation under OS/278. For example, when direct-loading K12MIT | |
137 | through the printer port into the DECmate, the following commands | |
138 | should be used: | |
139 | ||
140 | }LOAD FIELD0.SV/I,FIELD1.SV/I$*$ | |
141 | ||
142 | }SAVE SYS K12MIT.SV 00000-07577,10000-17577;00200=0001 | |
143 | ||
144 | As discussed earlier, the CCL form of the ABSLDR LOAD command | |
145 | works even though other seemingly equivalent forms don't. The | |
146 | complete SAVE command forces all parameters to be taken explicitly | |
147 | from the command; no reliance on system "assumptions" or loading | |
148 | artifacts. Always use the complete values for loading taken from the | |
149 | relevant program documentation. | |
150 | ||
151 | Most users of KERMIT-12 are running OS/8 V3D, etc., where this | |
152 | sort of system bug isn't seen. In the future, all KERMIT-12 | |
153 | documentation will give the "verbose" form of the command to contain | |
154 | this OS/278 V2-specific problem. | |
155 | ||
156 | Regarding .BOO format encoding: | |
157 | ||
158 | The newest release of KERMIT-12 includes .BOO format encoding of | |
159 | all binary files and TECO macros as an alternative to ENCODE format. | |
160 | ENCODE format is still the preferred method of distribution, but .BOO | |
161 | format allows for use with other systems, such as MS-DOS. For | |
162 | example, TECO macros used with OS/8 TECO can be interchanged in .BOO | |
163 | format with similar files used with MS-DOS TECO. Intermediary sites, | |
164 | such as unix systems will not destroy the "delicate" nature of such | |
165 | files, etc. | |
166 | ||
167 | The KERMIT-12 .BOO utilities are NOT totally compatible with | |
168 | existing .BOO utilities on other systems! Just like OS/8 ENCODE and | |
169 | DECODE, ENBOO and DEBOO do a perfect encoding/decoding of OS/8 files | |
170 | into their original form. When used with "foreign" .BOO decoders, | |
171 | some unpredictable things might occur. | |
172 | ||
173 | Certain other .BOO encoders are known to throw in extraneous null | |
174 | bytes at the end of the file. Further, there is a design weakness in | |
175 | the original .BOO format that causes more null bytes to possibly | |
176 | appear. The KERMIT-12 programs utilize a superset of the original | |
177 | format to ensure correct encoding/decoding. When passing these files | |
178 | which now contain "correction bytes" to older decoders, the files are | |
179 | decoded with inflated lengths because the older decoders don't | |
180 | recognize the length correction. When passing files created by older | |
181 | encoders to the PDP-8, the resultant decoded files will also have | |
182 | inflated lengths because the older encoders failed to place | |
183 | correction bytes into the file. | |
184 | ||
185 | The general rule for dealing with .BOO files originating from | |
186 | other systems is that they may have incorrect lengths. The resultant | |
187 | files may be (falsely) padded out with extraneous null bytes. In any | |
188 | case, since the files generally have no blocking structure, the files | |
189 | will be padded by OS/8 up to the nearest whole record or multiple of | |
190 | 384 bytes anyway. Unless the file is ASCII and has a ^Z at the end, | |
191 | there is no way to determine the original intended file length. | |
192 | Files may be padded by null bytes introduced by other systems' bugs, | |
193 | the inherent weakness of the original .BOO format, or ultimately by | |
194 | OS/8 padding requirements. | |
195 | ||
196 | ASCII files from other systems may be adjusted by using an editor | |
197 | such as TECO which stops at the ^Z. A second generation of the | |
198 | transferred file may be somewhat shorter when processed this way. | |
199 | ||
200 | Should a file originating in OS/8 be intended for OS/8 use only | |
201 | (such as an encoding of a .SV file), it should not be decoded on an | |
202 | intermediate system, because a re-encoded version may differ from the | |
203 | encoded original because of ignored correction bytes, bugs, or the | |
204 | inability to insert correction bytes. Violating any of these rules | |
205 | could lead to OS/8 files corrupted into being too long. It is | |
206 | conceivable that these altered files are even dangerous to use under | |
207 | OS/8 because of their inflated lengths. (Certain files are validated | |
208 | by their restricted size, such as .HN files which must be exactly two | |
209 | or three blocks long depending on whether they are for one or two | |
210 | page handlers. If a one-page handler became three pages in file | |
211 | length, it could conceivably be confused with a two-page handler, | |
212 | etc.) | |
213 | ||
214 | cjl | |
215 | ||
216 | ------------------------------ | |
217 | ||
218 | Date: Sun, 7 October 1990 12:00:00 EDT | |
219 | From: Charles Lasner <lasner@watsun.cc.columbia.edu> | |
220 | Subject: Kermit-12 patching restrictions | |
221 | ||
222 | All Kermit-12 configuration done according to the documentation | |
223 | works "as advertised." Users are tempted to patch the distributed | |
224 | image file K12MIT.SV as a "quick and dirty" method to make small | |
225 | modifications such as changing the default baud rate, etc. There is | |
226 | "conventional wisdom" that this can be accomplished using GET, SAVE | |
227 | commands to allow the use of ODT; this method is ordinarily used with | |
228 | other OS/8 family programs. It has been reported that this does NOT | |
229 | work on OS/278, the usual operating system for the DECmates. The | |
230 | following method should be avoided (a work-around is offered later): | |
231 | ||
232 | .GET SYS KERMIT [setup current image for patching] | |
233 | ||
234 | .ODT [call in ODT to patch the program] | |
235 | ||
236 | 7/ 0007 0012 [change default baud rate to 2400] | |
237 | ||
238 | ^C [^C to exit to monitor] | |
239 | ||
240 | .SAVE SYS KERMIT [save patched file] | |
241 | ||
242 | This method follows the exact procedure described in virtually every | |
243 | OS/8 document regarding patching of image files. The cited example | |
244 | changes the default baud rate from 1200 Baud to 2400 Baud by | |
245 | replacing the value chosen from the DEC standard table for 1200 Baud | |
246 | with the applicable value for 2400 Baud. This value is stored within | |
247 | Kermit-12 as the corresponding twelve-bit word with all high-order | |
248 | bits zeroed. (The location used is 000007; this is valid for Version | |
249 | 10g, but could change in later versions.) | |
250 | ||
251 | This attempt to make changes the "conventional" way produces a | |
252 | corrupted image file of K12MIT.SV (renamed to KERMIT.SV in the above | |
253 | example) when using OS/278 Version 2, the usual operating system on | |
254 | the DECmate II, etc. This method probably works in earlier (OS/8 | |
255 | V3D, etc.) systems, however no attempt has been made to trace this | |
256 | bug in prior systems. A "fool-proof" method is required that works | |
257 | in spite of bugs in the operating system. | |
258 | ||
259 | A work-around was attempted using OS/278 V2 on a DECmate II hard | |
260 | disk system: | |
261 | ||
262 | .LOAD SYS:KERMIT.SV/I [load the file in image mode] | |
263 | ||
264 | .ODT [call in ODT to patch the program] | |
265 | ||
266 | 7/ 0007 0012 [change default baud rate to 2400] | |
267 | ||
268 | ^C [^C to exit to monitor] | |
269 | ||
270 | .SAVE SYS KERMIT [save patched file] | |
271 | ||
272 | This also fails! | |
273 | ||
274 | For reasons not understood yet, the following seemingly | |
275 | equivalent command DOES work: | |
276 | ||
277 | .LOAD SYS:KERMIT.SV/I$*$ [load the file in image mode and then | |
278 | ask for more input. The $ which is | |
279 | printed signifies the use of <ESC> | |
280 | as the command terminator. The * is | |
281 | printed by the system command | |
282 | decoder requesting further input. | |
283 | The second $ signifies the use of | |
284 | <ESC> to end input to the command | |
285 | decoder. The loader program | |
286 | terminates and control returns to | |
287 | the keyboard monitor.] | |
288 | ||
289 | .ODT [call in ODT to patch the program] | |
290 | ||
291 | 7/ 0007 0012 [change default baud rate to 2400] | |
292 | ||
293 | ^C [^C to exit to monitor] | |
294 | ||
295 | .SAVE SYS KERMIT [save patched file] | |
296 | ||
297 | This allows ODT commands to patch the file as intended, and also | |
298 | causes the subsequent SAVE command to work properly. All OS/8 family | |
299 | systems support this command (as long as CCL is enabled), so it will | |
300 | "always" work. | |
301 | ||
302 | For those users who run with CCL turned off, the following | |
303 | sequence will also work: | |
304 | ||
305 | .R ABSLDR [run the loading program directly] | |
306 | *KERMIT.SV/I [load Kermit in image mode] | |
307 | *$ [<ESC> is typed to terminate the | |
308 | loading process.] | |
309 | ||
310 | .ODT [call in ODT to patch the program] | |
311 | ||
312 | 7/ 0007 0012 [change default baud rate to 2400] | |
313 | ||
314 | ^C [^C to exit to monitor] | |
315 | ||
316 | .SAVE SYS KERMIT [save patched file] | |
317 | ||
318 | The newer OS/8 family systems generally can't turn off the CCL | |
319 | mechanism. Since the R and RU commands are typically disabled on | |
320 | newer releases, only the CCL command work-around applies. Users | |
321 | opting to disable CCL are likely running "older" systems, such as | |
322 | OS/8 V3D on DECtapes. On these systems, ANY of the above methods | |
323 | should work, because the problematic bug didn't exist on those | |
324 | systems. Had DEC not gone "backwards" we could have avoided this | |
325 | entire discussion! | |
326 | ||
327 | It is assumed the user will make "correct" patches to KERMIT-12; | |
328 | at least there is a "safe and proper" mechanism available to | |
329 | accomplish it! | |
330 | ||
331 | cjl | |
332 | ||
333 | ------------------------------ | |
334 | ||
335 | Date: Thu, 6 September 1990 12:00:00 EDT | |
336 | From: Charles Lasner <lasner@watsun.cc.columbia.edu> | |
337 | Subject: Kermit-12 potential problems | |
338 | ||
339 | A newly implemented ENCODE/DECODE method should eliminate the | |
340 | reported problems with regard to passing encoded binary files through | |
341 | problematic "paths." The method chosen is a variant on the 5-bit | |
342 | encoding algorithm suggested. Encoded files now pass right through | |
343 | all of the WPS-related utilities. It is necessary to acquire | |
344 | virtually all files of this re-release of KERMIT-12 since all ENCODed | |
345 | files are different, as well as the source programs for the | |
346 | ENCODing/DECODing utilities themselves. Due to the file being | |
347 | "bare", the TECO macro K12GLB.TEC is possibly defective when it | |
348 | arrives at a user site; it will now be ENCODed as K12GLB.ENC to avoid | |
349 | this problem. | |
350 | ||
351 | The KERMIT-12 source files are different due to maintenance work, | |
352 | requiring the user to obtain the re-released files. The sources now | |
353 | include a file to "pre-clear" memory. This aids in reducing the size | |
354 | of the ENCODed binary file K12MIT.ENC since undefined areas are no | |
355 | longer "relics" of random values, rather they are all set to 0000 | |
356 | octal. The long strings of identical words will be eliminated since | |
357 | the new encoding format does repeat compression. | |
358 | ||
359 | KERMIT-12 has still not been tested on any DECmates other than | |
360 | the DECmate II, as no volunteers have come forward with the proper | |
361 | hardware: | |
362 | ||
363 | DECmate I with DP278A | |
364 | DECmate I with DP278B | |
365 | DECmate III without internal modem | |
366 | DECmate III with internal modem | |
367 | DECmate III+ | |
368 | ||
369 | A tentative volunteer for the DECmate I with DP278A configuration | |
370 | has been contacted, but testing has not yet started. | |
371 | ||
372 | cjl | |
373 | ||
374 | ------------------------------ | |
375 | ||
376 | 26-Jul-90 1:15:43-GMT,15259;000000000001 | |
377 | Return-Path: <lasner@cunixf.cc.columbia.edu> | |
378 | Received: from cunixf.cc.columbia.edu by watsun.cc.columbia.edu (5.59/FCB) | |
379 | id AA26223; Wed, 25 Jul 90 21:15:41 EDT | |
380 | Received: by cunixf.cc.columbia.edu (5.59/FCB) | |
381 | id AA11871; Wed, 25 Jul 90 21:16:19 EDT | |
382 | Date: Wed, 25 Jul 90 21:16:18 EDT | |
383 | From: Charles Lasner <lasner@cunixf.cc.columbia.edu> | |
384 | To: fdc@cunixf.cc.columbia.edu | |
385 | Subject: This was sent out to PDP8-LOVERS | |
386 | Message-Id: <CMM.0.88.648954978.lasner@cunixf.cc.columbia.edu> | |
387 | ||
388 | I thought you might want to see this; it refers to the encoding | |
389 | problem I reported for a user with the problem (he has no net | |
390 | capability) in the programs using that encoding scheme we | |
391 | discussed... | |
392 | ||
393 | From: Charles Lasner (cjl) | |
394 | To: PDP8-LOVERS | |
395 | Subj: Feedback on encoding issues regarding archived files. | |
396 | ||
397 | I have written a pair of OS/8 programs to ENCODE and DECODE | |
398 | binary files into an "ASCII-fied printable" format. Those of you | |
399 | familiar with either uuencode/uudecode or .BOO format will understand | |
400 | my intentions. They were originally written for the purpose of | |
401 | distribution of binary (.SV) files of KERMIT-12 by Columbia | |
402 | University in NY as part of the standard KERMIT collection (K12*.*). | |
403 | Columbia imposes a restriction on all files: they must be distributed | |
404 | in ASCII only. This is to ensure proper distribution regardless of | |
405 | the "path" taken between Columbia and the end user. Be advised that | |
406 | various problematic E-mailers, ASCII-EBCDIC EBCDIC-ASCII | |
407 | translations, filters for reserved codes, known problematic character | |
408 | substitutions, etc. are lurking out there! Consider yourself lucky if | |
409 | you get your sender's copy intact without some form of "cosmetic" | |
410 | reformatting. By encoding the binary files into an appropriate | |
411 | subset of ASCII, these problems hopefully are avoided. While we | |
412 | can't prevent ALL problems, we can usually tackle the most likely | |
413 | ones. | |
414 | ||
415 | My original design was based on a discussion I had with Frank da | |
416 | Cruz of Columbia University (of KERMIT fame) regarding what to | |
417 | restrict ourselves to in a robust format. He was "unhappy" with some | |
418 | of the vulnerabilities of the uuencode and .BOO formats, which while | |
419 | popular, are not impervious to some "real" problems that have come | |
420 | up. We essentially designed an archiving format that was PDP-8 | |
421 | oriented, but not limited to -8s only. Some of the highlights of the | |
422 | format are: | |
423 | ||
424 | a) File format restricted to "printable" six-bit subset of ASCII | |
425 | only. All else ignored; this was to minimize the "garble" factor, | |
426 | yet maintain a fairly high rate of efficiency: two ASCII characters | |
427 | equal one PDP-8 12-bit word. (This has proved to be problematic and | |
428 | is why we are here!) | |
429 | ||
430 | b) The archive file contains imbedded commands, not implied ones. | |
431 | By validating the commands, you can "trust" the contents. Commands | |
432 | are available for whatever purpose arises. Already implemented are | |
433 | commands to start ("(FILE filename.ext)") and end ("(END | |
434 | filename.ext)") the imbedded file, and an official comment command | |
435 | ("(REMARK anything)") to help document the contents of the rest of | |
436 | the file. This is of course expandable. My OS/8 programs create all | |
437 | three types of commands. The start and end commands also | |
438 | theoretically allow multiple files in an archive, but I ignore the | |
439 | end command in the decoder and only allow one file per archive. I do | |
440 | support the start command completely, which includes a suggested name | |
441 | for the file. This name can be used at the user's option, or can be | |
442 | locally overridden. The encoding program inserts the original file | |
443 | name in this field, as this is of course the most likely name for the | |
444 | file at the other end. | |
445 | ||
446 | c) The archive contains a checksum for its contents to ensure the | |
447 | validity of the file. | |
448 | ||
449 | d) All "white space" character considerations are ignored; imbedded | |
450 | extraneous space characters, formfeeds, extra CR/LF, etc. are | |
451 | harmless. The CR/LF must be present at appropriate intervals, but | |
452 | this is only a problem with files passed through unix systems that | |
453 | delete the CR. Since OS/8 requires the CR and LF to be considered | |
454 | "printable", this is not a problem; the use of programs such as | |
455 | c-KERMIT will insert the CR if configured properly (SET FILE TYPE | |
456 | TEXT). Programs such as Rahul Dhesi's FLIP program are available to | |
457 | correct the problem easily if necessary: FLIP -m *.* or equivalent | |
458 | will remedy this. | |
459 | ||
460 | e) There is an internal record length of 64 characters with framing | |
461 | characters, to ensure the validity of each record. This prevents the | |
462 | file from getting "out of sync" with its original. This causes each | |
463 | line to be 68 characters including CR and LF, which is usually | |
464 | reasonable to most systems. | |
465 | ||
466 | Unfortunately, this scheme has proved to be flawed in an | |
467 | important way that "matters." This format must deliver files to | |
468 | OS/278 systems by the prevailing paths of existent systems connected | |
469 | to DECmates containing only the normally present DEC release | |
470 | software. This could include sending the files via DEC-DX through | |
471 | WPS8, or acquiring the files on either DECmate CP/M-80 or DECmate | |
472 | MS-DOS, possibly using KERMIT-80, or KERMIT-MS as appropriate. If a | |
473 | file is received in the CP/M-80 environment, it can be converted to | |
474 | WPS8 format using a DEC-supplied program called WPSCONV. If a file | |
475 | is received in the MS-DOS environment, it can be converted to WPS8 | |
476 | format using a DEC-supplied program called CONVERT. Incidentally, | |
477 | CONVERT can also convert CP/M-80 files as well, using MS-DOS format | |
478 | as an intermediary; WPSCONV is known to have bugs, which were | |
479 | corrected in CONVERT (which requires the MS-DOS hardware, not just | |
480 | the CP/M-80 hardware). These CP/M-80 and MS-DOS files can also come | |
481 | to the DECmate directly from a Rainbow as well, since the | |
482 | corresponding Rainbow systems are format compatible with the DECmate. | |
483 | DECmate MS-DOS additionally supports IBM-PC diskettes (160K or 180K | |
484 | single-sided only and read-only) as yet another source. Thus there | |
485 | are many paths to WPS8 versions of our files. | |
486 | ||
487 | The problem with these methods is that apparently there is a bug | |
488 | in OS/278 WPFLOP, the only distributed conversion program a user | |
489 | would already have on his OS/278 system. (We haven't actually | |
490 | isolated the problem to WPFLOP, as the complaining user was taking | |
491 | the files from MS-DOS via CONVERT then to OS/278 via WPFLOP; | |
492 | conceivably the problem is in CONVERT, but in any case the problem | |
493 | exists somewhere in this supported path.) | |
494 | ||
495 | The internal encoding used is to break the 12-bit word into two | |
496 | six-bit halves. Each half is in the range 00-77 octal. Adding 041 | |
497 | to the value yields characters in the range of ! through ` or 041 | |
498 | through 140 octal. The codes for 101-132 are A through Z and can be | |
499 | replaced by 141-172 for a through z if desired. This prevents | |
500 | case-sensitivity which is another possible network anomaly. We | |
501 | identified the DECmate problem as an anomaly regarding @ and `. The | |
502 | character codes for 100 and 140 are not treated uniquely, so the | |
503 | resulting OS/278 file is an inaccurate representation of the file. | |
504 | The decoding program correctly failed the conversion on a checksum | |
505 | error, so at least the user was aware of the problem! | |
506 | ||
507 | As the PDP8-LOVERS, we will hopefully acquire an archive site for | |
508 | our files, so all of these considerations will apply. We need a file | |
509 | format that is "bullet-proof" to avoid problems like this one. I am | |
510 | soliciting suggestions for improvements on this encoding scheme (and | |
511 | any other overall file format suggestions as well) to provide an | |
512 | effective solution. The resultant programs will be added to the | |
513 | KERMIT-12 collection freely distributed by Columbia as well as other | |
514 | sources (DECUS, etc.). | |
515 | ||
516 | Some suggestions have already been made: | |
517 | ||
518 | 1. Just "quick-fix" the problem by providing an alternate character | |
519 | to the ` to make it non-anomalous with @. The available choices are | |
520 | { | } and ~ only. The DEL character (octal 177) is unsuitable for | |
521 | other reasons; all other characters are either already used, or | |
522 | unprintable, or lower-case. This has the advantage of being most | |
523 | compatible with the existing programs, since the original character | |
524 | code can be supported as well; the "preferred" character would be | |
525 | generated by all future versions of the ENCODE program, and existing | |
526 | files could be trivially edited for compatibility as needed. This | |
527 | would have to be tested -- it is possible that the bug would | |
528 | persist. The choice is further narrowed to { and | only, since 175 | |
529 | and 176 are sometimes treated as alternates to ESC. It is likely | |
530 | that systems which "mangle" the case of a character which is | |
531 | alphabetic could also do the same to { | } and ~ making them [ \ ] | |
532 | and _ respectively. This makes the entire suggestion unworkable. | |
533 | ||
534 | 2. Change the format to "Hexafied-ASCII" where each PDP-8 12-bit | |
535 | word becomes represented by three characters from a 16-character set | |
536 | such as 0-9,A-F or A-P. The alphabetic codes would be immune to case | |
537 | conversion, and virtually every system supports this subset of ASCII. | |
538 | Instead of 64 characters on a line representing 32 12-bit words, each | |
539 | line would be 72 characters on a line representing 24 12-bit words | |
540 | (not counting framing characters and CR/LF). This also allows for | |
541 | many additional codes if needed. This scheme has the drawback of | |
542 | making the encoded file more inefficient, as the file will generally | |
543 | be 50% longer than those created by the original six-bit scheme. | |
544 | This robust scheme is workable. | |
545 | ||
546 | 3. Modify 2. to include some form of compression. The easiest is to | |
547 | incorporate repeat compression. One simple scheme is to use an | |
548 | indicator character (R was suggested) as a prefix for an encoded | |
549 | count. It could be followed by three characters encoding the value | |
550 | of the 12-bit word and two characters encoding the value of the | |
551 | repeat count. Since this occupies six characters, as does two | |
552 | adjacent 12-bit encoded words, this scheme saves space when used for | |
553 | repeat compression lengths greater than two. The compressed field is | |
554 | the same length as two compressed "triplets", so overall file | |
555 | validation techniques wouldn't require special-case checks, as long | |
556 | as trailing "fill" characters were allowed for the last record before | |
557 | the short checksum record (which is signalled by its length). (T was | |
558 | suggested for this trailer character to be used to pad the last line | |
559 | with 0-69 characters.) This allows for compressing 3-258 repeated | |
560 | 12-bit words into six characters. This would benefit files | |
561 | containing large areas of zeroes or HLT instructions, etc., as this | |
562 | can be the actual contents of binary files. If a .BN file created by | |
563 | PAL8, etc. is loaded and saved, then "junk" areas are created in the | |
564 | .SV file. Unfortunately, this is the norm, and the junk increases | |
565 | the size of the encoded version of the file. If the .BN file is | |
566 | loaded AFTER loading an all-zeroes file such as the binary output of: | |
567 | ||
568 | *0 | |
569 | ||
570 | ZBLOCK 7600 | |
571 | ||
572 | $ | |
573 | ||
574 | or equivalent as necessary (extended memory zeroed if required, | |
575 | etc.), then the file has all-zeroes gaps in it. These would repeat | |
576 | compress out using this scheme. Incidentally, an additional | |
577 | advantage of this method is that the resulting "cleaner" core-image | |
578 | file is slightly easier to disassemble, in case the source is lost. | |
579 | (Anyone who ever disassembled a .SV file or equivalent understands | |
580 | what I mean!). This also makes a binary papertape file (such as a | |
581 | diagnostic) loaded into a .SV file a little easier to follow when | |
582 | consulting the write-up, as memory is zeroed in between the locations | |
583 | referenced in the listing. The .SV file is smaller when encoded than | |
584 | the .BN file due to elimination of the paper-tape encoding overhead. | |
585 | OS/8 files of diagnostics could therefore be more efficiently | |
586 | archived as .SV files (encoded) than .BN files. | |
587 | ||
588 | 4. Change to a 5-bit encoding with compression. This would use 32 | |
589 | codes chosen from A-Z, 0-9 to encode the file five bits at a time per | |
590 | character. Five PDP-8 12-bit words would be encoded in 12 | |
591 | characters. Since PDP-8 binary files are always multiples of 128 | |
592 | 12-bit word pages, there would need to be 4.8 "junk words" at the end | |
593 | of each block to encode the implied length of 130 words/block. Each | |
594 | line would be 78 characters (plus framing characters and CR/LF) so | |
595 | that four lines encodes a PDP-8 page, just as in the original six-bit | |
596 | scheme (the original scheme used 64 characters per line!). The last | |
597 | line of the file would contain 0-77 padding characters as necessary | |
598 | to maintain the line width as before. Repeat compression schemes can | |
599 | be expressed in any way that is a multiple of 12 characters; perhaps | |
600 | one or two adjacent expressions of repeat compression similar to | |
601 | above. Expected efficiency of this scheme is similar to the original | |
602 | six-bit method, or possibly slightly better; if compression is NEVER | |
603 | useful, then the file is 1.2 times as large. | |
604 | ||
605 | There is an implementation restriction placed on the DECODE | |
606 | program: it should be relatively short, since it must be distributed | |
607 | in source form. It must also be written in a subset of PAL8 | |
608 | compatible with the original PAL8 of the PS/8 days (ugh!) to ensure | |
609 | viability on any OS/8 family system. PAL8 Version B0 from OS/278 is | |
610 | distributed in ENCODed form, so this restriction need not apply to | |
611 | any other programs such as the ENCODE program or KERMIT-12, etc. It | |
612 | has been determined that PAL8 Version B0 and the companion CREF | |
613 | Version B0 will correctly function on any OS/8 family system on any | |
614 | PDP-8 member suitably configured to run the operating system the user | |
615 | already has running. (There is a minor anomaly when using input files | |
616 | from the TTY: handler; see K12MIT.DOC for a detailed explanation.) | |
617 | CPU extensions such as BSW and IAC RAL are not present in these | |
618 | programs, as was the original intention of OS/8 (which eventually was | |
619 | lost as newer members of DEC's programming staff were ignorant of | |
620 | this problem!). It is acceptable to have a "bare-bones" subset of | |
621 | the DECODE program distributed in "old" PAL8-compatible source form, | |
622 | along with a "fancier" version written in a more modern PAL8 | |
623 | language, as the binary could then be DECODed with the subset DECODE | |
624 | program, or the source could be assembled with PAL8 Version B0 to | |
625 | "bootstrap" the "full" version of the DECODE program as necessary. | |
626 | ||
627 | For those of you who can't wait, and want these utilities as they | |
628 | stand (using the fallible six-bit method), they are available via | |
629 | anonymous FTP from Columbia University (watsun) as | |
630 | /w/kermit/d/k12dec.pal and /w/kermit/d/k12enc.pal for the DECODer and | |
631 | ENCODer respectively. More information is available in | |
632 | /w/kermit/d/k12mit.doc or /w/kermit/d/k12mit.pal regarding use of | |
633 | PAL8 Version B0, other assemblers (such as PAL10 or P?S PAL) or other | |
634 | KERMIT-12 issues, etc. | |
635 | ||
636 | Charles Lasner (lasner@cunixf) | |
637 | cjl | |
638 | ||
639 | ------------------------------ | |
640 | ||
641 | Date: Fri, 4 May 90 13:55:02 EDT | |
642 | From: Charles Lasner <lasner@watsun.cc.columbia.edu> | |
643 | Subject: Kermit-12 problems | |
644 | ||
645 | If the release files of KERMIT-12 are brought to DECmate MS-DOS | |
646 | via any of the various paths that can be used (such as from a Rainbow | |
647 | in either CP/M RX50 or MS-DOS RX50 format, etc.; in this particular | |
648 | case the reporting user obtained them using IBM-PC SSDD 180k 5-1/4" | |
649 | PC-DOS format.) then the files are available as DECmate II MS-DOS or | |
650 | CP/M-80 files on one of its standard devices (a:,b:,c:,d: floppies or | |
651 | e:,f:,g:,h: hard disk volumes). | |
652 | ||
653 | The ultimate goal is to get these files (un-scathed!) to DECmate | |
654 | II OS/278 for KERMIT-12 installation. The standard DEC CONVERT | |
655 | program alledgedly can convert any combination of MS-DOS or CP/M-80 | |
656 | or WPS/8 from/to each other. By converting the files to WPS/8 | |
657 | documents, the files can be translated to OS/278 later (using the | |
658 | OS/278 WPFLOP utility). | |
659 | ||
660 | There is a problem with DEC's CONVERT.EXE: it only CORRECTLY | |
661 | supports Rainbow/DECmate RX50 MS-DOS and CP/M diskettes, so the other | |
662 | formats (8" CP/M-80 diskettes and one-sided PC diskettes) have to be | |
663 | pre-converted with the appropriate copy commands to a supported | |
664 | diskette or hard disk volume first before using CONVERT. This is not | |
665 | a big problem, as we are merely using standard procedures, but the | |
666 | point is that much of this is undocumented or obscure. (I had to | |
667 | help the reporting user to copy his files to a "friendlier" device | |
668 | for CONVERT's benefit which only delayed our discovery of the REAL | |
669 | problem!) | |
670 | ||
671 | The CONVERT program alledgedly supports ASCII/WPS format | |
672 | conversion from/to any of MS-DOS, CP/M-80, or WPS/8 (but only on | |
673 | a:,b:,c:,d:,e:,f:,g:,h: logical drives, not on the other hardware or | |
674 | media possibly hooked up to the DECmate!). Our purpose is to move | |
675 | the K12MIT files to WPS/8 format. This can be attempted with | |
676 | standard commands of CONVERT, but there apparently is a bug: | |
677 | ||
678 | When you boot to OS/278 and retrieve the WPS/8 documents (via | |
679 | WPFLOP) which are the ENCODed files of KERMIT-12 as OS/278 files, | |
680 | there is a character anomaly between two encoding characters | |
681 | (specifically @ and `) that destroys the integrity of the affected | |
682 | file. This is possibly due to a bug in OS/278 WPFLOP, but more | |
683 | likely is a problem with MS-DOS CONVERT. Regardless of the | |
684 | perpetrator, this path is not viable to obtain the ENCODed files of | |
685 | the KERMIT-12 release. | |
686 | ||
687 | Fortunately, the source files are not affected, as the anomalous | |
688 | characters are not part of the PDP-8 assembly language, and only | |
689 | comments could be affected. (As far as I can tell, there aren't any | |
690 | affected characters even in the comments!) It is therefore necessary | |
691 | to assemble KERMIT-12 directly from the sources when installing it on | |
692 | the DECmate II if obtaining it via any path which includes | |
693 | CONVERT/WPFLOP. The other ENCODed files are for PAL8 Version B0 and | |
694 | CREF Version B0 which are already present on the DECmate II as part | |
695 | of the standard release of OS/278 for the DECmate II and are thus | |
696 | superfluous. All ENCODed files can be recreated from OS/278 itself | |
697 | using the sources, etc., so the intended release files can be | |
698 | recreated for distribution to other OS/278 sites (bypassing the | |
699 | CONVERT/WPFLOP path). Future versions of the DECODE program will | |
700 | obviate this problem when an appropriate alternative format is | |
701 | supported properly which is immune to DEC's glitch. | |
702 | ||
703 | A related problem surrounds the GLOBAL TECO macro K12GLB.TEC (aka | |
704 | GLOBAL.TEC). Due to the "delicate" nature of TECO macros, they could | |
705 | get "mangled" by the time they get to a user site. Future releases | |
706 | of KERMIT-12 will "protect" the macro by ENCODing it into K12GLB.ENC. | |
707 | It has also been reported that there are problems running the macro | |
708 | on certain releases of OS/8 family TECO and on other TECOs for other | |
709 | machines, and also problems running certain versions of OS/8 TECO on | |
710 | the DECmates. The author will investigate this problem eventually, | |
711 | but the main usage of the macro is for KERMIT-12 source maintainence | |
712 | on an OS/8 V3D system using the corresponding version of TECO; it is | |
713 | beyond the scope of KERMIT-12 development to investigate the myriad | |
714 | releases of TECO and their hardware and operating system | |
715 | dependencies; perhaps some TECO hackers can assist us! | |
716 | ||
717 | An obscure problem indeed! Users give good feedback... | |
718 | ||
719 | Can you suggest a fix for the CONVERT/WPFLOP-induced corruption? | |
720 | One is to allow the current format as a subset, but use a | |
721 | substitution character for the garbled character. Our character set | |
722 | is the 64 characters from ! through `, so the anomalous occurrences | |
723 | of @ are problematic. If we change the preferred character for ` to | |
724 | a lower-case letter (only octal 141 up is available, so let's assume | |
725 | the use of a) we avoid the CONVERT/WPFLOP problem. Newer released | |
726 | ENCODed files would then be immune to the treachery, but would | |
727 | require the newer DECODing program (or use TECO to change all | |
728 | occurrences of a to ` and then use the old DECODE program). | |
729 | ||
730 | Should we abandon this inner format altogether? We could use an | |
731 | even more robust format like ASCII hex: 0-9 and A-F (allowing a-f as | |
732 | well!) at the expense of longer files (currently 2 characters=12 | |
733 | bits, but would become 3 characters=12 bits). This would also hold | |
734 | up better through EBCDIC network conversion... | |
735 | ||
736 | cjl |