20050814/taylors Version 1.0, initial revision. 20050815/taylors hp.c Fix detokenization of IMAGE statements. 20050824/taylors utils.c Fix computation of minutes, seconds and tenths in tape labels. TENTHS is two's-complimented there. tsb.c Remove display of tio.fd and add a space when displaying the tape image disk-file name. 20050827/taylors hp.c Fix computation of filesize for 2000/ACCESS ASCII files for correct assertion of expected vs. read file size. 20050828/taylors tsb.c Add coding to accomodate multi-tape dumps. Multiple uses of the "-f" flag will now specify the input files of multi-tape volumes. 20050828/taylors Version 1.1 release. 20050829/taylors all Change multiply-specified -v options to new -d x... options. 20050830/taylors hp.c Some filenames on ACCESS tapes can contain embedded spaces in place of slash (/). 20050831/taylors hp.c DEF FNX(y) wasn't always getting 'y' produced. tsb.c Don't try to deallocate the program block for 2000E--it's static. 20050901/taylors Version 1.2 release. 20050902/taylors tsb.c Add support for 2000A/2000B/2000C and 2000E SDUMP tapes. hp.c Fixups in 2000E Equipment Table display. Still some DIREC table confusion. tsb.c Fix the date displayed in 2000E DIR tape entry listings. all Add DEBUG_ASCII and DEBUG_TAPEBLOCKS debugging levels. Version 1.21 release. 20050909/taylors tsb.c Fixup to handle individual reels from a multi-reel volume. Version 1.22 release. 20050913/taylors tsbc.c A new command, tsbc, will convert extracted tokenized files to ASCII, adding (by default) .ascii to the new file name. many Changes to extract_entry to accomodate tsbc. Files changed are Makefile, tsb.h, hp.c, decl.h, tsbc.c (new) and tsb.c. Version 1.23 release. 20050914/taylors util.c New validate_id and validate_filnam functions. hp.c Utilize validate_id and validate_filnam in load_program_entry. 20050914/taylors tape.* Added write routines to tape library. Version 1.24 release. 20050915/taylors tsb.c Implementation of Tape Create (-c) functionality. Version 1.25 release. 20050917/taylors tsb.c, Split tsb_create from main. create.c 20050918/taylors hptoken.h Make all tables the same (struct BASIC_token). 20050919/taylors hp.c Fix ALPHA-NUM (A0 thru Z9) variables in parameters (i.e. DEF FNA(X1)). tokenize.c Begin adding tokenization code. Add DEF FNn() support to tokenization. Add IMAGE/FILES/REM support to tokenization. Add LET/implied-LET support to tokenization. 20050920/taylors hptsb.h Increase SST and SLT tables from 50 to 100 entries to accomodate larger System Segment and System Library record counts in ACCESS. 20050921/taylors util.c Fix IEEEtoHPf conversions for negative exponents. ans_formula.c numeric literal support, including scientific notation. ans_formula.c Added get_number() function, separated out from ans_formula. tokenize.c Add GOTO support to tokenization. 20050922/taylors tokenize.c Add GOSUB support to tokenization. ans_formula.c Change return value to be new stmt position. tokenize.c Add "IF ERROR" and "IF END #" tokenization. Add numeric IF tokenization (String capabilities not yet in ans_formula). Add RETURN, END and STOP tokenization. Add NEXT tokenization. Add RESTORE tokenization. Add LOCK and UNLOCK tokenization. 20050923/taylors tokenize.c Add INPUT tokenization. 20050924/taylors Add ADVANCE tokenization. Add CONVERT tokenization. Add SYSTEM tokenization. 20050925/taylors Add CREATE tokenization. Add PURGE tokenization. Add UPDATE tokenization. Add ASSIGN tokenization. Add LINPUT tokenization. Add FOR tokenization. Add IF tokenization. Add DATA tokenization. Add DIM tokenization. Add COM tokenization. Add READ tokenization. Add single-dimension subscript handling (get-subscript). 20050926/taylors tokenize.c Add two-dimension subscript handling (get-subscript). ans_formula.c Add re-entry capability to ans_formula, % get_subscript. Fully implemented subscript handling and (...) arithmetic expressions. Verified already-implemented tokenization functions with new subscript and arithmetic capabilities. 20050927/taylors tokenize.c Add ENTER tokenization. ans_formula.c If incoming buffer is empty, exit immediately and don't store sttype. tokenize.c Fixups for IMAGE, REM and FILES string storage. Fix for STEP value in FOR. Fixes to subscript handling when analysing formulas. Fixed LET processing. Reversed search order on token tables to catch "<>" before "<" when searching for operators. Implemented PRINT and PRINT # tokenization. Fix string literal everywhere... 0x377 != 0377 (duh). Change uint32 to int32 indexes in match_token. hp.c Cleaned up obsolete debugging output. ans_formula.c Fix get_number to allow leading decimal points. Fix get_number for numbered numeric variables on the left-hand side of '=' numeric expressions. Allow ans_formula to recursively call itself via get_source_string when handling LEN(). tokenize.c Free up malloc'd memory after each line tokenized. 20050928/taylors tokenize.c Add PRINT USING tokenization. Add CHAIN tokenization. Fix to PRINT numeric handling. 20050929/taylors hp.c Fix operand pair processing -- a dropped bit was losing P$ detokenizations (weirdness, for sure). tokenize.c Fixups to string handling in most operator-tokenizing codes. Fixes to file-reference tokenizations. Remove stack/recusive-calls -- the compiler does indeed handle this properly. Fixed subscriting of image string in PRINT USING. ans_formula.c Implemented POS() and NUM(). Implemented CHR$(). When exiting because of disallowed operators, write out a pending operator that is already processed. Set up utility main routine to compare original ASCII programm to tokenize.c de-tokenized-and-re-tokenized text and report errors. Fixups to CONVERT formatting. all Tokenization compared to de-tokenization. Tested against about 450,000 program statements. ASCII representations after conversions (both ways) match with only MAT (unimplemented) and some 2000/F->ACCESS conversion issues exceptional. 20050929/taylors Version 1.30. 20050930/taylors tokenize.c Fixups for right-hand parenthesis duplication tokenization in CHR$(). ans_formula.c Fix for ** in numeric expressions. Fix for FNx() instead of FNx[]. tokenize.c Add MAT tokenization. 20050930/taylors Version 1.31. All tokenizations now match de-tokenization on the ASCII level. All (believed) known valid 2000/ACCESS statements and functions now implemented. Tokenization level not yet compared. util.c Hardcode adding decimal point to values -99999 and -999999 in ftoHP(). This is necessary due to float rounding and identifying those two numbers. HP TSB adds decimals to them on output. 20051001/taylors tokenize.c Fix for "READ #x,y". tokenize.c Disabled stand-alone tokenization and moved to tsb.c. tsb.c Added tokenization (-T) to create (-c) function. create.c Created tsb_tokenize() for tokenization (-T). tokenize.c Tokenization messages now indicate statement text, graphical position pointer to the error, error code and error messages text. 20051001/taylors Version 1.32. 20051002/taylors Standardized error messages. Added check for proper receiving variable type in CONVERT. Modify tokenize (-T) to load ASCII program, tokenize, and then detokenize. Still being used for testing porpoises. tsb.c Fixed processing state (bitrot) when detecting 2000F and ACCESS DUMP tapes. Begin comparing tokenization at the octal level to TSB-generated tapes. tokenize.c Accomodate a bug in TSB with arguments to functions: If a function takes only one argument, tokenize an extra word of zero as a no-op--TSB writes all function this way unnecessarily which is a waste of space, but probably was overlooked or saved on coding the OS parsing. ans_formula.c Found an errant zero-byte token being written at the end of ans_formula(); tokenize.c Fixed ASSIGN file reference syntax. Fixed IF..THEN to combine the target statement number with the THEN operator token word. Object variable in MAT statements should be type 3 (unknown size matrix var) instead of 4 (simple num var). 20051003/taylors ans_formula.c Remove varnum from ans_formula(), as it's not needed. Remove all pre-get_subscript variable stores. Added mode to ans_formula to differentiate between ='s "Assignment" and "Equals" tokenizations. When writing string literals, ensure the unused byte in odd-length literals is null. tokenize.c Remove varnum from calls to ans_formula. Fix for REM statements with only zero or one characters. Redo get_subscript to perform the variable storage into stmt; remove all pre-get_subscript variable stores. Remove many dprintf()s no longer needed for debugging. Force all variables printed with "MAT PRINT" to type '3' (Unknown-size array variable). When writing string literals, ensure the unused byte in odd-length literals is null. 20051003/taylors all Tokenization and de-tokenization has been verified to be 100% accurate to TSB/ACCESS when converting program XREF.A000. 20051004/taylors tokenize.c Change get_subscript() to honour a new global variable FNparse, which when set will vause FNvarname and FNvartype to be compared to the incoming varname and vartype, setting op15 in the operand pair token word when they match. This is to properly set FNx(Z) variable Z in the current line to a Parameter state rather than a regular variable state. create.c Increate alloc size of the incoming ASCII line during tokenization by one to allow for the terminating NULL character. util.c Fix to IEEEtoHPf to make the HP Mantissa 2's-complimented (Was 1's-compliment in the code) when the mantissa sign is set. tsb.c Add call to fesetround(FE_UPWARD) to try to match the HP-2000 floating-point format when converting to/from IEEE. tokenize.c Fixups to MAT: Ensure all source variables are type 3 (for non-specified dimensions), or 1 and 2 (for specified dimensions). Ensure the target variable is also typed as a parameter. STILL need to work in ZER[x] dimensioning, etc. 20051005/taylors util.c Tweaks to IEEE->HP float conversions. Still trying to get as close as possible to HPs representations before hardcoding individual conversions. 20051008/taylors hp.c Handle NULLs in IMAGE, REM and FILES. For testing purposes, and probably just leave it this way, convert embedded NULLs to '@'. util.c Added hard-coded corrections to display of floating values less than .001. printf() '%G" refuses to use scientific notation when significant digits are fewer than 7 and leading zeroes follow the decimal point. ftoHP() should now match TSB numeric output. 20051010/taylors Version 1.33. all Split struct definitions for token tables into hptoken.h and create extern declarations in hptokdecl.h to get rid of unused-definition warnings. ans_formula.c Disallow CHR$() and UPS$() as numeric expressions. hp.c Fixup to widen password field when dumping ID tables; control-character passwords greater than length 3 weren't being completely displayed. hp.c Fixed REM/FILES detokenization (again!). tokenize.c Macro-ified common statement word stores. Fixed newly-broken MAT A=(n)*B, where '=' was being dropped. 20051011/taylors tokenize.c Remove obsolete eval_number() and demand_string() functions. Add block-style comments to statement-type code sections. Add comment header descriptions to functions. Fixed return-when-no-subscript for A0$-type variables in get_subscript(). ans_formula.c Fix for newly-broken LEN() processing. hp.c Fixed BASIC-FILE detokenization. Do not convert or extract ASCII-FILE device entries. Fixed display of non-shareable devices in directory output. All Tokenization and detokenization now appears to be 100% for all types of files, with the exception of 2000F->ACCESS converted programs. 20051011/taylors Version 1.34. 20051012/taylors hp.c Add display output identifying TSC system binaries in 2000F directory listings (@nnn account files). 20051016/taylors tokenize.c More syntax checking for ASSIGN, PRINT and CONVERT. 20051016/taylors Makefile Include tape.h in INCLUDES so the distribution picks it up. 20051017/taylors Version 1.35. 20070616/taylors Unspecified changes. Version 1.36. 20120730/taylors tsb.c Fix op 31 processing for 2000/F, C-200 and TSC (2000/F formats); PRINT # detokenization. Version 1.37. 20120816/taylors tsb.c Duplicate 2000E tape block processing for 2000A processing. 2000S differs somewhat from 2000E. I've gotten TSB 2000A booted, and created a minimal dump tape to use for accomodating the 2000A SLEEPs. 2000B will likely be handled by this code as well. hp.c Duplicate 2000E structures for 2000A. 20120817/taylors tsb.c Add time allowed/used and sectors allowed/used to IDT track data output for 2000A and 2000E. This completes all field definitions in that particular tape record. 20120818/taylors tsb.c Completed 2000A tape processing, at least for tapes with no user-stored files; I'm unable to run 2000A with a multiplexor under simh, so I can't create a SLEEP tape with user files to verify user data on disc. 20120818/taylors Version 1.38. 20120917/taylors tsb.c Rework fenv(3) as the default IEEE FP support, as newer util.c linux distributions no longer support ieeefp.h. Revert to using ieeefp.h by setting -DOLDIEEE in the Makefile CFLAGS. 20120917/taylors Version 1.39. 20120923/taylors ALL Rename local dprintf() to Dprintf. Clean up unused variables for gcc 4.6's stricter default warnings. Add stdint.h includes for Linux. Disable pointer/int casting warnings, as tsb redefines a few function return value usages. 20120923/taylors Version 1.39.1. ANOMALIES: 000 *SOME* Print statements break up '10'10'10"ABCD" into two strings, the first being the quoted ascii and the second being "ABCD". 000 Certain LSB precision errors in HP->IEEE->HP float conversions. Inconsequential, but annoying when comparing tokenizations. The resultant numbers are still computationally equivalent. DETOKENIZATION ISSUES (Not TOKENIZATION related): The following are the result of the de-tokenization process. Original tape tokens will have to be examined to determine what the issue is: o Original: 1080 PRINT " 700 PRINT ""I;N(1);N(3);N(5);N(7);N(9)" Tokenize: 1080 PRINT " 700 PRINT I;N(1);N(3);N(5);N(7);N(9)" Original: 270 PRINT "BLOCK OF ROUGHLY 68 CHARACTERS""MORE TEXT" Tokenize: 1080 PRINT "BLOCK OF ROUGHLY 68 CHARACTERSMORETEXT" Normally, TSB disallows this type of text-literal concatenation. However, due to 2000F conversions, tokenizations of this type can exist. This will be allowed in the future with the "-E 1" exception allowance. o Source has: 1413 PRINT '13'10371"DATA --TRAINING EXPENSE--" (DECSN, 36065 REV C) The DECSN file on PATTI says this as well, but the syntax is invalid. The statement executes correctly, however, outputting: 371 DATA --TRAINING EXPENSE-- 4200 PRINT "PLTL"'13'10;100;INT(B8);'13'10;5000;INT(B8);'13'109900;INT(B8) This issue cannot be resolved. It is the result of ^N/^O conversions by the CONVERT utility program when converting from 2000F/2000C' to ACCESS. It is impossible to determine, from the ascii representation, if this is '10'10;371 or '10'103;71. The detokenization of such conflicts will be converted to '10'10;371 with the "-E 2" exception allowance in the future. o 870 IMAGE #," = ",3D,"^M " This issue cannot be resolved. Is the result of ^N/^O conversions by the CONVERT utility program when converting from 2000F/2000C' to ACCESS. o Embedded NULLs in REM, FILES and IMAGE statements can't be de-tokenized. The above ANOMALIES were found by tokenizing 400,000 lines from PATTI's and CSC's tapes. Tokenization was successful with the exception of the above ANOMALIES. The tokenized programs will still need to be compared against their tokenized originals, which will identify bugs in both the tokenization and the detokenization processes. TODO: o Tokenization calling from tsb (-T) will need to create dummy headers for the stored tokenized program. Use 2000/ACCESS header info. o Implement '-E' (exception) command-line argument to acces a comma- separated numeric list indicating specific processing exceptions to accomodate. o Option to always de-tokenize and re-tokenize BASIC programs. This will result in CSAVE being uncompiled, and possible code hacks and questionable code (trojans, bad tokenizations) being cleaned. o Tokenization of data files to BASIC-format or ASCII-format TSB files.