An In-Depth Look at Simple Streams

1. An In-Depth Look at Simple Streams By Duane Rettig October, 2002

2. What is a Lisp Stream? Lisp stream stream internal device externaldevice

3. An In-Depth Look at Simple Streams • Simple Streams Design Goals • What is a Simple Stream? • Simple-stream Concepts • Opening, closing, blocking, reading, writing • Character Strategies • Common Windows Rewrite • Further changes in next version (references distributed throughout presentation)

4. Design Goals • CL compliance • Bivalence • Thin Strategy layer that is uniform • Extensibility via specializations/mixins and encapsulations • Minimal decision points in critical strategy code • Minimal calls to generic-functions • External-format ready for 8 and 16 bit lisps

5. Gray vs. Simple Stream API call Strategy Object functionality Device interface Gray Stream buffer Simple Stream

6. Dual-channel stream • Two octet buffers • External-format character translation API Ext. format Out buffer In buffer Device methods External device

7. Single-channel stream • One octet buffer • External-format character translation API Ext. format direction buffer Device methods Internal or External device

8. String stream • Zero, one, or two string buffers • No external-format translations • Plugable character/buffer strategies API Out buffer In buffer Device methods Internal device

9. Opening a Stream • Creates or reuses a stream object • Makes or retains a connection to a device • Initializes character strategies • Marks the stream as open

10. Open open make-instance (gf) ... shared-initialize (gf) shared-initialize :after [Method] device-open [Method]

11. (Open): Simple encapsulation (setq bun (open "sesame")) (setq big-mac (make-instance 'all-beef :input-handle bun ...)) big-mac bun all-beef sesame

12. (Open): shared-initialize (defmethod shared-initialize :after ((stream simple-stream) slots &rest initargs) (declare (ignore slots)) (unless (device-open stream initargs) (device-close stream t)) stream)

13. (Open): device-open • At device-open time: • All instance slots have been initialized • connection has been made or will be made by device-open • When device-open returns: • If non-nil, the stream is ready for appropriate actions • If nil, then the open has failed.

14. (Open): device-open • device-open must: • Ensure a connection before returning • Ensure buffers are in place • Initialize/reinitialize pointers • Return nil for failure, or non-nil after success • device-open must not: • Assume that the stream was closed • close the stream

15. (Open): subclass example (defclass file-with-header (file-simple-stream) ((header :initform nil :accessor file-header-info))) (defmethod device-open ((stream file-with-header) options) (declare (ignore options)) (let ((success (call-next-method))) (when success (setf (file-header-info stream) (read-file-header stream)) t)))

16. (Open): device-open:before methods • In future releases, for string-simple-streams: • A catch-all primary method returns true for an open stream. • Strategy installation functions are directional and do not override. • install-string-character-strategy deprecated, replaced by install-string-{input,output}-character-strategy • All :before methods fire according to CPL, and may shadow default actions.

17. (Open): device-open: string-input (future) (defmethod device-open :before ((stream string-input-simple-stream) options) (with-stream-class (string-input-simple-stream stream) (let ((string (getf options :string))) (when (and string (null (sm buffer stream))) (let ((start (getf options :start)) (end (or (getf options :end) (length string)))) (setf (sm buffer stream) string (sm buffpos stream) start (sm buffer-ptr stream) end))))) (install-string-input-character-strategy stream) (add-stream-instance-flags stream :string :input :simple))

18. (Open): device-open: string-output (future) (defmethod device-open :before ((stream string-output-simple-stream) options) (with-stream-class (string-output-simple-stream stream) (unless (sm out-buffer stream) (let ((string (getf options :string))) (if string (setf (sm out-buffer stream) string (sm max-out-pos stream) (length string)) (let ((buflen (max (device-buffer-length stream) 16))) (setf (sm out-buffer stream) (make-string buflen) (sm max-out-pos stream) buflen))))) (unless (sm control-out stream) (setf (sm control-out stream) *std-control-out-table*))) (install-string-output-character-strategy stream) (add-stream-instance-flags stream :string :output :simple))

19. Closing a Stream • Flushes output if any. • Breaks connection to device. • Secures against accidental future operations. • Does not change-class. (defmethod close ((stream simple-stream) &key abort) (device-close stream abort))

20. Close (defmethod device-close :around ((stream simple-stream) abort) (let (res) (when (pseudo::open-stream-p stream) (unwind-protect (progn (when (output-stream-p stream) (ignore-errors (if abort (clear-output stream) (force-output stream)))) (setq res (call-next-method))) (without-interrupts (pseudo::unset-open-flags stream) (setf (stream-input-handle stream) nil (stream-output-handle stream) nil)) (setf (stream-external-format stream) (find-external-format :void)) res))))

21. (Close): device-close • device-close should: • flush all data (unless aborting) • disconnect handles of any encapsulated streams • call lower-levels to close as necessary • device-close should not: • operate on a closed stream • close an encapsulated stream

22. Blocking • Issues with listen • read direction only • covers character availability and not blocking • assumes character as the basic data unit • stream-listen (carried over from Gray streams) • read-no-hang-p and write-no-hang-p

23. Blocking styles • Non-blocking • Blocking • B/NB (blocking, then non-blocking) Character: One element Octet: One element

24. Reading and Writing • Mostly symmetrical • Device methods obey B/NB discipline

25. Basic read strategy (block read (when (>= buffpos buffer-ptr) (let ((res (device-read stream nil 0 buffpos blocking))) (when (< res 0) (pseudo::do-error-handling)) (when (= res 0) (if blocking (pseudo::do-eof-handling) (return-from read nil))) (setq buffer-ptr res buffpos 0))) (prog1 (aref buffer buffpos) (incf buffpos)))))

26. Basic write strategy (block write (when (>= out-pos max-out-pos) (when (> out-pos 0) (erroring-device-write-whole stream nil 0 out-pos t)) (setq out-pos 0)) (setf (aref buffer out-pos) value) (incf out-pos))

27. Implementing force-output, finish-output • Both are implemented using device-write • blocking argument determines whether force (nil) or finish (t) • device-writebuffer argument is :flush (future)

28. Reading and Writing Sequences • read-sequence and write-sequence • are width sensitive and by-element • require blocking semantics • do not return a count • read-vector and write-vector • are octet-based • employ B/NB semantics • return a count, 0, or error code

29. Deprecated generic function: device-extend • History and relationship to device-read and device-write • structural differences • purity of reference • Problems • one generic-function for two directions • inconsistent interface

30. Replacement of device-extend • Actions :input and :input-check become blocking argument to device-read • Actions :output and :output-check become blocking argument to device-write • Extra actions come through via device-read and device-writebuffer argument

31. Record orientation • Via device-finish-record gf • dual-channel input • string output

32. Other generic-functions • device-buffer-length • device-clear-input • device-clear-output • device-file-length • device-file-position

33. Strategies • What is a strategy’s purpose? • To satisfy the high-level requirements of a specified behavior in a uniform manner. • Octet strategies are non-programmable. • read-byte, write-byte are cast in concrete. • Character strategies are programmable...

34. Character strategies • Programmable, replaceable. • External-formats use template functions for dual-channel and single-channel streams. • String-streams use simple strategy functions. • (future): Subclassed strategy functions can shadow more general ones. • Occupy “joint” slots in the stream.

35. Example strategy set for *terminal-io* cl-user(5): :i terminal-simple-stream @ #x711e057a = #<terminal-simple-stream [initial terminal io] fd 0/1 @ #x711e057a> 0 Class --------> #<standard-class terminal-simple-stream> 1 j-unread-char -> #<Function dual-channel-unread-char> 2 j-write-chars -> #<Function (:efft dc-write-chars :latin1-base)> 3 j-write-char -> #<Function (:efft dc-write-char :latin1-base)> 4 j-read-chars -> #<Function (:efft dc-read-chars :latin1-base)> 5 j-read-char --> #<Function (:efft dc-read-char :latin1-base)> 6 j-listen -----> #<Function (:efft dc-listen :latin1-base)> ... 34 src-position-table -> The symbol nil cl-user(6):

36. Example strategy set for string-output cl-user(9): :i string-output-simple-stream @ #x719cb5e2 = #<string-output-simple-stream "" pos 0 @ #x719cb5e2> 0 Class --------> #<standard-class string-output-simple-stream> 1 j-unread-char -> The symbol nil 2 j-write-chars -> #<Function string-output-write-chars> 3 j-write-char -> #<Function string-output-write-char> 4 j-read-chars -> The symbol nil 5 j-read-char --> The symbol nil 6 j-listen -----> The symbol nil ... 33 out-buffer ---> A simple-string (4096) that starts "xxxxxxxxxxxxxxxx” cl-user(10):

37. Example strategy set for string-input cl-user(14): :i string-input-simple-stream @ #x719e74aa = #<string-input-simple-stream "abc" pos 0 @ #x719e74aa> 0 Class --------> #<standard-class string-input-simple-stream> 1 j-unread-char -> #<Function string-input-unread-char> 2 j-write-chars -> The symbol nil 3 j-write-char -> The symbol nil 4 j-read-chars -> #<Function string-input-read-chars> 5 j-read-char --> #<Function string-input-read-char> 6 j-listen -----> #<Function string-listen> ... 31 src-position-table -> The symbol nil cl-user(15):

38. j-read-char • Args: stream eof-error-p eof-value blocking • Implements read-char and read-char-no-hang functionality directly (after argument resolution) • blocking argument is nil/true dichotomy

39. j-write-char • Args: character stream • Implements write-char directly (after argument resolution) • Writing nil as the character argument will flush an external-format’s output state.

40. j-read-chars • Args: stream string search start end blocking • Implements read-sequence, read-line • blocking argument is nil, :bnb, or true

41. j-write-chars • Args: stream string start end • Implements write-sequence, write-string • Always blocks (for now)

42. j-listen • Args: stream • Implements stream-listen directly • Must have a complete character or an error condition in order to return true.

43. j-unread-char • Args: stream relaxed • May need to unread multiple characters as part of a composing or encapsulating character. • relaxed allows non-error unreads beyond the first unread.

44. Taking strategies beyond ANS • Other strategies can be created for operations that don’t fit into CL standard functionalities. • Example: one-buffer ring fifo queue

45. Common Windows low-level rewrite • Source code comparison between Gray and simple streams • Demo