Skip to content
VIPS HandbookVIPS Handbook

Reader/Writer streaming I/O (govips v2.19.0)

Copy page

govips v2.19.0 can connect an io.Reader directly to libvips and write encoded chunks to an io.Writer. This avoids copying the complete compressed file into a Go byte slice with io.ReadAll, and gives HTTP bodies, object-storage streams, and pipes a common interface.

Responsibilities of the three APIs

Section titled “Responsibilities of the three APIs”
API Role Access strategy
LoadImageFromReader Create an ImageRef from any io.Reader Materialized by default; lazy with explicit AccessSequential
SaveToWriter / typed variants Write encoded chunks to an io.Writer Codec-dependent; formats such as TIFF may need full buffering
TranscodeStream Combine a Reader, optional auto-rotation, and a Writer Selects sequential or materialized loading from orientation metadata

These APIs are not present in govips v2.17.0. This page and its executable example are pinned to v2.19.0.

Executable sequential-stream example

Section titled “Executable sequential-stream example”

The reference implementation explicitly selects AccessSequential and keeps the input Reader open until ImageRef.Close(). All four bindings produce the same product result. The sharp, pyvips, and CLI versions are comparison implementations; they do not imply identical Reader/Writer APIs.

Run command

go run . input.jpg output/

Complete source

package main

import (
	"fmt"
	"io"
	"os"
	"path/filepath"

	"github.com/davidbyttow/govips/v2/vips"
	"github.com/webp-sh/vips-handbook/examples/internal/recipeutil"
)

type countingWriter struct {
	writer  io.Writer
	written int
}

func (w *countingWriter) Write(data []byte) (int, error) {
	n, err := w.writer.Write(data)
	w.written += n
	return n, err
}

func main() {
	if err := run(os.Args); err != nil {
		fmt.Fprintln(os.Stderr, err)
		os.Exit(1)
	}
}

func run(args []string) error {
	if len(args) != 3 {
		return fmt.Errorf("usage: stream-basic <input.jpg> <output-directory>")
	}
	inputPath, outputDir := args[1], args[2]
	if err := os.MkdirAll(outputDir, 0o755); err != nil {
		return fmt.Errorf("create output directory: %w", err)
	}

	vips.LoggingSettings(nil, vips.LogLevelError)
	if err := vips.Startup(nil); err != nil {
		return fmt.Errorf("start libvips: %w", err)
	}
	defer vips.Shutdown()

	input, err := os.Open(inputPath)
	if err != nil {
		return fmt.Errorf("open input: %w", err)
	}
	defer input.Close()

	outputPath := filepath.Join(outputDir, "streamed.webp")
	output, err := os.Create(outputPath)
	if err != nil {
		return fmt.Errorf("create output: %w", err)
	}

	counter := &countingWriter{writer: output}
	result, streamErr := streamWebp(input, counter)
	closeErr := output.Close()
	if streamErr != nil {
		return streamErr
	}
	if closeErr != nil {
		return fmt.Errorf("close output: %w", closeErr)
	}

	result.Path = filepath.Base(outputPath)
	result.Size = counter.written
	if err := recipeutil.WriteResult(outputDir, "stream-basic", inputPath, []recipeutil.Output{result}); err != nil {
		return err
	}

	fmt.Printf("stream-basic: %dx%d, %d bytes\n", result.Width, result.Height, result.Size)
	return nil
}

func streamWebp(input io.Reader, output io.Writer) (recipeutil.Output, error) {
	params := vips.NewImportParams()
	params.Access.Set(vips.AccessSequential)

	image, err := vips.LoadImageFromReader(input, params)
	if err != nil {
		return recipeutil.Output{}, fmt.Errorf("load sequential stream: %w", err)
	}
	defer image.Close()

	export := vips.NewWebpExportParams()
	export.Quality = 82
	export.StripMetadata = true
	if err := image.SaveToWriterWebp(output, export); err != nil {
		return recipeutil.Output{}, fmt.Errorf("save WebP stream: %w", err)
	}

	return recipeutil.Output{
		Parameters: map[string]any{
			"access":   "sequential",
			"input":    "io.Reader",
			"output":   "io.Writer",
			"pipeline": "LoadImageFromReader → SaveToWriterWebp",
			"quality":  82,
			"metadata": "strip",
		},
		Format:   "webp",
		Width:    image.Width(),
		Height:   image.Height(),
		Bands:    image.Bands(),
		HasAlpha: image.HasAlpha(),
	}, nil
}

Inspect the generated result

Section titled “Inspect the generated result”
stream-basic output
WEBP1938 × 1292, 398,898 bytes, 3 bands

Generated with govips v2.19.0 / libvips 8.17.3.

  • Format: WEBP
  • Dimensions: 1938 × 1292
  • Bands: 3
  • Alpha: No
  • File size: 398,898 bytes
  • Generated with: govips v2.19.0 / libvips 8.17.3

Automated tests cover propagation of Reader and Writer failures in the reference implementation. A sequential load normally consumes only enough bytes for the header. Truncation or a mid-stream failure can therefore surface later, when SaveToWriterWebp first consumes all pixels.

Sequential and materialized paths

Section titled “Sequential and materialized paths”

The sequential path works for shrinking, cropping, flattening, colourspace conversion, horizontal flips, and most line-kernel operations. Operations that read rows out of order—90°/180°/270° rotation, vertical flip, and smart crop, for example—must materialize first or fail with out of order read.

params := vips.NewImportParams()
params.Access.Set(vips.AccessSequential)


image, err := vips.LoadImageFromReader(input, params)
if err != nil {
    return fmt.Errorf("load stream: %w", err)
}
defer image.Close()


if err := image.SaveToWriterWebp(output, vips.NewWebpExportParams()); err != nil {
    return fmt.Errorf("save stream: %w", err)
}

Use TranscodeStream when the pipeline must choose materialization from EXIF orientation:

err := vips.TranscodeStream(input, output, &vips.TranscodeOptions{
    Format:     vips.ImageTypeWEBP,
    AutoRotate: true,
    ExportParams: &vips.ExportParams{
        Quality:      82,
        Interlaced:   false,
        StripMetadata: true,
    },
})
if err != nil {
    return fmt.Errorf("transcode stream: %w", err)
}

TranscodeStream keeps orientations 1 and 2 on the sequential path. Orientations that transpose or reverse row order are materialized first.

Untrusted input and error boundaries

Section titled “Untrusted input and error boundaries”
  • Limit the complete request body at the HTTP layer. SetPipeReadLimit only limits header buffering for non-seekable Readers; it is not an upload-size limit.
  • In sequential mode, keep the Reader alive until ImageRef.Close().
  • Original Reader and Writer errors are wrapped and remain detectable with errors.Is.
  • Progressive JPEG, interlaced PNG, HEIF decoding, and TIFF output can require additional buffering. Do not infer a memory ceiling from the API name alone.
  • For cancellation in an explicit Load/Save flow, another goroutine can call image.SetKill(true) and clear the flag after completion. TranscodeStream does not currently accept a context.Context.

Reproducible observation

Section titled “Reproducible observation”

When comparing sequential and materialized paths, hold the input, libvips/govips versions, concurrency, and encoding parameters constant. Record peak RSS, exit status, output dimensions, and file size. A single-machine observation describes that environment only; it is not a universal performance claim.

Next

Section titled “Next”

Sources

Section titled “Sources”