Diagnosing out of order read

out of order read does not mean that pixels are stored in the wrong order in the file. It means that the current pipeline requested pixels in an order that the loader’s access mode cannot supply. The usual cause is a sequential input followed by an operation that reads backwards, revisits pixels, or consumes the same pipeline twice.

Symptoms

A vertical flip, 90°/180°/270° rotation, or Smart Crop reports out of order read during export.
The first export succeeds, but a second export from the same sequential image fails.
The transform call returns no error; Export* or SaveToWriter* fails later.
Switching to the default loading path removes the error but increases memory or temporary-disc use.

Root cause: a sequential image can be consumed once

libvips is demand-driven. Transform calls normally build a pipeline, while a downstream sink such as an encoder triggers pixel evaluation. A transform that creates an incompatible access pattern can therefore appear to succeed and fail only when export begins.

VIPS_ACCESS_SEQUENTIAL promises roughly top-to-bottom access with a small buffer behind the read point. It works for one-pass pipelines, but not for these patterns:

Pipeline behaviour	Why it needs random access
Vertical flip	The first output row needs the last input row
90°/270° rotation	Output scanlines correspond to input columns rather than consecutive rows
180° rotation	Rows and pixels must be read in reverse order
Smart Crop / content analysis	Several regions must be inspected before choosing the output window
Exporting the same sequential image twice	The first export has already consumed the input stream

Minimal reproduction

This program explicitly requests AccessSequential and then applies a vertical flip. With govips v2.19.0, Flip builds the pipeline and ExportJpeg returns VipsJpeg: out of order read when it first consumes the pixels.

package main

import (
  "fmt"
  "log"
  "os"

  "github.com/davidbyttow/govips/v2/vips"
)

func main() {
  if len(os.Args) != 2 {
    log.Fatalf("usage: %s INPUT", os.Args[0])
  }
  if err := vips.Startup(nil); err != nil {
    log.Fatal(err)
  }
  defer vips.Shutdown()

  input, err := os.Open(os.Args[1])
  if err != nil {
    log.Fatal(err)
  }
  defer input.Close()

  params := vips.NewImportParams()
  params.Access.Set(vips.AccessSequential)
  image, err := vips.LoadImageFromReader(input, params)
  if err != nil {
    log.Fatal(err)
  }
  defer image.Close()

  if err := image.Flip(vips.DirectionVertical); err != nil {
    log.Fatal(err)
  }
  if _, _, err := image.ExportJpeg(vips.NewJpegExportParams()); err != nil {
    fmt.Printf("expected failure: %v\n", err)
    return
  }
  log.Fatal("expected an out-of-order error")
}

Fixes

1. Use the default materialized path for random-access operations

Do not set AccessSequential when the pipeline contains a vertical flip, rotation, Smart Crop, or multiple exports. In govips v2.19.0, LoadImageFromReader(input, nil) materializes the image before returning: decoded images at or below the threshold go to memory, and larger images use a scratch file. The Reader can be closed after the function returns.

package main

import (
  "log"
  "os"

  "github.com/davidbyttow/govips/v2/vips"
)

func main() {
  if len(os.Args) != 3 {
    log.Fatalf("usage: %s INPUT OUTPUT", os.Args[0])
  }
  if err := vips.Startup(nil); err != nil {
    log.Fatal(err)
  }
  defer vips.Shutdown()

  input, err := os.Open(os.Args[1])
  if err != nil {
    log.Fatal(err)
  }
  image, err := vips.LoadImageFromReader(input, nil)
  closeErr := input.Close()
  if err != nil {
    log.Fatal(err)
  }
  if closeErr != nil {
    image.Close()
    log.Fatal(closeErr)
  }
  defer image.Close()

  if err := image.Flip(vips.DirectionVertical); err != nil {
    log.Fatal(err)
  }
  buffer, _, err := image.ExportJpeg(vips.NewJpegExportParams())
  if err != nil {
    log.Fatal(err)
  }
  if err := os.WriteFile(os.Args[2], buffer, 0o644); err != nil {
    log.Fatal(err)
  }
}

Materialization is an explicit resource trade-off: it enables random access but can use more memory or temporary disc space. SetStreamDiscThreshold and SetStreamScratchDir control govips’ threshold and scratch location.

2. Keep only sequential-safe operations on the fast path

Keep AccessSequential for a single format conversion, downscale, colourspace conversion, or another genuinely top-to-bottom pipeline. The Reader must remain available until ImageRef.Close(), and the sequential image must not branch into multiple consumers.

3. Let TranscodeStream decide for EXIF auto-rotation

For “read → optional auto-rotate → write,” govips v2.19.0 provides TranscodeStream. Orientations 1 and 2 remain on the sequential path; orientations that transpose or reverse row order are materialized first. More complex custom transforms still require the caller to choose the access strategy before loading.

Verify the fix

Repeat the failing path with the same input and export options, and confirm that out of order read is gone.
Check output dimensions, orientation, and pixels; “no error” is not sufficient acceptance evidence.
Record peak RSS, scratch-disc use, and the failure stage. Materialization can trade an access error for resource pressure.
For multiple outputs, confirm that they read one materialized image or independently reload the input rather than consuming the original sequential stream again.

Affected versions and scope

The failure was reproduced with govips v2.19.0 and libvips 8.17.3; the default materialized path produced a valid 1938×1292 JPEG.
The exact error prefix may name JPEG, PNG, or another loader. Do not match only VipsJpeg.
sharp, pyvips, and the vips CLI expose access strategy differently. This page does not claim that they share govips’ defaults.