GHSA-73v3-wf97-vpgj: msgpack-ruby JRuby decoder makes uncapped header-count allocations; a 5-byte message forces multi-GiB OOM

  • Severity: HIGH
  • CVE: CVE-2026-126366
  • Ecosystems: rubygems
  • Packages: msgpack

### Summary

The **JRuby** implementation of msgpack (`msgpack` gem, `java` platform — pure Java decoder under `ext/java/`) sizes containers directly from the element count written in the message header, with **no cap at all**: no equivalent of the C unpacker’s `SHRT_MAX` clamp (since 1.7.3) and none of the buffered-bytes cap that commit c8d42d8 added for CRuby in 1.8.6. A single 5-byte message decides how much memory a JVM process tries to allocate:

* `array32` header declaring `0x7FFFFFFF` elements → `new IRubyObject[0x7FFFFFFF]` (~8 GiB of reference slots) attempted immediately → `java.lang.OutOfMemoryError: Requested array size exceeds VM limit`.
* Counts *below* the VM’s max-array-size but above available heap (e.g. 5×10⁸ with a 512 MiB heap) produce a **real allocation attempt** — `OutOfMemoryError: Java heap space` — the JVM genuinely tries to take the memory before failing.
* `str32`/`bin32` headers declaring `0x7FFFFFFF` bytes do the same via `readBytes(size)` → `new byte[size]`.
* Below the failure threshold there is a **silent band**: one streaming unpacker fed only an `array32` header with count 2×10⁷ (5 wire bytes, zero elements) grows used heap **+78 MiB immediately**, no exception (`each` swallows the underflow), and the footprint stays elevated after `Unpacker#reset` + full GC. Linear in declared count (~4 B/element); nothing bounds it — not delivered bytes, not a per-container cap.

Only counts ≥ 2³¹ are refused (a clean `MessagePack::UnderflowError` from `g…

Source: GitHub Security Advisories · 2026-11-12