membrane_mp4_plugin has an unauthenticated denial-of-service via BEAM atom table exhaustion

Summary

membrane_mp4_plugin interns every 4-byte MP4 box name as a BEAM atom while parsing container headers, with no validation against an allow-list. Any code path that calls Membrane.MP4.Container.parse/1 (or the bang variant) on attacker-controlled MP4 bytes can be made to exhaust the BEAM atom table, which is uncapped at the source but bounded by the runtime ceiling (around 1,048,576 atoms) and never garbage-collected. Once the ceiling is hit, the entire BEAM node aborts.

Details

The MP4 container parser walks the input stream box-by-box. For each box, Membrane.MP4.Container.Header.parse/1 in lib/membrane_mp4/container/header.ex extracts the 4-byte name field and delegates to the private helper parse_box_name/1, which trims trailing spaces and passes the bytes through String.to_atom/1. Because String.to_atom/1 creates a new atom unconditionally, every distinct attacker-chosen 4-byte sequence becomes a permanent allocation in the global atom table. Atoms are not subject to garbage collection, so unique names accumulate for the lifetime of the node.

The fix replaces the unsafe interning with String.to_existing_atom/1 and treats unknown names as the :unknown atom downstream (with related plumbing in parse_helper.ex, serialize_helper.ex, and the ISOM/CMAF demuxer engines so the new error variant flows through cleanly).

PoC

1. Generate an MP4-shaped payload of roughly 1.1 million minimal box headers, each 8 bytes long (4-byte size of 8, followed by a unique 4-byte ASCII name produced by enumerating combinations in the printable range).
2. Concatenate them into a single binary (~8 MB total).
3. Call Membrane.MP4.Container.parse!/1 on the binary inside any process running under the target BEAM node.
4. The node aborts once the atom table ceiling is reached.

Impact

An attacker who can get a single crafted MP4 file in front of any system that demuxes user-supplied media with membrane_mp4_plugin can crash the entire BEAM node hosting that pipeline, taking down every Erlang/Elixir application sharing the runtime. No authentication, network position, or user interaction is required beyond delivering the file to whatever processing path eventually calls the parser.