Skip to content

feat(text chat): expose image input via repeatable --image (multimodal M3, multi-image) - #225

Open
shoemoney wants to merge 2 commits into
MiniMax-AI:mainfrom
shoemoney:feat/text-chat-image-flag
Open

feat(text chat): expose image input via repeatable --image (multimodal M3, multi-image)#225
shoemoney wants to merge 2 commits into
MiniMax-AI:mainfrom
shoemoney:feat/text-chat-image-flag

Conversation

@shoemoney

@shoemoney shoemoney commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

Closes #224.

What

mmx text chat now takes a repeatable --image <path-or-url>:

# single image
mmx text chat --image ./photo.jpg --message "What breed is this dog?"

# multiple images in one call
mmx text chat --image ./before.png --image ./after.png \
  --message "List every visual difference between these two."

Why

M3 is multimodal and already accepts multiple images through --messages-file, but there was no CLI path to it — you had to hand-write a base64 messages JSON. Worse, the documented OpenAI shape ({"type":"image_url", ...}) is rejected, because text chat posts to the Anthropic-compatible /anthropic/v1/messages endpoint and needs {"type":"image","source":{"type":"base64",...}}. --image emits the right shape for you.

How

  • toImageBlock() in src/utils/image.ts wraps the existing toDataUri() (local paths, http(s) URLs, and pre-made data URIs all work) and returns an Anthropic image block.
  • src/commands/text/chat.ts appends the blocks to the last user message, promoting content from a string to a block array. Text goes first so the model reads the instruction before the pixels. With no --message, the images become the user message.
  • ContentBlock gains the image variant.
  • When --image is present and --model is not, the model resolves to MiniMax-M3 — otherwise a text-only defaultTextModel in config would silently break every image request. An explicit --model still wins.
  • Docs updated: README.md, README_CN.md, and skill/SKILL.md (which also now calls out the OpenAI-vs-Anthropic block-shape gotcha).

Verification

Dry run of the built binary:

$ mmx text chat --image a.png --image b.png --message "diff these" --dry-run --output json
{
  "request": {
    "model": "MiniMax-M3",
    "messages": [ { "role": "user", "content": [
      { "type": "text",  "text": "diff these" },
      { "type": "image", "source": { "type": "base64", "media_type": "image/png", "data": "..." } },
      { "type": "image", "source": { "type": "base64", "media_type": "image/png", "data": "..." } }
    ] } ],
    "max_tokens": 4096,
    "stream": false
  }
}

That is byte-for-byte the payload shape I confirmed M3 answers correctly in #224.

Six new tests in test/commands/text/chat.test.ts cover block shape, multi-image, image-without-message, the model override, explicit --model winning, and the missing-file error. bun test 455 pass / 0 fail; bun run typecheck and bun run lint clean.


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.

MiniMax-M3 is multimodal, but `text chat` had no way to send an image —
users had to hand-write a base64 messages JSON file, and the obvious
OpenAI `image_url` shape is rejected because the CLI posts to the
Anthropic-compatible /messages endpoint.

- `--image <path-or-url>` on `text chat`, repeatable, so multi-image
  compare/diff works in one call
- new `toImageBlock()` in utils/image reuses `toDataUri()` (local paths,
  http(s) URLs, existing data URIs) and emits the Anthropic block shape
  `{ type: 'image', source: { type: 'base64', media_type, data } }`
- images append to the last user message, promoting string content to a
  block array; with no `--message` they become the user message
- images force `MiniMax-M3` when `--model` is unset, so a text-only
  `defaultTextModel` in config can't silently break the request
- docs: README, README_CN, skill/SKILL.md (incl. the OpenAI-vs-Anthropic
  block-shape gotcha)

Closes MiniMax-AI#224

@NianJiuZst NianJiuZst left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I reviewed the current head and the feature direction is useful, but I do not recommend merging this revision yet.

[P2] Enforce the M3 image contract before encoding. The current MiniMax Anthropic API docs cap each image at 10 MB and the whole request body at 64 MB, and support JPEG/PNG/GIF/WEBP. The new path delegates to toDataUri(), which leaves local/data-URI inputs unbounded, permits remote images up to 50 MB, rejects local GIF, and accepts HEIC/HEIF. A local 11 MiB .png was encoded to 15,379,116 base64 characters without error; repeated --image inputs can therefore exceed the request cap and fail only after substantial memory/network work. Please add chat-specific per-image/format validation and an aggregate request-size check.

Official contract: https://platform.minimax.io/docs/api-reference/text-anthropic-api

Local verification: typecheck, lint (one pre-existing warning), build, 17 focused tests, and the full suite (455/455) passed. Those mocks do not exercise the oversized/unsupported real-provider cases.

Per review on MiniMax-AI#225: the chat --image path delegated to toDataUri(), which
left local and data: URI inputs unbounded, allowed remote images up to
50 MB, rejected local GIF, and accepted HEIC/HEIF — none of which matches
the Anthropic-API contract (10 MB per image, 64 MB request body,
JPEG/PNG/GIF/WEBP).

toDataUri/toImageBlock now take optional per-caller constraints. Omitted,
every existing caller behaves byte-identically; chat passes the M3 limits:

- local files are stat'd before they are read
- remote responses are checked by content-type and content-length before
  the body is buffered, then re-checked against the real byte length
- data: URIs have their declared type and decoded size validated without
  a full base64 decode
- the running request-body total is accumulated across --image inputs so
  a run bails as soon as the 64 MB cap is unreachable

6 focused tests; suite 461/461 ✅
@shoemoney

Copy link
Copy Markdown
Contributor Author

Thanks — the 11 MiB repro was the useful part, that path was genuinely unbounded. Pushed a fix in 91889a1.

toDataUri/toImageBlock now take optional per-caller constraints. Omitted, every existing caller (vision/describe, video/generate, image/generate, the SDK) runs byte-identically to before — same 50 MB remote cap, same HEIC/HEIF allowlist. IMAGE_MIME_TYPES is untouched. Chat passes the M3 limits explicitly:

  • Local files are statSync'd before readFileSync, so an 11 MiB PNG is rejected without ever being read, let alone base64-encoded.
  • Remote URLs have their content-type checked against the allowlist and their content-length checked before res.arrayBuffer(), then the real byteLength re-checked after — content-length can be absent or lie.
  • data: URIs have their declared media type validated and their decoded size computed arithmetically from the base64 length, without decoding.
  • Formats for chat are exactly image/jpeg, image/png, image/gif, image/webp. GIF is now accepted; HEIC/HEIF are now rejected.
  • The 64 MB aggregate is accumulated as each --image is processed, starting from the serialized message and system text, so a run bails as soon as the cap is unreachable rather than after fetching and encoding everything. The running total counts base64 length, since that is what actually goes on the wire.

Six focused tests cover each of those: oversized local, HEIC rejected, GIF accepted and labeled image/gif, remote over-cap, data: URI over-cap, and seven images each under 10 MB that trip the 64 MB aggregate. Full suite is 461/461, typecheck clean, lint unchanged at the one pre-existing warning.

One thing I could not black-box in a test: that the content-length check strictly fires before buffering. Bun's Response recomputes Content-Length to match the real body, so a spoofed header collapses to the true size and both checks agree. The code does the header check first as you asked; the test only asserts the rejection.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

text chat: expose image input (enable multimodal M3, incl. multi-image)

2 participants