to_entropy() cannot read the japanese mnemonics to_mnemonic() writes - #145
Open
fametrano wants to merge 1 commit into
Open
to_entropy() cannot read the japanese mnemonics to_mnemonic() writes#145fametrano wants to merge 1 commit into
fametrano wants to merge 1 commit into
Conversation
This was referenced Aug 5, 2026
to_mnemonic joins a japanese mnemonic with U+3000 while to_entropy, check and expand split on " ", so the library cannot read the sentences it writes: to_entropy raises on all 24 japanese vectors of vectors.json and expand returns the sentence unchanged. The separator is read after NFKD, which maps U+3000 to U+0020, so it cannot be self.delimiter -- that was trezor#110, reverted in df3e150 for failing CI. What is left to split on is any run of whitespace, which is what detect_language already reads. to_seed follows the same rule, so that a sentence check refuses no longer stretches into a different seed in silence. The tests fail without the fix: to_entropy raises on every japanese vector, check refuses a sentence separated by anything but one space, and expand hands back a tab- or U+3000-separated sentence unexpanded. The existing round trip hides the first by splitting the sentence itself before handing it over. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
fametrano
force-pushed
the
japanese_delimiter_read_back
branch
from
August 13, 2026 07:59
2cb21cc to
bd50ae2
Compare
Author
|
@prusnak could you approve the workflow run here? This is my first pull request from this fork, so CI has never started, and the local run in the description is the only evidence in the thread. The branch is a single commit now. Since filing I added one test for |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Mnemonic("japanese")joins words with U+3000, the ideographic space(
#L73,#L210),as BIP39's Japanese vectors do.
to_entropyandexpandsplit on" ", sothey see a single word: the first raises, the second returns the sentence
unchanged.
checkaccepts it, because it normalizes first and NFKD mapsU+3000 to U+0020.
The README's own steps are the reproduction. On master, python 3.12:
That block is a doctest, and passes as written on master; with this patch
applied it fails, because
to_entropythen returns the entropy. Onvectors.json,checkaccepts all 288 mnemonics whileto_entropyfailson all 24 Japanese ones and none of the other 264.
#110 fixed this by splitting on
self.delimiter; it was merged as71cf5203and reverted asdf3e1500for failing CI. That direction cannotwork in
check, which normalizes before splitting: after NFKD there is noU+3000 left to split on, so every Japanese mnemonic fails validation
instead. Hence any run of whitespace, read after normalization — which is
what
detect_languagealready does.The fourth changed line applies the same rule to
to_seed, whose NFKD-onlyreading silently stretches a different seed out of the input
checkrejects: a leading space, a trailing newline, a tab. If that is unwanted,
drop it along with the last assertion of
test_whitespace_runs; the otherthree still fix the Japanese bug.
Compatibility: all 288 vectors are unchanged —
test_vectorsassertsmnemonic, seed and xprv for each, and passes. No input
checkaccepts todayderives a different seed; what changes is input it rejects, which now
derives the canonical one. The passphrase is untouched, its whitespace being
part of the secret.
CI, since #110 was reverted for failing it:
python tests/test_mnemonic.pyis
OKon python 3.8 through 3.14 and on pypy 3.11, andblack --check,isort --check-only,flake8andpyrightreport nothing onsrc tests tools.Written with machine assistance and reviewed against master
b57a5adbefore filing. Found while checking btclib's BIP39 reading against this
implementation: btclib-org/btclib#258.
🤖 Generated with Claude Code