FEAT: Add multilingual scenario - #2392
Conversation
| self._build_atomic_attack( | ||
| context=context, | ||
| seed_groups=seed_groups, | ||
| converter=RandomTranslationConverter( |
There was a problem hiding this comment.
Rich agrees with this comment but it is copilot generated:
The selected language pool changes random-translation behavior, but it is not included in the converter or technique evaluation identity. Runs with different pools can therefore be treated as behaviorally equivalent in cached results and technique statistics. Include a canonicalized language list in the random translation converter identifier and test that different pools produce different evaluation hashes.
| Returns: | ||
| AtomicAttack: The configured attack and its resolved seed groups. | ||
| """ | ||
| converter_config = AttackConverterConfig( |
There was a problem hiding this comment.
Model multilingual testing as a scenario axis rather than defining translation as a technique.
The matrix would be registered compatible techniques × datasets × languages. The languages or num_languages parameter would define the language axis. The scenario would append the corresponding translation converter after each technique’s built-in converters. I suggest omitting user-supplied converters to keep the behavior simple.
This has one prerequisite: update AttackTechniqueFactory so the scenario can include only techniques whose built-in converter chain ends in text.
Keep the default small, such as prompt_sending × 5 objectives × 5 languages, plus one untranslated baseline. All other compatible techniques would remain available through explicit selection.
3 participants
Description
A new AIRT scenario that tests the resilience of a target model against non-English attacks.
Two attack techniques:
TranslationConverter.RandomTranslationConverter.The number of languages randomly selected from the most spoken languages (
languages_most_spoken.yaml) is configurable, or callers can pass their own language list.Tests and Documentation