Researchers test AI safety training that targets harmful requests without blocking entire topics

The approach was tested on political prompts. False refusals fell from 33% to 4%, while the model’s rejection of harmful requests declined from 92% to 88%.

Researchers at Multiverse Computing are studying how to make AI safety refusals more selective. Working with Qwen3-8B, they adjusted training data to help the model reject harmful requests while continuing to answer legitimate questions about the same subject.

The distinction depends on what a prompt asks the model to do. A factual question about an election and a request for targeted political manipulation concern the same subject, but require different responses under the policy the researchers tested. Training only for refusal can push the model to reject both.

The team used the model to generate its own refusal examples, retaining those verified by a separate safety classifier. It retried failed examples with stronger instructions instead of dropping them from the training set. To counter unnecessary refusals, the researchers also included safe prompts whose wording could look dangerous.

One configuration showed how far safety tuning could overshoot. It reduced the average unsafe-response rate across three broader safety tests from 26.26% to 0.14%. Yet the same configuration refused 74% of safe prompts in XSTest, up from 2% before training. Measuring harmful answers alone would have concealed that loss of usefulness.

For a separate evaluation of selectivity, the researchers used 1,539 pairs of prompts excluded from training. Each pair shared a topic but differed in intent: one called for refusal, the other for an answer. This tested whether the model could distinguish nearby permitted and prohibited requests.

In the paired evaluation, models trained with safe examples near the refusal boundary made fewer mistakes on legitimate requests than those trained without them. Some harmful requests also became more likely to receive an answer. This comparison isolates the effect of those training examples; it measures a different task from the broader XSTest check.

The authors argue that safety evaluations need to measure both unwarranted refusals and failures to refuse. Their results concern one model trained on political requests. They propose applying the method to other subjects, but have not demonstrated the same improvement there.