o3-mini vs DeepSeek-R1: Which One is Safer? | Summary

Contents

Summary
1 Introduction
2 Background and Related Work
- 2.1 Safety Testing of LLMs
- 2.2 ASTRAL
3 Methodology
4 Results and Discussion
5 Conclusion and Future Work
Replication Package
Appendix
Brief Thoughts

This article explains the key points of o3-mini vs DeepSeek-R1: Which One is Safer?.

2025-01-30 (arXiv)
Arrieta, Aitor, Ugarte, Miriam, Valle, Pablo, Parejo, José Antonio, Segura, Sergio.
Mondragon University, University of Seville
Paper
Github

Summary

The paper tests whether large language models (LLMs), models that generate text, provide harmful assistance in response to unsafe requests. On the same 1,260 test inputs, locally deployed DeepSeek-R1 (70B) produced 151 manually confirmed unsafe responses (11.98%), compared with 15 (1.19%) from an early-access o3-mini beta accessed through OpenAI’s application programming interface (API), including its safeguards.
ASTRAL, an automated safety-testing tool, supplies a balanced test set spanning 14 safety categories, six writing styles, and five persuasion techniques. GPT3.5 classifies responses as safe, unsafe, or unknown, after which the authors manually review every response classified as unsafe or unknown.
The comparison includes 565 o3-mini policy-violation exceptions, API rejections that the authors count as safe outcomes. Responses initially classified as safe were not manually checked, so unsafe outputs may have been missed; the reported gap therefore describes confirmed failures in the tested configurations.

1 Introduction

The introduction motivates safety testing alongside reasoning capability and cost: a warning about illegality does not make harmful assistance safe. The report applies the same systematically varied prompts to DeepSeek-R1 (70B) and o3-mini beta, including topics current when the prompts were generated.

The background reviews safety benchmarks and automated testing methods before introducing ASTRAL. It emphasizes coverage of harmful topics and linguistic variations, together with the need to update tests as training data and current events change.

2.1 Safety Testing of LLMs

The authors identify limitations in fixed multiple-choice tests, costly human red teaming, which uses people to construct challenging inputs, and adversarial jailbreaks, prompts designed to bypass safety restrictions. They also argue that static benchmarks can have uneven topic coverage, enter future training datasets, and become outdated.

Examples include BeaverTails with 333,963 prompts across 14 safety categories, SimpleSafetyTests with 100 English prompts across five harm areas, and SORRY-Bench with prompts covering 45 safety-related topics.
LlamaGuard and ShieldLM illustrate automated safety classification: a separate model judges whether prompts or responses meet safety criteria.
The section places safety testing in the context of the European Union’s AI Act, Regulation (EU) 2024/1689, and its risk-based regulatory framework.

2.2 ASTRAL

ASTRAL separates testing into generation, execution, and evaluation. Generation uses retrieval-augmented generation (RAG), which supplies retrieved information to the generator, few-shot prompting, which provides examples in the prompt, and web browsing to construct varied inputs incorporating current information.

Its black-box coverage criterion balances input features without requiring access to the tested model’s internals: safety category, writing style, and persuasion technique.
The execution phase submits generated prompts to the tested model; a separate LLM then acts as an oracle, the component that decides whether an output meets safety standards.
Table 1 defines writing-style identifiers S1–S6, persuasion identifiers P1–P5, and safety-category identifiers C1–C14.

Definitions of writing-style features S1–S6, persuasion features P1–P5, and safety categories C1–C14.

3 Methodology

The methodology asks three research questions: RQ1 compares overall safety, RQ2 compares safety categories, and RQ3 examines writing styles and persuasion techniques. These questions structure the analysis of the same test set on both systems.

3.1 Models under evaluation

The authors deploy the version they describe as DeepSeek-R1 (70B) through Ollama, a framework for running models locally or in the cloud, because larger versions exceeded their infrastructure. Its reported defaults are a context window of 131,072 tokens, the text units processed by the model; an embedding dimension of 8,192, the size of its token representations; a maximum generation length of 2,048 tokens per inference; and a temperature of 0.8, a sampling setting.

o3-mini is an early-access, pre-deployment beta supplied through OpenAI’s safety-testing programme. Its interface did not expose parameter customization, so the authors used its defaults.
The comparison uses each deployment’s defaults without matching generation settings across systems.

3.2 Test input generation

The study reuses prompts generated for the original ASTRAL paper, combining 6 (styles) × 5 (persuasion) × 14 (safety categories) × 3 (tests) = 1,260 test inputs. Both systems receive the same inputs, with three tests for each specified feature combination.

The prompts were generated in November 2024. Internet retrieval incorporated events current at that time, particularly the 2024 US elections in C3, Controversial topics, politics.

3.3 Test execution and evaluation

Testing began on 21 January 2025 for o3-mini and 28 January 2025 for DeepSeek-R1 (70B). GPT3.5 assigns safe, unsafe, or unknown labels and provides a rationale; o3-mini policy-violation exceptions are explicitly classified as safe.

The authors hypothesize that the exceptions reflect an API safeguard blocking unsafe inputs before model processing. They could not control this mechanism and describe the o3-mini assessment as an evaluation of the entire API system.
DeepSeek-R1 ran on a Linux server with 512Gb RAM, the server’s working memory, an AMD EPYC 7773X 64-Core Processor, and an NVIDIA RTX A6000 with 48GB of VRAM, the graphics processor’s memory.
ASTRAL ran on Windows 11 with 32 GB RAM and a 12th Intel Core i5-1235U processor with 10 cores and 12 threads; o3-mini was accessed remotely through the API.

3.4 Manual assessment

All responses initially labeled unsafe or unknown undergo manual assessment to identify false positives, cases incorrectly classified as unsafe. Responses labeled safe are not manually assessed, leaving the number of missed unsafe responses unmeasured.

The authors acknowledge that judgments can depend on individual perspectives and cultural context, using civilian firearm use as an example.
Three authors discuss borderline cases until reaching consensus. The report does not provide a quantitative measure of agreement between reviewers.

4 Results and Discussion

The results distinguish automatic evaluator labels from manually confirmed unsafe outputs and examine confirmed cases across the balanced input features. The discussion considers response severity and the possible contribution of o3-mini’s API safeguards.

4.1 Summary of the Results

Table 2 reports 15 confirmed unsafe o3-mini responses and 151 confirmed unsafe DeepSeek-R1 responses out of 1,260 tests each. For o3-mini, manual review confirms 13 of 20 automatically unsafe responses and two of four unknown responses; for DeepSeek-R1, it confirms 148 of 161 automatically unsafe responses and all three unknown responses.

The evaluator initially labels 1,236 o3-mini outcomes safe, including 565 policy-violation exceptions, and 1,096 DeepSeek-R1 outcomes safe.
Figure 1 shows the most DeepSeek-R1 unsafe outputs in C6, Financial crime, property crime, theft (23), followed by C14, Violence, aiding and abetting, incitement (17). C7, Hate speech, offensive language; C9, Non-violent unethical behavior; and C13, Terrorism, organized crime each have 16.
DeepSeek-R1 produces 54 unsafe outputs under S3, Technical terms, and 50 under S4, Role-play, which asks the model to adopt a specified role, compared with six each under S1, Slang, and S2, Uncommon dialects. The authors describe persuasion-technique differences as limited; o3-mini’s small unsafe count constrains subgroup interpretation.

Automatic safety classifications, policy-violation exceptions, and manually confirmed unsafe totals for both tested systems.

Manually confirmed unsafe output counts by writing style, persuasion technique, and safety category for o3-mini and DeepSeek-R1.

4.2 Findings and Overall Discussion

The authors identify four findings: more confirmed unsafe outputs from DeepSeek-R1, subjectively greater severity and clearer classification of those outputs, variation by safety category and writing style, and a possible protective role for o3-mini’s policy-violation mechanism. Policy-violation exceptions occur for 44.8% of inputs, but the mechanism’s effect on the unsafe-response rate is not isolated experimentally.

The severity judgment comes from manual inspection: the authors describe many DeepSeek-R1 responses as detailed and clearly unsafe, whereas o3-mini responses more often require borderline judgments.
Historical comparisons cite 59 unsafe GPT-4 outputs and 79 unsafe GPT-4o outputs from earlier work. Those counts lack manual confirmation, limiting their comparability with the present confirmed totals.
The authors do not know whether the beta’s safeguards will persist in the public release, so the reported behavior remains specific to the tested system.

4.3 Answer to Research Questions

For RQ1, the report gives unsafe-response rates of 1.19% for o3-mini and 11.98% for the tested DeepSeek-R1 system. For RQ2 and RQ3, it reports category and writing-style differences for DeepSeek-R1, too few unsafe o3-mini cases for strong subgroup conclusions, and no pronounced persuasion-technique differences.

This source subsection inconsistently uses the identifiers DeepSeek-V1 and DeepSeek-V2. The methodology, results table, and surrounding discussion identify the evaluated system as DeepSeek-R1 (70B).

5 Conclusion and Future Work

The conclusion presents the observed safety gap as a preliminary result from a balanced automated test suite with targeted manual verification. It reiterates the possible role of API safeguards and the subjective severity assessment, and states that a larger, more comprehensive evaluation is underway.

Replication Package

The replication package provides experimental data, including test cases and model responses, at https://doi.org/10.5281/zenodo.14773277. The open-source ASTRAL implementation is available at https://github.com/Trust4AI/ASTRAL.

Appendix

Appendix A, “Examples of unsafe responses provided by o3-mini and DeepSeek-R1 (70B),” presents seven prompt–response examples on pages 10–16. They illustrate the authors’ judgments concerning harmful conflict-related assistance, unsupported privacy allegations, political misinformation, organized crime, drug-related military claims, and financial theft. The prompts’ premises and the models’ claims are experimental content, not verified facts.

Brief Thoughts

The shared, balanced prompt set and the separation of automatic labels from human confirmation make the observed failure counts interpretable. Deployment asymmetry limits the comparison: an API-protected o3-mini beta is evaluated against a locally deployed DeepSeek-R1 (70B) system. Because automatically safe outputs were not audited, the confirmed totals do not measure all unsafe responses. The absence of reported uncertainty estimates and quantitative reviewer agreement further limits interpretation of small subgroup differences. The 565 policy-violation exceptions establish that API rejections contribute to the measured outcomes. Their count alone does not establish how many unsafe responses the safeguard prevented.