Captcha Attack: Turning Captchas Against Humanity | Summary

This article explains the key points of Captcha Attack: Turning Captchas Against Humanity.

• 2022-01-11 (arXiv)
• Conti, Mauro, Pajola, Luca, Tricomi, Pier Paolo.
• Department of Mathematics, University of Padua
• Paper, version 3 (revised 2022-04-04; this review uses the revised version)

Read this article in Korean

Summary

• The paper studies how images containing distorted text can evade automatic systems that screen online posts for harmful content. In a constructed pipeline combining optical character recognition (OCR), which extracts text from images, with a text moderator, CAPtcha Attack (CAPA) achieved attack success rates close to 100% for Claptcha and Multicolor images, compared with much lower rates for undistorted images.
• A user study reported 94.53% average transcription accuracy on harmless examples. A defense based on Local Outlier Factor (LOF), which identifies samples unlike ordinary posts, achieved about 80% average F1-score, a measure combining precision and recall, across three social media datasets but flagged approximately 9% of benign posts.

I. Introduction

Online platforms use automatic content moderators (ACM) to help human reviewers handle potentially harmful content. Crafted inputs can exploit weaknesses throughout the moderation pipeline, including preprocessing.

a) Contribution

The authors organize observed obfuscations into a taxonomy, introduce CAPA, test moderation APIs, and evaluate supervised and outlier-detection defenses. Their examples include text distortions and substitutions that can remain understandable to readers.

b) Paper organization

The paper proceeds from background and the obfuscation taxonomy to the attack, dataset, results, defenses, and conclusions.

II. Background & Related Works

The background covers security of machine learning applications, content moderation, and the development of textual CAPTCHA challenges.

A. Security of Machine Learning Applications

An evasion attack supplies crafted inputs to affect a system’s predictions, while a poisoning attack changes training data. CAPA targets image-to-text extraction without optimizing against a particular classifier.

B. Content Moderators

Moderators handle text, images, video, and audio; hateful meme detection can combine visual and textual information. The paper cites prior examples of image transformations and textual changes affecting moderation systems.

C. Captcha

A CAPTCHA is a challenge intended to distinguish humans from computers. Textual variants use transformations such as rotation and distortion, while advances in automated recognition have motivated other challenge types.

III. The New Era of Textual Captcha

The authors group observed obfuscations by whether they may disrupt text extraction or the subsequent classifier.

A. Challenges from OSN’s users: a Taxonomy

The taxonomy comes from manual inspection of the latest 100 posts on each of three English and three Italian Instagram pages and the latest 1,000 English posts for four hashtags. Hard backgrounds appeared in every listed source and reached 77% for one page; the selected sources do not establish platform-wide prevalence.

B. OCR-failure

OCR-failure techniques include scene text, meaning text in real-world scenes, emoji, letter-shaped objects, hard backgrounds, distorted text, and occluding items. These can complicate extraction or require interpretation beyond ordinary document text.

The taxonomy separates extraction failures from classifier failures, showing how transformations can affect different pipeline stages.

Hard backgrounds range from 14.0% to 77.0% across the listed sources, while letter-shaped objects are rare in this sample.

C. Classifier-failures

Classifier-level obfuscations include slang, leet speech, which replaces characters with visually similar symbols, and typos. These can affect classification even when the text is extracted.

IV. Our Attack: CAPA

CAPA turns attacker-selected text into a custom textual captcha, aiming to preserve human readability while disrupting moderation.

A. Target System

The target abstraction routes plain text and images to language and vision classifiers; images containing text require an additional extraction step. The authors do not specify how the classifiers should cooperate.

B. Attack Motivation

The authors use generated captchas as a scalable simplification of observed posts because collecting a large and varied dataset of real obfuscations is difficult. Their generated images omit harmful visual content beyond the text.

Text embedded in an image passes through OCR before language classification, creating the extraction dependency exploited by CAPA.

C. Attack Execution

For moderator M and offensive text x, the attack seeks a transformation T for which M(x) = c_i and M(T(x)) = c_j, with c_i offensive and c_j non-offensive. T first renders text as an image and then applies image transformations; it is designed to require no knowledge of the target model.

The diagram shows the sequence from attacker-selected text to an image and then a transformed captcha posted online.

V. CAPA Dataset

The CAPA dataset tests four image styles, and a separate user study assesses readability.

A. Dataset Generation

From 1,383 English words associated with hateful sentences, the authors retained 502 for which Microsoft Content Moderator assigned a score above 0.5 in at least one of three inappropriate-content categories. Four styles per word produce 2,008 images: Clean, Claptcha, Multicolor, and Homemade; the dataset is available to researchers on request.

B. Captchas Readability

Fifty participants transcribed harmless English verbs in 600 generated images, with no time limit; the paper says each participant annotated 50 samples and each sample was processed by five participants, counts that are not mutually consistent. Mean accuracy was 94.53%, character error rate, a measure of character differences from the reference, was 1.31%, and mean difficulty scores were 1.3 for Claptcha, 1.5 for Multicolor, and 1.3 for Homemade.

Clean uses only domain transfer, whereas the other styles add different combinations of rotation, background, font, color, and occlusion transformations.

The same example word illustrates how the four styles differ visually, from plain text to distorted or occluded text.

The distributions concentrate near high transcription accuracy and low character error for all three transformed styles in the readability study.

VI. Attack Results

The experiments separately assess deployed image moderation APIs and a constructed pipeline that extracts image text before text moderation.

A. Overview

The authors tested APIs instead of posting harmful material to social networks. Clean images test whether image moderators handle embedded text, while transformed images test the OCR-based pipeline.

B. Image Moderators

Attack success rate (ASR) is the fraction of unsafe samples not detected. For Clean images, ASR was 1.0 for Amazon and Microsoft image moderation and 0.96 for Google SafeSearch; Google’s flagged 4% were classified as spoofed.

C. Cross-domain Moderators

The constructed pipeline pairs Amazon, Google, Microsoft, or Tesseract OCR with Microsoft Content Moderator for text. Claptcha and Multicolor yielded ASR close to 1.0, while Homemade averaged 0.8; normalized Levenshtein distance, a length-normalized character edit distance between extracted and true text, showed similar trends.

The transformed styles evade the constructed pipelines much more often than Clean images, although Homemade varies by OCR service.

VII. Defense: Challenges & Possible Directions

The paper evaluates both supervised captcha classification and unsupervised outlier detection as possible defenses.

A. Overview

The proposed pipeline places a CAPA detector before image analysis and sends detected images for blocking or human review. This requires distinguishing suspicious captcha-like images from ordinary posts.

B. A Supervised Approach: Classification

The supervised experiments combine 7,700 examples from 11 known captcha schemes with Pinterest, Twitter, or YFCC100M images, using 70% training, 10% validation, and 20% testing splits. A small convolutional classifier and fine-tuned Alexnet, Resnet18, and VGG obtain near 100% F1 on known schemes, but detection is zero in 29 of 36 model, dataset, and unseen CAPA-style combinations.

• Training uses stochastic gradient descent with learning rate 0.001, momentum 0.9, cross-entropy loss, early stopping after five epochs without validation-loss improvement, and a maximum of 200 epochs.

The added detector diverts recognized CAPA images before the usual OCR and classifier path.

C. An Unsupervised Approach: Outlier Detection

The unsupervised approach trains on ordinary posts and uses pretrained ResNet-18 image representations, principal component analysis to reduce their dimensions, and four outlier detectors. Across three datasets, LOF achieves about 80% average F1 and performance stabilizes from four known validation styles, but approximately 9% of benign posts are false positives.

• For each platform, 50,000 ordinary images are split 70%/10%/20%; validation uses k in {2, 4, 6, 8, 10} known captcha styles, with five style combinations per setting.
• Grid search covers principal component counts 2, 8, 64, and 128 and assumed outlier proportions of 0.1, 0.05, and 0.01. Unknown styles appear only in the test set.

The defense datasets combine 7.7 thousand captcha examples with much larger non-captcha sets, reflecting the intended class imbalance.

The two-dimensional embedding shows separate clusters for several captcha styles and a broad Pinterest cluster, supporting the authors' investigation of outlier detection.

LOF generally produces the strongest F1 curves across the three datasets, with smaller gains after four known validation styles.

VIII. Conclusions and Future Works

The authors identify open problems in defining the boundary between captchas and ordinary images, collecting data for the broader taxonomy, and detecting varied obfuscations.

Appendix

• Appendix A illustrates the 11 C11 captcha schemes and transformations. Appendix B plots attack success and normalized edit distance across OCR services; Appendix C reports supervised results and outlier detection accuracy for known and unknown styles.

Brief Thoughts

The separation between the image-only API tests and the constructed OCR-plus-text pipeline makes the failure modes interpretable, but neither directly measures a complete social network’s deployed moderation system. The defense’s unseen-style failures and benign-post false positives make generalization and review workload practical limitations.