How does AI manga translation work?

AI manga translation works in five steps: find the speech bubbles and text, read the text with OCR, translate it, erase the original lettering and rebuild the art underneath, then typeset the translation back into the bubble. Each step is a separate model or program, and the weakest one decides how good the finished page looks.

Step 1: Find the speech bubbles and text

The first job is to locate where the words are. A detection model looks at the page and draws a box or an outline around every block of text: speech bubbles, narration boxes, captions and sometimes lettering that sits directly on the art. Finding the bubble matters as much as finding the words, because the bubble is the space the translation has to fit into later.

This step is why a clean, large scan helps so much. Faint, tiny or tilted lettering is easy to miss, and a block of text that is never found is never translated.

Step 2: Read the text with OCR

OCR (optical character recognition) turns the pixels inside each box into characters a computer can handle. Manga is harder than a printed page. Japanese is often vertical, small kana called furigana sit beside kanji, fonts are stylized and the background can be screentone instead of plain white. Korean and Chinese bring their own problems, such as lookalike characters that only differ by a thin stroke.

If OCR reads a character wrongly, the translation engine will faithfully translate the wrong word. Many results that look like bad translation are really bad reading.

Step 3: Translate the text

The recognized text is sent to a machine translation engine. The engine only sees text, not the picture, so it does not know who is speaking or what is on the page unless the program passes that along. This is the step people usually mean when they say AI translation, but it is only one of five. Slang, jokes, honorifics and short exclamations are where machine translation is weakest.

Step 4: Erase the original lettering

Before the new text can go in, the old text has to go out. The simplest method is to paint a white box over the bubble. That works in a plain white bubble and looks wrong everywhere else. A better method is inpainting: a model fills the erased area by continuing the pattern around it, so screentone, gradients and background lines come back instead of a hole.

Step 5: Typeset the translation

Typesetting means placing the translated words inside the bubble in a readable size, wrapped into lines that fit the shape. English is usually longer than Japanese, so the program shrinks the text until it fits, and it has to avoid breaking words in awkward places. A tall, narrow bubble made for vertical Japanese is the hardest case for horizontal English.

How RuneManga does it today

• Text is found and read on each page. Japanese uses a reader built for manga. Korean and Chinese are supported, and their recognition is still improving.
• Translation runs on RuneLingua, our own AI translator built for manga and manhwa dialogue.
• The original lettering is erased and the background under it filled in, then the translation is set in the same bubble.
• Only the text found on a page goes to the translation provider, not the image.

Where it still fails

• Text outside speech bubbles, such as narration boxes and loose text on the art, is not translated yet.
• Sound effects are drawn into the art and are not translated yet.
• Vertical lettering, stylized fonts and very small or blurry text are the hardest cases for reading.
• A very long translation in a very small bubble comes out small, because the text is shrunk to fit.

Questions

• What is RuneManga? A manga and manhwa translator
• Why RuneManga misses some text on a page
• Why translated text does not fit the bubble
• What is RuneLingua?