Source-preserving Persian integration

Add Persian diacritics in Python without changing the original text.

Suppose a student types a Persian sentence into your app. You want to add vowel marks while keeping their spelling, punctuation and half-spaces exactly as entered. This Python example does both.

Keep the source; add a reading layer

POST the exact Persian string to /v1/diacritize. Keep original as your source of record. Use marked for a reading display, or apply operations to the original string yourself. Verify that the returned source is identical and that the marked form can be reconstructed entirely from those additions.

The same spelling can need different vowels

A flower

گل قرمز را در گلدان گذاشت.

“She/he put the red flower in the vase.” Here گل is gol, a flower.

Mud

پس از باران، گل به کفش‌ها چسبیده بود.

“After the rain, mud had stuck to the shoes.” Here گل is gel, mud.

These original examples cannot be solved by replacing every occurrence ofگل with one marked word. A Persian diacritizer has to read the sentence. Ezafe adds another decision: whether a noun connects to the following adjective, possessor or name. The homograph guide and Ezafe guide explain the language behind those decisions.

You may see this task called Persian vowelization, vocalization, harakat or adding Farsi vowel marks. Shared Arabic-script marks do not make an Arabic tashkeel tool a Persian pronunciation engine. Persian has its own sound system and sentence context.

Send the sentence and reconstruct the marked form

Get approved Developer access and a key with thediacritize scope. Keep it in VOWELMARKS_API_KEY. Save a UTF-8 Persian file as script.txt, set a stableVOWELMARKS_REQUEST_ID such as lesson-001-marks, then runpython3 persian-diacritics.py script.txt. Python 3.10 or later is sufficient; no external package is required.

Download the full Python example.

"""Apply API mark operations to the exact source. Python 3.10+."""
import json
import os
import re
import sys
import unicodedata
from urllib.error import HTTPError
from urllib.request import Request, urlopen


ARABIC_MARKS = re.compile(r"^[\u0610-\u061a\u064b-\u065f\u0670\u06d6-\u06ed]+$")


def apply_operations(source, operations):
    additions = {}
    for operation in operations:
        index, mark = operation["source_index"], operation["mark"]
        if type(index) is not int or not 0 <= index < len(source):
            raise ValueError("Mark operation has an invalid source index")
        character = source[index]
        is_arabic_letter = (
            unicodedata.category(character).startswith("L")
            and unicodedata.name(character, "").startswith("ARABIC ")
            and character != "\u0640"  # Tatweel is a joining extender, not a source letter.
        )
        if not is_arabic_letter or not isinstance(mark, str) or not ARABIC_MARKS.fullmatch(mark):
            raise ValueError("Operation must attach an Arabic mark to an Arabic-script letter")
        marks = additions.setdefault(index, [])
        if mark not in marks:
            marks.append(mark)
    return "".join(character + "".join(additions.get(index, []))
                   for index, character in enumerate(source))


def checked_reading(source, result):
    if result["original"] != source:
        raise ValueError("The API original does not match the submitted text")
    reconstructed = apply_operations(source, result["operations"])
    if reconstructed != result["marked"]:
        raise ValueError("Marked text does not match the documented operations")
    return reconstructed


def diacritize(source, request_id, include_ezafe=True):
    base = os.environ.get("VOWELMARKS_API_BASE_URL", "https://api.vowelmarks.com")
    payload = {"text": source, "include_ezafe": include_ezafe}
    request = Request(
        base.rstrip("/") + "/v1/diacritize",
        data=json.dumps(payload, ensure_ascii=False).encode("utf-8"), method="POST",
        headers={
            "Authorization": "Bearer " + os.environ["VOWELMARKS_API_KEY"],
            "Content-Type": "application/json", "Idempotency-Key": request_id,
        },
    )
    try:
        with urlopen(request, timeout=120) as response:
            result = json.load(response)
    except HTTPError as error:
        detail = json.loads(error.read()).get("error", {})
        raise RuntimeError(
            f"HTTP {error.code}: {detail.get('code', 'api_error')}; "
            f"retryable={detail.get('retryable', False)}; "
            f"Retry-After={error.headers.get('Retry-After', 'not supplied')}"
        ) from None
    return checked_reading(source, result)


if __name__ == "__main__":
    if len(sys.argv) != 2:
        raise SystemExit("Usage: python3 persian-diacritics.py script.txt")
    with open(sys.argv[1], encoding="utf-8", newline="") as file:
        source = file.read()
    print(diacritize(source, os.environ["VOWELMARKS_REQUEST_ID"]))

The response supplies original, marked andoperations. An operation such as{"source_index": 3, "mark": "ِ", "type": "ezafe"}adds a linking kasra after the fourth code point. It does not replace the fourth letter or point to the fourth word. The operations describe where to put marks, not how confident the system is about a reading.

Do not strip every mark to prove preservation

A tempting check is to remove all combining marks from the result and compare it with the input. That fails when the input already contains marks. It can also erase meaningful combining characters from mixed-language text. The example takes a stricter approach: begin with the original characters and add only the returned operations.

  • original == source checks the exact parsed string, including whitespace.
  • Each operation adds a permitted Arabic-script mark after an Arabic-script letter, never after Latin text, a digit or punctuation.
  • The reconstructed string must equal marked; it cannot hide a spelling rewrite.
  • Existing marks, digits, punctuation, Latin words and the half-space remain source characters.

A useful mixed-text test

نرگس می‌گوید: «گُل را ببر.» ۲ کتاب در PDF است. 🙂

This original test includes a ZWNJ in می‌گوید, an existing vowel in گُل, quotation marks, a Persian digit, an English abbreviation and an emoji. Add a line break and two consecutive spaces to your local test too. Preserve all of them, whether or not the reading engine adds a mark to every Persian word.

Python 3 indexes strings by Unicode code point. JavaScript strings use UTF-16 code units: an emoji can occupy two of those units. Convert the API source withArray.from(result.original) before applying code-point indexes in JavaScript. Neither language should use UTF-8 byte positions for these operations.

Make Ezafe a deliberate display choice

The example sends include_ezafe: true. To omit newly added Ezafe marks, calldiacritize(source, request_id, include_ezafe=False) with a new request key. This leaves existing Ezafe marks and other short-vowel cues intact. The original text remains the version you store and edit.

In a teaching interface, one useful view shows the original source first and reveals the marked reading after a learner has tried it. In an editor, show both versions so the author can inspect the reading without accepting a replacement document.

Render the text without turning it into HTML

Keep the original string in your data model. For a simple display, assign the checked marked string with textContent, not innerHTML. Give the elementlang="fa" and dir="rtl". A sentence containing a less-than sign or an embedded link should remain text, not become executable markup.

If you color newly added marks, build text nodes and spans from the indexed operations. Do not use a search-and-replace on word spellings: a word may occur twice with different readings. Save the source and its version together with the reading result so old offsets cannot be applied to newly edited text.

Preservation and pronunciation are different tests

Passing the reconstruction check proves that the reading layer did not rewrite the source. It does not prove that every added vowel is the intended one. Review names, ambiguous sentences and specialized vocabulary with a proficient Persian reader before publishing a lesson. Do not force one answer onto a sentence that genuinely permits two.

The VowelMarks accuracy report separates evaluation evidence from source integrity. For integration details, use the diacritization reference; for spoken lesson audio, continue with the Python and JavaScript speech tutorial.

Building a reading app or lesson tool? Explore the Persian API and apply for access. To mark a passage without code, open the Persian diacritics tool.

Frequently asked questions

Can Python add Persian vowel marks automatically?

Python can call the VowelMarks diacritization API and render the returned reading marks. The contextual reading comes from the API, not a rule that inserts the same vowels into every matching spelling.

How do I check that Persian diacritization did not rewrite my text?

Compare original with the submitted string, reconstruct marked text by adding the returned operations to the exact source, and compare that reconstruction with marked. Preserve existing marks, half-spaces, punctuation and whitespace.

Are source_index values Python string indexes?

They are Unicode code-point indexes, matching Python 3 string indexing for valid Unicode text. In JavaScript, use Array.from(text), not UTF-16 string offsets or byte offsets.

Does turning off Ezafe remove all Persian vowel marks?

No. include_ezafe: false omits newly generated operations classified as Ezafe; other reading marks and anything already present in your source remain.

Sources and review notes

Sources are listed for the claims they support. Original practice examples are identified in the article. Product capabilities and access terms were checked on the review date and can change.

  1. VowelMarks diacritization reference. Exact source text, code-point operation offsets and the include_ezafe option.
  2. VowelMarks OpenAPI v1. The public request, response and operation fields used in the Python example.
  3. University of Texas at Austin, Ezāfe 1. The linking sound in Persian noun phrases and its written forms.
  4. Rahmati and Sameti, GE2PE (2024). Persian pronunciation research treats Ezafe, homographs and unfamiliar words as distinct challenges. This is not an accuracy claim for the example.