Python

I have written previously about stripping syllabic stress marks from Russian text using a Perl-based regex tool. But I needed a means of doing in solely in Python, so this just extends that idea.

#!/usr/bin/env python3

def strip_stress_marks(text: str) -> str:
   b = text.encode('utf-8')
   # correct error where latin accented ó is used
   b = b.replace(b'\xc3\xb3', b'\xd0\xbe')
   # correct error where latin accented á is used
   b = b.replace(b'\xc3\xa1', b'\xd0\xb0')
   # correct error where latin accented é is used
   b = b.replace(b'\xc3\xa0', b'\xd0\xb5')
   # correct error where latin accented ý is used
   b = b.replace(b'\xc3\xbd', b'\xd1\x83')
   # remove combining diacritical mark
   b = b.replace(b'\xcc\x81',b'').decode()
   return b

text = "Том столкну́л Мэри с трампли́на для прыжко́в в во́ду."

print(strip_stress_marks(text))
# prints "Том столкнул Мэри с трамплина для прыжков в воду."

The approach is similar to the Perl-based tool we constructed before, but this time we are working working on the bytes object after encoding as utf-8. Since the bytes object has a replace method, we can use that to do all of the work. The first 4 replacements all deal with edge cases where accented Latin characters are use to show the placement of syllabic stress instead of the Cyrillic character plus the combining diacritical mark. In these cases, we just need to substitute the proper Cyrillic character. Then we just strip out the “combining acute accent” U+301 → \xcc\x81 in UTF-8. After these replacements, we just decode the bytes object back to a str.

For one-off projects that target Anki collections, I often use Python in a standalone application rather than an Anki add-on. Since I’m not going to distribute these little creations that are specific to my own needs, there’s no reason to create an add-on. These are just a few notes - nothing comprehensive - on the process.

One thing to be aware of is that there must be a perfect match between the Anki major and minor version numbers for the Python anki module to work. If you are running Anki 2.1.48 on your desktop application but have the Python module built for 2.1.49, it will not work. This is a huge irritation and there’s no backwards compatibility; the versions must match precisely.

I write in Markdown because it’s much easier to keep the flow of writing going without taking my hands off the keyboard.

I also like to write content in Anki cards in Markdown. Over the years there have been various ways in of supporting this through add-ons:

• The venerable Power Format Pack was great but no longer supports Anki 2.1, so it became useless.
• Auto Markdown worked for a while but as of Anki version 2.1.41 does not.
• After Auto Markdown stopped working, I installed the supposed fix Auto Markdown - fix version but that didn’t work either.
• It’s possible that the Mini Format Pack will work, but honestly I’m tired of the constant break-fix-break-fix cycle with Anki.

The problem

The real problem with Markdown add-ons for Anki is the same as every other add-on. They are all hanging by a thread. Almost every minor point upgrade of Anki breaks at least one of my add-ons. It’s nearly impossible to determine in advance whether an Anki upgrade is going to break some key functionality that I rely on. And add-on developers, even prominent and prolific ones come and go when they get busy, distracted or disinterested. It’s one of the most frustrating parts of using Anki.

As readers of this blog know, I’m an avid user of Anki to learn Russian. I have a number of sources for reference content that go onto my Anki cards. Notably, I use Wiktionary to get word definitions and the word with the proper syllabic stress marked. (This is an aid to pronunciation for Russian language learners.)

Since I’m lazy to the core, I came up with a system way of grabbing the stress-marked word from the Wiktionary page using lxml and XPath.

W3schools.com has a CSS library that’s quite nice. I often use Bootstrap; but I like some of the visual features here better. For example, I like their tags because they have more flexible use of colour.

If you want to fetch from a Python dictionary, but you need a default value, this is how you do it:

upos_badge = {'noun': 'lime','verb': 'amber', 'adv': 'blue',}
badge_class_postfix = upos_badge.get(value.lower(), 'light-grey')

I recently learned about DeepL as an alternative to Google Translate. It seems really good.

Previously, I wrote about adding syllabic stress marks to Russian text. Here’s a method for doing the opposite - that is, removing such marks (ударение) from Russian text.

Although there may well be a more sophisticated approach, regex is well-suited to this task. The problem is that

def string_replace(dict,text):
   sorted_dict = {k: dict[k] for k in sorted(dict)}
   for n in sorted_dict.keys():
      text = text.replace(n,dict[n])
   return text

dict = { "а́" : "а", "е́" : "е", "о́" : "о", "у́" : "у",
      "я́" : "я", "ю́" : "ю", "ы́" : "ы", "и́" : "и",
      "ё́" : "ё", "А́" : "А", "Е́" : "Е", "О́" : "О",
      "У́" : "У", "Я́" : "Я", "Ю́" : "Ю", "Ы́" : "Ы",
      "И́" : "И", "Э́" : "Э", "э́" : "э"
   } 
   
print(string_replace(dict, "Существи́тельные в шве́дском обычно де́лятся на пять склоне́ний."))

This should print: Существительные в шведском обычно делятся на пять склонений.

Building on my earlier explorations of the UDAR project, I’ve created a macOS Service-like method for in-situ marking of syllabic stress in arbitrary Russian text.

The Keyboard Maestro is simple; we execute the following script, bracketed by Copy and Paste:

#!/Users/alan/.pyenv/shims/python3
import xerox
import udar
import re

rawText = xerox.paste()
doc1 = udar.Document(rawText, disambiguate=True)
searchText = doc1.stressed()
result = re.sub(r'( ,)', ",", searchText)
xerox.copy(result)

This presumes that udar and its prerequisites have already been installed, of course.

So why not build out this idea as an actual macOS text service? In theory, it should be possible. Maybe even ideal but for Python version management I use pyenv. Because some of the UDAR dependencies will not run under my current system Python version of 3.8.5, I use 3.7.4 under pyenv and it appears that the correct Python version run under whatever environment the service launches. Someone with deeper system knowledge could undoubtedly figure it out; but instead I accomplished the same effect via a Keyboard Maestro macro.

After installing Stanza as dependency of UDAR which I recently described, I decided to play around with what is can do.

Installation

The installation is straightforward and is documented on the Stanza getting started page.

First,

sudo pip3 install stanza

Then install a model. For this example, I installed the Russian model:

#!/usr/local/bin/python3
import stanza
stanza.download('ru')

Usage

Part-of-speech (POS) and morphological analysis

Here’s a quick example of POS analysis for Russian. I used PrettyTable to clean up the presentation, but it’s not strictly-speaking necessary.

One of the challenges that Russian learners face is the placement of syllabic stress, an essential determinate of pronunciation. Although most pedagogical texts for students have marks indicating stress, practically no tests intended for native speakers do. The placement of stress is inferred from memory and context.

I was delighted to discover Dr. Robert Reynolds’ work on natural language processing of Russian text to mark stress based on grammatical analysis of the text. What follows is a brief description of the installation and use of this work. The project page on Github has installation instructions; but I found a number of items that needed to be addressed that were not covered there. For example, this project (UDAR) depends on Stanza; which in turn requires a language-specific (Russian) model.