dehyphen

dehyphen is a Python package for the dehyphenation of broken text, e.g. extracted from a PDF. It reconstructs the original continuous text by choosing the most probable way to join lines or paragraphs, and removes hyphens on the way. The options are scored with the perplexity of character-based Flair language models. dehyphen reads Flair’s model files and runs them itself, with numpy on the CPU: it needs neither Flair nor PyTorch.

For this input

die Bedeutung der finan-

ziellen Interessen der Union

dehyphen joins the lines and removes the -:

die Bedeutung der finanziellen Interessen der Union

pip install dehyphen

Usage and the available models: https://github.com/pd3f/dehyphen#readme