Add support for Unicode surrogate characters and planes above zero. #213

This adds support for emoticons and unicode planes above 0, but requires using a font that support such characters.
The changes are internal and can be backported to R14.1.x.
A more complete solution involves searching for an alternative suitable font when unsupported characters are encountered. This requires API changes and therefore will only be part of R14.2.0 and will be addressed in subsequent PRs.

This adds support for emoticons and unicode planes above 0, but requires using a font that support such characters. The changes are internal and can be backported to R14.1.x. A more complete solution involves searching for an alternative suitable font when unsupported characters are encountered. This requires API changes and therefore will only be part of R14.2.0 and will be addressed in subsequent PRs.

Solved TDE/tde#218 and partially addresses TDE/tqt3#162.

I did a test building on R14.1.x and everything seems to be fine.
I do not see side effects, I see unicode smileys in Kopete history.

I did a test building on R14.1.x and everything seems to be fine. I do not see side effects, I see unicode smileys in Kopete history.

I'm a bit late but I can confirm it works with KWrite.

First screenshot is using default monospace font. Emoji does not show indeed but the correct number of tofu boxes is shown.
Second screenshot is using the Noto Color Emoji font.

There seems to be a bug, it seems like the editbox still treats the emoji as two characters (I assume this is due to the character not being treated as a surrogate pair). The first one renders as the emoji correctly and the second one renders as a white space. You can highlight any one of those two characters but then the emoji disappears unless you highlight both. Pressing backspace deletes the second character first and then the first one is still left and renders itself as white space.

~~I assume this bug is related to the widget as Konqueror seems to render the emoji correctly.~~ Konqueror also seems to treat it as two characters but somehow renders it correctly as one?

I'm a bit late but I can confirm it works with KWrite. - First screenshot is using default monospace font. Emoji does not show indeed but the correct number of tofu boxes is shown. - Second screenshot is using the Noto Color Emoji font. There seems to be a bug, it seems like the editbox still treats the emoji as two characters (I assume this is due to the character not being treated as a surrogate pair). The first one renders as the emoji correctly and the second one renders as a white space. You can highlight any one of those two characters but then the emoji disappears unless you highlight both. Pressing backspace deletes the second character first and then the first one is still left and renders itself as white space. ~~I assume this bug is related to the widget as Konqueror seems to render the emoji correctly.~~ Konqueror also seems to treat it as two characters but somehow renders it correctly as one?

1.png

17 KiB

2.png

17 KiB

Hi Philippe,
thanks for testing and for the valuable feedback. It definitely seems there is more work to do as you highlighted.
For info, here is how surrogates work at the moment:

they are represented as two TQChar, so a string containing a single emoji will consist of two TQChar and the length will be 2. This is the same in Qt4.
when rendering a surrogate character, the font engine checks whether we have a high surrogate TQChar and if so, it checks if the following TQChar is a low surrogate. If this happens, the two TQChar are represented as a single glyph (for example an emoji).

I suspect we will need some more changes to handle:

deletion of a surrogate character should remove two TQChars from a string. This should fix the problem that currently shows up when pressing backspace in the editor right after an emoji
the widget displaying the text is clearly displaying things in a wrong way now, so likely there is some logic that needs to be changed there.

I will be looking at both issues in coming days.

Hi Philippe, thanks for testing and for the valuable feedback. It definitely seems there is more work to do as you highlighted. For info, here is how surrogates work at the moment: - they are represented as two TQChar, so a string containing a single emoji will consist of two TQChar and the length will be 2. This is the same in Qt4. - when rendering a surrogate character, the font engine checks whether we have a high surrogate TQChar and if so, it checks if the following TQChar is a low surrogate. If this happens, the two TQChar are represented as a single glyph (for example an emoji). I suspect we will need some more changes to handle: - deletion of a surrogate character should remove two TQChars from a string. This should fix the problem that currently shows up when pressing backspace in the editor right after an emoji - the widget displaying the text is clearly displaying things in a wrong way now, so likely there is some logic that needs to be changed there. I will be looking at both issues in coming days.

I did some investigation on Qt4/KDE4 and Qt5/KDE5 and the situation is as follow.

Qt4/KDE4: deletion of a surrogate char only deletes the low surrogate QChar. The high surrogate is then rendered as a tofu box. Selection of characters with mouse or keyboard works fine.
Qt5/KDE5: deletion of a surrogate char correctly removes both QChar and no tofu boxes are displayed. Selection of characters with mouse or keyboard also works fine.

Looking at the code, QString does not do any special handling, which means it is possible to delete the low surrogate without deleting the high surrogate if someone wants to do it. Instead the logic for selection/deletion seems to be handled by the widget through the use of the charStop (Qt4) or graphemeBoundary (Qt5, see here) in the QCharAttributes struct.
TQt3 also has such structure, so it seems to me the right approach is to fix the widget classes without modifying TQString logic on char deletion.

What do you think?

I did some investigation on Qt4/KDE4 and Qt5/KDE5 and the situation is as follow. - Qt4/KDE4: deletion of a surrogate char only deletes the low surrogate QChar. The high surrogate is then rendered as a tofu box. Selection of characters with mouse or keyboard works fine. - Qt5/KDE5: deletion of a surrogate char correctly removes both QChar and no tofu boxes are displayed. Selection of characters with mouse or keyboard also works fine. Looking at the code, QString does not do any special handling, which means it is possible to delete the low surrogate without deleting the high surrogate if someone wants to do it. Instead the logic for selection/deletion seems to be handled by the widget through the use of the `charStop` (Qt4) or `graphemeBoundary` (Qt5, see [here]) in the `QCharAttributes` struct. TQt3 also has such structure, so it seems to me the right approach is to fix the widget classes without modifying `TQString` logic on char deletion. What do you think? [here]: https://codebrowser.dev/qt5/qtbase/src/widgets/widgets/qwidgetlinecontrol.cpp.html#_ZN18QWidgetLineControl3delEv

This indeed looks like an acceptable solution to me.

This indeed looks like an acceptable solution to me.

Ok, I will need some time to study the code related to TQCharAttributes, so it may take a while to get it fixed, but I will proceed in this direction.

> This indeed looks like an acceptable solution to me. Ok, I will need some time to study the code related to TQCharAttributes, so it may take a while to get it fixed, but I will proceed in this direction.

Ok, I will need some time to study the code related to TQCharAttributes, so it may take a while to get it fixed, but I will proceed in this direction.

PR #217 fixes editing in TQt3, for example TQLineEdit and TQTextEdit. KEdit also benefits from this PR, while KWrite and Kate are likely to need more work in tdelibs/tdebase.

> Ok, I will need some time to study the code related to TQCharAttributes, so it may take a while to get it fixed, but I will proceed in this direction. PR #217 fixes editing in TQt3, for example TQLineEdit and TQTextEdit. KEdit also benefits from this PR, while KWrite and Kate are likely to need more work in tdelibs/tdebase.

Labels Milestones

Add support for Unicode surrogate characters and planes above zero. #213

Reviewers