🎨 Added an option to use unicode slugs in URL:s

[dittnamn]

fixes #3224
The slugs generated by Ghost have had a lot of issues in how they've been transliterated. Through a change in @tryghost/string a new transliteration library, any-ascii, has been proposed to replace unidecode, see TryGhost/SDK#1011

However, to get a fully internationalized site along with SEO optimized URL:s, that change isn't fully enough. The current change adds an option in Labs where it's possible to enable Unicode slugs instead of performing the transliteration. The Unicode characters in use are limited to those that are registered as letters and numbers, which means that emojis, special characters, etc. still will be removed.

The only "major" change that has been done in Ghost to make this work is actually just to normalize the URL slugs before looking them up in the database. The rest of the code changes are mostly settings to turn the new feature on or off, and to pass the setting along all the way to the slugify() function. Due to the rest of the system already being in Unicode, everything seems to work as it's supposed to.

Instead of using safe() from @tryghost/security, the slug generation has also been changed to utilize slugify() from @tryghost/string directly instead. The routing controllers for rss, collection and channel were previously using safe() as well to slugify the lookups before going to the router, but the entry controller didn't do this. As the router already checks the slugs with isSlug() before passing them further, this was unnecessary and had the potential to break lookups, so it was simply removed.

In addition to using Unicode slugs, there's also an option added to switch which separator the slugs use. Previously, dashes (-) were used as the hardcoded default, but for URL readability, underscores (_) could be preferred, like in the URL:s Wikipedia use. There's also an option for using spaces ( ), but this might still be seen as a bit foreign in URL:s. Out of the major browsers, it's currently just Firefox that show these as spaces by default, with Safari doing it in some situations. Other browsers can show the spaces as %20.

When activated, the Unicode slugs aren't added to member tags, newsletter, services, integrations or benefits, as these aren't user facing.

Note that for this change to work, the changes in TryGhost/SDK#1011 and TryGhost/framework#860 are required.

• I've read and followed the Contributor Guide
• I've explained my change
• I've written an automated test to prove my change works

[github-actions]

It looks like this PR contains a migration 👀
Here's the checklist for reviewing migrations:

General requirements

• ⚠️ Tested performance on staging database servers, as performance on local machines is not comparable to a production environment
• Satisfies idempotency requirement (both up() and down())
• Does not reference models
• Filename is in the correct format (and correctly ordered)
• Targets the next minor version
• All code paths have appropriate log messages
• Uses the correct utils
• Contains a minimal changeset
• Does not mix DDL/DML operations
• Tested in MySQL and SQLite

Schema changes

• Both schema change and related migration have been implemented
• For index changes: has been performance tested for large tables
• For new tables/columns: fields use the appropriate predefined field lengths
• For new tables/columns: field names follow the appropriate conventions
• Does not drop a non-alpha table outside of a major version

Data changes

• Mass updates/inserts are batched appropriately
• Does not loop over large tables/datasets
• Defends against missing or invalid data
• For settings updates: follows the appropriate guidelines

[9larsons]

Hey @dittnamn, thanks for the suggestion. This is a reasonable request, though it should be a feature request given the scope and it would be good to start with the Ghost forums for that.

As a note, this is lacking a migration and the wiring in Settings isn't quite right. Any new Settings should get reviewed on their own and sequenced ahead, which is just more reason why this should be a discussion first.

[dittnamn]

Hey @dittnamn, thanks for the suggestion. This is a reasonable request, though it should be a feature request given the scope and it would be good to start with the Ghost forums for that.

As a note, this is lacking a migration and the wiring in Settings isn't quite right. Any new Settings should get reviewed on their own and sequenced ahead, which is just more reason why this should be a discussion first.

This was initially reported as a bug twelve years ago in #3224, has been a feature request on the forums for the past eight years, and been up in discussions several times since that as well. Here's a few examples:

https://forum.ghost.org/t/to-turn-off-automatic-url-translation-into-latin/1817
https://forum.ghost.org/t/how-to-use-multi-language-urls/33576
https://forum.ghost.org/t/unable-to-add-regional-language-url-in-the-post-url/50326
https://forum.ghost.org/t/can-i-off-the-automatic-post-url-function-for-korean/3822
https://forum.ghost.org/t/prevent-ghost-from-auto-translate-my-tags/4095
https://forum.ghost.org/t/support-devanagari-script-in-url/4122

I see the current codebase as ready for this change to come in place. With the migration, I assume you mean the updated slugify() function, and I mostly wrote this PR as a method for you to fully test that one out.

For the settings part, I could split that into a separate PR if you feel ready to implement this. I did it like this as a quick way to avoid rewriting the Labs settings to store strings, but I understand it wouldn't be production-ready like this.