I made a quick script to pull the characters that WikiEditor currently has:

VE's current list of special characters is in the 'visualeditor-specialcharinspector-characterlist-insert' message

Note the typeof check - because WikiEditor has this: https://github.com/wikimedia/mediawiki-extensions-WikiEditor/blob/master/modules/jquery.wikiEditor.toolbar.config.js#L581-L643 - I don't think VE can currently handle that.

I think we should, after discussing with @Jdforrester-WMF:

• Move WikiEditor characters out into a separate JSON file in a ResourceLoader module.
• Share it between WikiEditor and VisualEditor somehow
• In VE-MW, override VE core's system for dealing with these, instead building groups from the RL module and allow an extra "Common characters" group set up by a MediaWiki message for the local wiki to customise as desired.

In T91608#1102537, @Krenair wrote:

I think we should, after discussing with @Jdforrester-WMF:

• Move WikiEditor characters out into a separate JSON file in a ResourceLoader module.
• Share it between WikiEditor and VisualEditor somehow
• In VE-MW, override VE core's system for dealing with these, instead building groups from the RL module and allow an extra "Common characters" group set up by a MediaWiki message for the local wiki to customise as desired.

Fully support. The more customizable; the better!

@Catrope, @Krinkle: What do you think about the first two items there?

To possibly make this even worse: we probably want this list to be shared between VE core and WikiEditor, right?

I agree with the course of action outlined (use JSON files, custom RL module for VE-MW) but I have no good ideas as to how this data should be shared between WE and VE (much less VE core).

We decided just to share it between VE-MW and WE for now, by putting a ResourceLoader module in core.

@Catrope: So if we put it into MW core, what about those non-trivial entries like at https://github.com/wikimedia/mediawiki-extensions-WikiEditor/blob/master/modules/jquery.wikiEditor.toolbar.config.js#L581-L643 ?

Change 197357 had a related patch set uploaded (by Alex Monk):
WIP - Load all of WikiEditor's special characters

https://gerrit.wikimedia.org/r/197357

Change 197478 had a related patch set uploaded (by Alex Monk):
Load special characters in Platform instead of SpecialCharacterDialog

https://gerrit.wikimedia.org/r/197478

Change 197478 merged by jenkins-bot:
Load special characters in Platform instead of SpecialCharacterDialog

https://gerrit.wikimedia.org/r/197478

Change 200184 had a related patch set uploaded (by Alex Monk):
Move WikiEditor's special character data and messages into core for use in other extensions

https://gerrit.wikimedia.org/r/200184

For reference, here is the script I used to generate the data to put in the core commit in the end:

var result = {};
$.each( $.wikiEditor.modules.toolbar.config.getDefaultConfig().toolbar.characters.pages, function ( groupId, characterGroup ) {
	result[groupId] = characterGroup.characters;
} );
JSON.stringify( result );

Change 200184 merged by jenkins-bot:
Move WikiEditor's special character data and messages into core for use in other extensions

https://gerrit.wikimedia.org/r/200184

Change 200249 had a related patch set uploaded (by Alex Monk):
Move WikiEditor's special character data and messages out to core

https://gerrit.wikimedia.org/r/200249

Change 197357 merged by jenkins-bot:
Load all of WikiEditor's special characters

https://gerrit.wikimedia.org/r/197357

Change 200249 merged by jenkins-bot:
Move WikiEditor's special character data and messages out to core

https://gerrit.wikimedia.org/r/200249

Thanks @TheDJ.

So I think we have all the previously-WikiEditor-only special character data which VE core can support now. There's some more complex entries which are just ignored for now, but I think they're very much the minority (79 out of 2223). Can we resolve this ticket anyway?

In T91608#1159718, @Krenair wrote:

Thanks @TheDJ.

So I think we have all the previously-WikiEditor-only special character data which VE core can support now. There's some more complex entries which are just ignored for now, but I think they're very much the minority (79 out of 2223). Can we resolve this ticket anyway?

I dug into the WikiEditor and VE core code and discovered that we can actually handle 70 of the remaining 79. Details in https://gerrit.wikimedia.org/r/200504

The Bangla character set is incomplete. Not only that, alternate character forms (for R and W) used in Assamese and Manipuri languages should also be there. I suggest incorporating the English Wikisource CharInsert set for Bengali characters. (available at https://en.wikisource.org/wiki/MediaWiki:Gadget-charinsert-core.js)

In T91608#1218308, @Hrishikes wrote:

The Bangla character set is incomplete.

See T96494. One problem per report only, please.