Ignore this post if you’re reading it on a Mac.
Either I was installing it wrong previously, or Microsoft fixed it in 1.4, and I was using 1.3, but finally I got Microsoft Keyboard Layout Creator to install and run in Windows 7, and I used it to build a custom German/English “typographer’s keyboard” layout. (My physical keyboard has a standard German layout.) This enabled me to add a bunch of extra punctuation and some archaic stuff such as ſ.
I also managed to add two dingbats from the Unicode Zapf Dingbats subset range, U+2766 and U+2767.
If you can see those, you are seeing them in pure Unicode. Windows is switching fonts somewhere to display them.