JSON Output & Variables with Emojis - Encoding Woes

[Jasondm007]

I’m struggling to get a script filter to work which outputs a variable containing a mixture of text and emojis. In the output, Alfred just spits out empty characters for the emojis (⍰ question mark in a box). There’s obviously an encoding issue I’m overlooking, and I haven’t been able to quite put my thumb on it.

To explain, I’ve created a script filter with AppleScript that’s supposed to let the user select a playlist from their Apple Music library (to play it, move it, etc). I recently moved over from Spotify to Apple Music, and I have a ton of playlists (put that's a separate problem - ha)! And, as you might guess, some of my playlists include emojis.

When I run the script filter, Alfred’s output just shows empty characters where the emoji’s used to be. Consider the following example:

 

• Playlist: Radiohead ⭐ Faves
• Script Filter/Alfred Output: Radiohead ⍰ Faves
• Debug: Radiohead \u2b50 Faves

 

So, while most of this is over my head, from what I understand, my JSON output is being spit out as formatted for C/C++/Java Src (when it should be in the usual utf8 unicode format). Any suggestions for fixing it? I've never tinkered around with JSON.scpt file that does all of the heavy lifting, so hopefully it's just a minor issue with my code. Admittedly, I’ve always been confused about how JSON and AppleScript work together. Although I’ve been able to create script filters in the past thanks to everyone’s help, I’ve never made one that dealt with text that containing emojis. Thanks for any help you can lend!

 

For testing purposes, I’ve uploaded a test workflow here: https://share.getcloudapp.com/xQuDN0dl. And, here's screenshot of its script filter:

 

[deanishe]

I don’t think AppleScript uses UTF-8. It’s MacRoman by default. You could try retrieving playlist names with _playlist's name as Unicode text (though I think that's actually UTF-16), but the simple answer is “don’t use AppleScript”. It doesn’t support JSON natively, and that library you’re using is really buggy. Use JXA instead. It has real JSON support.

Edited July 7, 2020 by deanishe

[Jasondm007]

@deanishe Thanks for getting back to me. While I might not have done it correctly, before posting this, I tried adding "Unicode" to the text references in the script above. However, it didn't fix the problem. I might be totally wrong about it encoding an issue?

 

Oddly enough, the emojis and everything seem to show up just fine in Script Editor.

 

Thanks!

[deanishe]

5 hours ago, Jasondm007 said:

I tried adding "Unicode" to the text references in the script above. However, it didn't fix the problem.

 

Yeah, like I said, I think that’s actually UTF-16, which is no good. It’s something like «class utf8» for UTF-8.

 

5 hours ago, Jasondm007 said:

Oddly enough, the emojis and everything seem to show up just fine in Script Editor.

 

It's not odd, tbh. The problem is the JSON you're generating isn't correct because the library you're using isn't very good.

 

Just use JavaScript:

function run() {
	const app = Application('Music')
	let playlists = ObjC.unwrap(app.sources['Library'].userPlaylists)

	let items = []
	for (let i = 0; i < playlists.length; i++) {
		let p = playlists[i]
		items.push({
			title: p.name(),
			uid: p.name()
		})
	}

	return JSON.stringify({items: items})
}

 

[vitor]

Unfortunately, JXA’s utf-8 support isn’t great either, at least when reading files (note the case here). You have to use the Objective-C bridge to get accurate results. I’m guessing no one works on JXA or AppleScript, at Apple. A single engineer working full-time on these could have fixed most bugs and made them better languages.

[Jasondm007]

@deanishe Thanks a ton for taking a crack at it. I really appreciate it. As you would expect, you script works perfectly!

 

And, @deanishe & @vitor your comments were also really helpful, too. I'm going to try and wrap my head around them, and get back to you. At the end of the day, it's obvious that I just need to start getting better with other languages 🤦‍♂️. 

 

Thanks again!

[deanishe]

5 hours ago, vitor said:

Unfortunately, JXA’s utf-8 support isn’t great either

 

JXA is a pretty big mess, imo, but if you need to generate JSON, it beats AppleScript hands-down.

 

4 hours ago, Jasondm007 said:

At the end of the day, it's obvious that I just need to start getting better with other languages 🤦‍♂️. 

 

If you're coding in AppleScript, then absolutely. It's an awful language. JavaScript is not a good language, either, and JXA is a particularly awful flavour of it.

 

You should seriously consider learning a language like Ruby or Python before coding in AppleScript and/or JavaScript gives you too many absolutely bonkers ideas about programming.

[Jasondm007]

@deanishe & @vitor Yeah, I hear you. I keep trying to peck at the other languages, but have struggled. If I told you how much time I've spent on these (and other terrible) workflows, you'd laugh. As you can tell, I don't work in tech and barely understand AppleScript. Baby steps! Ha

 

While I'm obviously not familiar with these things, I still find it somewhat surprising there isn't a better library/dictionary for AppleScript to use for purposes of JSON outputs. The one that I relied on in the example - which is also used by a lot of other Alfred workflows - hasn't been maintained for years. In fact, its creator commented a few years ago that he was surprised people were still using it.

 

In any case, thanks to @deanishe & @vitor's help, I created another test workflow. For those who might be interested, one of the script filters allows you to play any playlists in your library and the other allows you to move a selected playlist to any playlist folders you might have created. The later is super helpful if, like me, you hate dragging things around or right-clicking a ton of times. The former runs a little slow on my machine, but that's probably more due the mess of playlists I have in my library than anything. 

 

Click here to download: https://share.getcloudapp.com/NQugBAdO 

 

 

PS: @deanishe I'm sorry for sending the (JavaScript) script filters to AppleScript. 🤦‍♂️

[deanishe]

17 minutes ago, Jasondm007 said:

I don't work in tech and barely understand AppleScript

 

Almost nobody understands AppleScript. It's an insane language and it's profoundly incapable. It lacks the most rudimentary features that every other language ever has (like the ability to generate valid JSON, for example).

 

Broadly speaking, you're just making your own life more difficult by using it (because it can't do anything).

 

It's the programming equivalent of trying to build things with half-melted Duplo blocks.

[vitor]

6 hours ago, Jasondm007 said:

I still find it somewhat surprising there isn't a better library/dictionary for AppleScript to use for purposes of JSON outputs.

 

Don’t be surprised. The reason is simple:

 

5 hours ago, deanishe said:

Almost nobody understands AppleScript. It's an insane language and it's profoundly incapable.

 

I’ll stop the quote there, but @deanishe’s full post is spot-on. We’ve been telling you to let go of AppleScript for a reason.

Its syntax tries to solve the wrong problem. Looking like written english doesn’t make it simpler, because it’s as rigid as any other language. AppleScript deviates from other languages (making it harder to pick up) and gives you nothing in return. The fact that it can be used to interface with GUI apps has nothing to do with the language itself; they could’ve chosen any other language to give you that ability, and eventually they did (JavaScript).

 

6 hours ago, Jasondm007 said:

As you can tell, I don't work in tech and barely understand AppleScript. Baby steps!

 

You seem to have this notion that AppleScript is ground zero, a simple language that if you can’t grok you won’t be able to move on to better things. That is absolutely wrong and you’re crippling your own progress. Not sure how to make that clearer.

 

You keep insisting on using AppleScript and we’ll keep insisting you get out of it.

 

6 hours ago, Jasondm007 said:

I keep trying to peck at the other languages, but have struggled.

 

Because you keep clinging to AppleScript. By continuing to do so, you’re wasting your time, because whatever you’re learning isn’t easily transferable, and you’re already feeling that.

 

Invest in something else. It doesn’t really matter which language, as long as it’s relatively popular. Being popular means more people rely on it, so more people make tools and libraries to improve their usage of the language and more people discuss it which means you have more resources to learn, understand, and fix your problems. It also means it’s more likely to share concepts with other programming languages. Learn a decent language, and your knowledge will be transferable.

Edited July 9, 2020 by vitor

[deanishe]

2 hours ago, vitor said:

By continuing to do so, you’re wasting your time, because whatever you’re learning isn’t easily transferable, and you’re already feeling that.

 

This so much. If you know any "mainstream" language, that knowledge is completely transferable to every other language because they're all basically some variation on the C model. It's common for developers to know 5 or 6 languages because they're all very similar. Any new language you look at is just a novel combination of features cribbed from other languages.

 

If programming languages were real languages, AppleScript would be Klingon. Very difficult to learn because it's completely different from everything else, and almost entirely useless.

Edited July 9, 2020 by deanishe