Automated Testing of Workflow

[Eli Jordan]

Hi

 

I'm pretty new to Alfred, I've only been using it for a few weeks. Its a great productivity tool, and I really like that it has a plugin model (i.e. workflows).

I have started writing a workflow here https://github.com/eli-jordan/alfred-jenkins (it's not published yet, since I'm still working on it).

However, I've found it pretty frustrating to have to constantly manually run a set of "regression" tests on the code when I make changes.

 

Has anyone tried to write automated tests that invoke alfred and verify the items the are returned?

If not, how do people test the integration points between their code, the config in the Alfred workflow UI and Alfred itself?

 

 

[deanishe]

9 hours ago, Eli Jordan said:

Has anyone tried to write automated tests that invoke alfred and verify the items the are returned?

 

You can't, really. Alfred's output only goes to its own UI. But as workflows are basically command-line programs that output JSON, it's not too hard to call the binary/script with the appropriate CLI arguments and environment variables and check that it's returning the expected JSON. I don't think anyone actually does that, though.

 

For my part, I tend to put non-UI stuff in libraries with their own unit tests. A workflow's frontend is usually simple enough that it's not a problem to test manually.

 

Your workflow appears to include its own mini library for interacting with Alfred. In a statically-typed language, that’s something I’d put in a separate library/package with its own unit tests. Same goes for the caching and the code that interacts with Jenkins.

 

If you decouple those parts from the application, there shouldn't be too much complexity left.

 

Here’s my Go library for workflows. That contains all the code for caching, storing credentials in Keychain, generating JSON and otherwise interacting with Alfred, and it has a whole bunch of unit tests. With all that stuff moved elsewhere, and well-tested, there isn't so much that needs testing in an actual workflow.

Edited July 24, 2020 by deanishe

[Eli Jordan]

Quote

You can't, really

 

In the world of web application, there are tools such as selenium that allow you to create automated UI tests. I guess I was imagining that you might be able to do something similar with apple script to interact with the Alfred UI.

 

Quote

Your workflow appears to include its own mini library for interacting with Alfred.

 

Regarding splitting out libraries and adding unit tests. Thats something I'm planning on doing, I'm working on unit tests now. I'll split out a library at some point, maybe when I start developing another workflow. I have also created a plugin for sbt (Scala's main build tool) to package workflows, which is something I haven't seen in the other workflows I have looked at.

 

Quote

 Here’s my Go library for workflows

 

 

Yep, I've seen that. I used a lot of the ideas I saw in that library and the python equivalent when bootstrapping this workflow. Being able to copy patterns that work was super valuable 

 

A big part of the reason for writing this workflow was to get a chance to try out Graal native-image which allows compiling JVM applications into native binaries. So, I had to build from scratch rather than using one of the existing libraries.

Edited July 24, 2020 by Eli Jordan

[deanishe]

57 minutes ago, Eli Jordan said:

In the world of web application, there are tools such as selenium that allow you to create automated UI tests. I guess I was imagining that you might be able to do something similar with apple script to interact with the Alfred UI.

 

Not really realistic, imo. UI scripting is notoriously unreliable (you're basically just blindly firing asynchronous events at an application), and you'll probably end up spending more time debugging the testing code than the code you're actually trying to test. You can use Alfred's External Triggers to drive your workflow, but they're relatively limited compared to workflow-internal code (you can't pass in variables via AppleScript, for example).

 

57 minutes ago, Eli Jordan said:

Being able to copy patterns that work was super valuable 

 

Thanks. No wonder I liked the design of your workflow so much

 

I'd recommend the Go library as the better one to crib. It has a lot of improvements over the older Python "prototype". It's probably more relevant to Scala, too.

 

The JSON feedback-generation code is probably heavier-duty than it need be: it dates back to when Alfred only accepted XML, which was a much more complicated format that needed to be abstracted away.

 

57 minutes ago, Eli Jordan said:

A big part of the reason for writing this workflow was to get a chance to try out Graal native-image which allows compiling JVM applications into native binaries.

 

I noticed that. I've been reading a bit about Graal recently, and it looks really interesting. I'll have to give it a try. (Perhaps using Clojure, so I finally get around to learning a Lisp.) The JVM is a poor fit for workflows in general, imo, because startup time is critical for workflows given the way Alfred runs them.

 

Edited July 24, 2020 by deanishe

[Eli Jordan]

Quote

Not really realistic, imo. UI scripting is notoriously unreliable

 

I gave this a shot, just to see if its possible. However, from what I could tell the items in the Alfred UI list are not accessible through "System Events" and the accessibility API. So, I don't think its even possible. 

 

I'm not sure if any of the developers of Alfred monitor this forum, but it would be interesting to get their feedback on whether this is something that is possible.

[deanishe]

4 hours ago, Eli Jordan said:

I'm not sure if any of the developers of Alfred monitor this forum, but it would be interesting to get their feedback on whether this is something that is possible.

 

There's only one developer, but the Alfred team reads pretty much every post.

 

I don’t think that approach would work. One of Alfred’s main features is that it learns from your behaviour, so it changes the order of results. As such, unless you disable that feature (which you generally shouldn’t because it’s awesome), the results may or may not be in the order your workflow emitted them.

 

Alfred just isn't set up to be used that way. You'd have to quite severely restrict the features you use to get Alfred to behave predictably enough to run automated tests on it. Any meaningful tests would be unfixably non-portable because people do things like turn off subtitles. You can forget about calling via keyword because there's no guarantee that only your workflow will be run.

 

Your best bet is probably to try to do as much in code as possible and minimise the number of workflow elements you use, then forget about Alfred and test the binary directly.