Leveraging x-callback-url in workflows

[chrisbro]

Hello! I started to work on creating my own workflow that would allow me to create and search Bear notes. I quickly ran into some problems that I don't think there are reasonable workarounds for, but I wanted to get some opinions first.

 

My problem is that Bear's extensibility comes via their implementation of the x-callback-url spec. I need to be able to call these URLs from within an Alfred workflow and parse the response. My first approach has been to use AppleScript...this works fine for creating a new note, as I don't need a response. 

set createUrl to "bear://x-callback-url/create?title=" & my urlencode(sTitle) & "&text=" & my urlencode(sText)
open location createUrl

However, I now want to leverage their search endpoint, which requires parsing the response. This doesn't seem doable from within Alfred. The AppleScript open location command does not provide a result - so I can call a URL, but not retrieve anything from it. Therefore, I can't leverage the search endpoint.

 

I tried getting clever by calling a shell script that then called curl, but curl errors out because it doesn't recognize the "bear://" protocol. I could do "open" from the shell, but that opens the app - not very useful when I just want to grab search results and put them into Alfred.

 

Any thoughts? Any other clever ways to do this? Given that this x-callback-url spec seems to be gaining traction, it might make a good feature request to add this inter-app messaging natively into Alfred as an action. 

 

Thanks!

[vitor]

Where you need to ask about this is on the Bear forums.

 

Your problem isn’t in accessing Bear with Alfred; it’s in accessing Bear with a command-line interface. Solve that, and you’ve solved it in Alfred.

 

16 minutes ago, chrisbro said:

might make a good feature request to add this inter-app messaging natively into Alfred as an action.

 

I’ve been on these forums every day for years and don’t remember a similar request, so I doubt it’d be worth the time to implement in Alfred. Especially since x-callback-url is a crappy and hacky solution and it’s insane to do it on macOS. Most apps don’t do it for good reason. For as much as we complain about AppleScript, an AppleScript dictionary is still miles ahead of an x-callback-url in terms of experience.

Edited June 13, 2017 by vitor

[chrisbro]

15 minutes ago, vitor said:

Especially since x-callback-url is a crappy and hacky solution and it’s insane to do it on macOS. Most apps don’t do it for good reason. For as much as we complain about AppleScript, an AppleScript dictionary is still miles ahead of an x-callback-url in terms of experience.

 

That's fair. Thank you for your time.

 

Anyone else have ideas on how to get responses back from arbitrarily-protocoled HTTP requests in a way that Alfred can parse it? Just saw this ("Fetching Data from the Web") and will give that a go next.

[deanishe]

I think you may have misunderstood how x-callback-url works. It isn't HTTP. It is implemented by iOS/macOS, and the system handles the calls.

 

An app registers a URI scheme, e.g. bear://, and when you call such a URI via macOS, it directs the call to the registered application.

 

As I understand it, you also do not get a response from the application. What you do instead is pass an x-callback-url for the application to call with the result, i.e. it calls your app back with the result.

 

As such, you need to create your own app that registers your own URI scheme.

[chrisbro]

Just now, deanishe said:

I think you may have misunderstood how x-callback-url works. It isn't HTTP. It is implemented by iOS/macOS, and the system handles the calls.

 

An app registers a URI scheme, e.g. bear://, and when you call such a URI via macOS, it directs the call to the registered application.

 

As I understand it, you also do not get a response from the application. What you do instead is pass an x-callback-url for the application to call with the result, i.e. it calls your app back with the result.

 

As such, you need to create your own app that registers your own URI scheme.

 

Thanks for the response! I'm definitely misunderstanding, it looks. I was getting thrown off because I can hit that x-callback-url create endpoint in the code snippet in my first post, and it works just fine - it does indeed create a note like I'd expect. I just have no way to get at the callback response from AppleScript. My original assumption is that I was looking for a normal ol' synchronous response, not a callback, because that's what I get for trying to use my brain in the middle of the night when I can't sleep. So this all makes sense now.

 

@deanishe it looks like I can't call back to Alfred this way, though? I see a little conversation here but can't make out if Alfred has a callback endpoint I can leverage. 

[deanishe]

No, you can't use Alfred's URI scheme. As it says in that thread, it's for importing/exporting web searches. It's not connected to workflows in any way.

 

In theory, @Andrew could add the ability to call workflows' External Triggers via the URL scheme, but as @vitor said, it's a sucky way to do automation on macOS. It's a nasty hack for iOS, where there is no real system for inter-app communication. macOS does have one.

 

Unfortunately, an awful lot of iOS developers don't bother to implement an AppleScript dictionary in their macOS versions (it is a lot of work, to be fair), leaving you with the ghetto IPC that is x-callback-url.

 

You can use the xcall program to get a response from an x-callback-url call, but it probably won't work very well in many cases: calling an x-callback-url tends to activate the associated application, which will also hide Alfred.

 

If you're after showing search results in Alfred, you'll probably have to re-open Alfred yourself by calling an External Trigger in your own workflow.

[chrisbro]

@deanishe I really appreciate the explanation - makes complete sense. Thank you!

[deanishe]

Looking at the docs for the bear:// URL scheme, the search endpoint doesn't actually provide a way to get search results from the app (i.e. there's no x-success parameter to return anything). All it does it show the search results in the app itself.

 

If the app doesn't have an AppleScript dictionary, your only realistic option is to access its datastore directly. If it's based on CoreData, which is likely, it's using sqlite3 under the hood, which means you can pull data straight out of its datastore, like this OmniFocus workflow does.

 

The advantages of that method are that it doesn't activate the application and it's stupendously fast compared to AppleScript. Like, insanely fast. The disadvantages are that it's fairly complicated to do (compared to using AppleScript) and the database layout may change at any time, breaking your workflow.

 

Edited June 13, 2017 by deanishe

[chrisbro]

Ooo, intriguing - I'd actually be way more comfortable with that approach given my development background. I'm going to give that a spin next.

[dfay]

Look at the work Rob Walton (and I, in lesser capacities) have done with the Ulysses workflow : 

A lot of the challenges are similar - the solution there relies on parsing the library for some parts of the workflow and using the URL scheme for others.

 

(Incidentally I was in the early Bear betas and ended up moving everything to Ulysses when I ran into the limits of hashtag-based organization--haven't looked back).

[dfay]

Also - writing a callback handler in AppleScript and registering it is pretty easy - I have one that lets me create a new Ulysses sheet then pass the URL back to BibDesk to attach to a publication record , & I'm happy to share it and the links I used to figure it out if you're interested.  I'd expect that Bear's URL scheme will eventually provide callback info & this may yet be an option.

[chrisbro]

@dfay I'd appreciate that, thank you! That way I can refer back to this post if they ever release that capability.

 

Update if anyone's interested: looks like they have a pretty straightforward sqlite3 schema, so this shouldn't be anything crazy to make a workflow for.

[dfay]

These were the directions I followed:

 

https://yourmacguy.wordpress.com/2013/07/17/make-your-own-url-handler/

 

and here's the script that I used in the handler (complete with all my commented-out testing lines):

 

on open location theURL
    -- set theURL to "sourceapp://x-callback-url/success?targetId=jsvEPFa6vvq3vDmrAt4KGA&targetURL=ulysses://x-callback-url/open?id%3DjsvEPFa6vvq3vDmrAt4KGA"
    
    --    display dialog theURL
    
    set x to the offset of "ulysses" in theURL
    set y to the offset of "?id%3D" in theURL
    
    --    display dialog y
    
    set theNewSheet to text from x to y of theURL & "id=" & text from (y + 6) to -1 of theURL
    
    --    display dialog theNewSheet
    
    tell application "BibDesk"
        set thePublications to the selection of document 1
        repeat with thePub in thePublications
            add theNewSheet to thePub
        end repeat
    end tell
    
end open location

 

and here's the script for the workflow itself:

 

tell application "BibDesk"
	activate
	-- without document, there is no selection, so nothing to do
	if (count of documents) = 0 then
		beep
		display dialog "No documents found." buttons {"•"} default button 1 giving up after 3
	end if
	
	set thePublications to selection of document 1
	
	repeat with thePub in thePublications
		export document 1 using template "exportMarkdown" to clipboard for thePub
		set Markdown_link to ""
		set CiteKey to cite key of item 1 of thePub
		set ItemURL to "x-bdsk://" & CiteKey
		--			set itemName to title of item i of thePub
		set theWikiLink to "[" & CiteKey & "]" & "(" & ItemURL & ")"
		set Markdown_link to (Markdown_link) & theWikiLink
		set theText to ("# " & (the clipboard) & Markdown_link)
		
		set theKeywords to get keywords of thePub as string
		set theKeywords to my replace_chars(theKeywords, " ", "_")
		set theHashtagKeywords to "#" & my replace_chars(theKeywords, ",_", " #")
		
		set theTemplate to "
Date: " & my todayISOformat() & "
Keywords: " & theHashtagKeywords
		
		set theCommand to quoted form of ("ulysses://x-callback-url/new-sheet?x-success=ulbd://x-callback-url/success&text=" & theText & theTemplate)
		do shell script "open " & theCommand
	end repeat
end tell

-- -- --  Helpers -- -- --

on todayISOformat()
	set theDate to current date
	set y to text -4 thru -1 of ("0000" & (year of theDate))
	set m to text -2 thru -1 of ("00" & ((month of theDate) as integer))
	set d to text -2 thru -1 of ("00" & (day of theDate))
	return y & "-" & m & "-" & d
end todayISOformat

on replace_chars(this_text, search_string, replacement_string)
	set AppleScript's text item delimiters to the search_string
	set the item_list to every text item of this_text
	set AppleScript's text item delimiters to the replacement_string
	set this_text to the item_list as string
	set AppleScript's text item delimiters to ""
	return this_text
end replace_chars

Most of it's getting stuff ready in BibDesk but the line "set theCommand...." is the important one -- it sets the x-success to return a ulbd:// URL which is what the handler is registered for.

 

It's kind of a kludge since it's really only designed for this one scenario, rather than processing the callback and doing different things with it but it should give the overall idea.

 

Edited June 14, 2017 by dfay

[chrisbro]

@dfay Thanks again! And to anyone interested, I've gotten some basic search functionality working by heavily borrowing from the Omnifocus workflow. Never worked with Python before so it's a bit slow going but nothing too tough.

[chrisbro]

Got the first version up, post is here. 

[robwalton]

Hi @dfay, have been out for a while. How goes it?

On 14/06/2017 at 3:29 AM, dfay said:

Also - writing a callback handler in AppleScript and registering it is pretty easy - I have one that lets me create a new Ulysses sheet then pass the URL back to BibDesk to attach to a publication record , & I'm happy to share it and the links I used to figure it out if you're interested.  

 

I had an attempt at packaging up something to make x-callback-url calls and then receive the callbacks. You might try https://github.com/martinfinke/xcall from the command line. I've put a python wrapper around it here: https://github.com/robwalton/python-xcall . I've not really used it in earnest, or checked it for race conditions or threads safety and whatnot. But it might help and it has tests so we might be able to improve it if you come across any issues! It may also be total overkill, if you've found a lighter way @dfay. I wanted something signed with apple that could be distributed easily.

[dfay]

I haven’t done any more work on the Ulysses workflow - but I’ve been using it many times a day with no problems - until earlier today, actually.  I had to reset LaunchServices to clear up some Quicklook problems (cf. 

 ) and it broke my custom url handler as a result.  Fortunately reopening the applet directly from /Applications re-registered it and got it working fine again.