Jump to content

alp: A Python Workflow Module


Recommended Posts

I poked a bit more at the various options and can confirm once more that leaving the double quotes away is a bad idea. But the world isn't as simple as neither of us thought: Actually, the correct implementation seems to be to tick all escape options, use /bin/bash, escape the {query} with double quotes:

python yourscript.py "{query}" arg2 arg3 ...

and then unescape semicolon, round brackets, and space in the code. This at least works for all characters which Alfred offers to escape to begin with. I didn't test (and I'm not very interested in, because I lose testabiity that way) inline Python code.

Link to comment

Both, unescaping and decoding were features I introduced in python-alfred and which Daniel pulled trusting in my Python skills (and I do understand shell escaping as well, just didn't test it thoroughly enough). I've now tested that you don't need to (and hence shouldn't, so I'll remove that feature from alfred-python) unescape sys.argv when calling a Python script from the command line (choose /bin/bash and tick all escape options) via

python script.py "{query}"

and not, as you write, without the quotation marks.

{query} should NOT have quotes around it, i.e. use

python script.py {query} 

with all the escaping ticked. This works perfectly even when the query contains spaces and quotes. Can you give me any examples where this doesn't work?

 

You would have to do something slightly different if you use inline Python code, but I think that's a bad idea for various other reasons.

 

 

 

As to decoding, you definitely should move from UTF-8 to unicode to have an internally consistent handling of characters, and you should use u'string' literals throughout, and finally encode everything to UTF-8 again. The normalizing is necessary if you try and feed Python such strange things as my surname (Müller). The ü will end up basically as a UTF-8 version of the points followed by a u which is a Mac OS X thing and by no means a representation Python understands without normalizing.

Ah, thanks for that example. You're right, decoding does seem necessary.

 

 

Again, it eliminates the need to search the docs for the right formatting codes, which I can personally never remember, and standardizes the way timestamps are used as UIDs for the benefit of Alfred's sorting mechanism. Most of the module boils down to things that anyone could do on his or her own—in fact, all of it does; there's nothing totally "new" in here, as far as I can tell, just a collection of useful "old" tasks made simpler. E.g., yes, anyone could write the ctypes interface to the Keychain functions; the point is that now no one has to.

I think the keychain functions are definitely useful because it takes a fair amount of effort to read through the docs and figure out how it should be done, as well as fix any bugs along the way. The difference with timestamps is that I don't think it would be as frequently used (I might be wrong though), and it's built into Python. Getting a timestamp is just a one-liner. It is true, though, that it can take some time to figure out the right format.

Edited by johnjddoe
Link to comment

In my experience, guys, it makes sense to be agnostic about the use of quotation marks around {query} and how much escaping you want; there's no best practice to dictate what you "should" or "shouldn't" do. You might want one workflow to treat the user's space-delimited items as separate arguments, you might want another to treat it all as one complete string. I know that the workflow I'm trying to fix now, my scp workflow, works better if I treat every input as a separate argument to sys.argv[] and thus don't escape anything; others I've made use the quotation-mark scheme or more escaping. It really just depends on the case.

 

The timestamps are used—more frequently than the Keychain, in fact—to try to get Alfred to sort items in the order they're programmed. Setting the uid to a timestamp is an easy and often-effective way to do this. However, the timestamp has to be in a format like 2013-03-19-090700 in order to be sorted alphabetically—something that might not necessarily be obvious. So I think it's useful enough to warrant the five lines of code, in this single Alfred-related case.

Link to comment

{query} should NOT have quotes around it, i.e. use

python script.py {query} 
with all the escaping ticked. This works perfectly even when the query contains spaces and quotes. Can you give me any examples where this doesn't work?
 

Sure: Just use a single quote '. That doesn't even trigger the Python script without double quotes around {query}.

 

 

In my experience, guys, it makes sense to be agnostic about the use of quotation marks around {query} and how much escaping you want; there's no best practice to dictate what you "should" or "shouldn't" do.

Disagreed. I think the best practice is to take care that any string arrives in Python just as the user entered it in Alfred (straight from the horse's mouth). If you've got that worked out, you wouldn't want to change that any more. Of course, you're free to interpret that string at will and as it suits your specific application. Edited by nikipore
Link to comment

 

Sure: Just use a single quote '. That doesn't even trigger the Python script without double quotes around {query}.

I hadn't thought of that. It's strange that Alfred doesn't include options to escape single quotes or, for that matter, backslashes. Not sure if it's worth filing a feature request for it. Personally, I think it's nicest and cleanest to not need to use quotes at all, and Alfred should provide you with all the escaping you need.

 

So as far as I can tell, there is no way to currently properly handle all user input (at least not easily). The best compromise I can see is using double quotes, and escaping only backquotes, double quotes and dollar signs. This should handle everything except backslashes.

 

Disagreed. I think the best practice is to take care that any string arrives in Python just as the user entered it in Alfred (straight from the horse's mouth). If you've got that worked out, you wouldn't want to change that any more. Of course, you're free to interpret that string at will and as it suits your specific application.

I agree with your disagreement. :)

Link to comment

So as far as I can tell, there is no way to currently properly handle all user input (at least not easily). The best compromise I can see is using double quotes, and escaping only backquotes, double quotes and dollar signs. This should handle everything except backslashes.

I escape backlashes in Alfred and unescape them in python-alfred, and that approach also treats backslashes correctly. Which characters or character sequences are left which aren't treated correctly?
Link to comment

I escape backlashes in Alfred and unescape them in python-alfred, and that approach also treats backslashes correctly. Which characters or character sequences are left which aren't treated correctly?

Alfred doesn't have an option to escape backslashes, and double quotes are somewhat funky with backslashes. They work fine with backslash followed by an alphabetical character, but try using e.g.

\\

 

as your query. Then your script will be passed only a single backslash. For 99.9% of scripts, this won't matter, but it still annoys me. You could use single quotes around {query}, which pass the backslashes correctly, but since Alfred doesn't have an option to escape single quotes, then the query can't have any single quotes in it. So I think the best option is double quotes with some things escaped, as I mentioned in my previous post.

Link to comment

@johnjddoe: I can confirm that 2n-1 and 2n backslashes (n>0) within double quotes arrive in Python as n backslashes. Annoying indeed. I think we should file an issue to request a proper unescaping (if possible at all). Maybe the cleanest way out is to offer a streaming mode where the script harvests UTF-8 (either Mac style or normalized) on stdin and yields UTF-8 on stdout.

Link to comment

@johnjddoe: I can confirm that 2n-1 and 2n backslashes (n>0) within double quotes arrive in Python as n backslashes. Annoying indeed. I think we should file an issue to request a proper unescaping (if possible at all). Maybe the cleanest way out is to offer a streaming mode where the script harvests UTF-8 (either Mac style or normalized) on stdin and yields UTF-8 on stdout.

I've filed a feature request here. Not quite sure what you mean by the streaming mode. It is possible to run command line stuff using NSTask, where parameters aren't interpreted in any way at all. So if {query} was one of the parameters, there would be no need for escaping. However, it is a little harder to use and I'm not sure how to provide a good interface to allow that. But it shouldn't be necessary once proper escaping is implemented.

Link to comment

I've filed a feature request here. Not quite sure what you mean by the streaming mode. It is possible to run command line stuff using NSTask, where parameters aren't interpreted in any way at all. So if {query} was one of the parameters, there would be no need for escaping. However, it is a little harder to use and I'm not sure how to provide a good interface to allow that. But it shouldn't be necessary once proper escaping is implemented.

Thanks for the feature request. Not quite sure whether it's really possible to pass every conceivable string as a command-line parameter. By "streaming" I mean that Alfred could feed {query} on the stdin of the subprocess; streaming via stdout is the way the communication from the subprocess into Alfred is set up already. I'll maybe add the streaming part as an alternative suggestion to your feature request.

In Python, the communication on behalf on the subprocess would look as follows:

query = sys.stdin.read()

Up to decoding and normalizing the binary stream (which could then even be encoded as properly normalized UTF-8), that would be the whole story. Clean and easy, no escaping needed at all.

Link to comment

Hey fellow workers of flow! Sorry to resurrect an old thread, but I'm sort of curious to know how alp is working out for people. I've been a little shocked and overwhelmed by the response—it currently has nearly 60 watchers and a few forks on Github—so I wanted to touch base. It seems like it's stable as of now, meaning that I've used just about every feature in one or another of my own workflows without bumping into errors, but I wonder if there are other problems left over. Anything from bugs to inefficiencies, there's never a bad time to report them (or, better yet, submit new code on Github). And, failing slash on top of that, I'm interested in compiling a list of workflows that use alp for the Github repo and its section on my Alfred page, so if you've found it useful, let me know here, via PM, or at d at daniel dot sh and I'll add links to your work everywhere I can.

 

Thanks for the support and help, folks. This is a cool little community that's coalescing here.

Link to comment
  • 1 month later...

Trying to read a pList file from iTerms2 and I'm getting this error

 

 

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "alp/core.py", line 84, in readPlist
    return plistlib.readPlist(path)
  File "/usr/local/Cellar/python/2.7.4/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plistlib.py", line 78, in readPlist
    rootObject = p.parse(pathOrFile)
  File "/usr/local/Cellar/python/2.7.4/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plistlib.py", line 406, in parse
    parser.ParseFile(fileobj)
xml.parsers.expat.ExpatError: not well-formed (invalid token): line 1, column 8

Any idea why this happens? 

Edited by soulesschild
Link to comment

Hi there,

I'm honestly not sure, but I suspect that there's something wrong with the plist such that it's generating invalid XML. My guess, but it's only a guess, is that there's a UTF-8 character in the property list. You may have to use the codecs module to open the file, read it into memory, and then pass it to plistlib.readPlistFromString() by hand. I'll rewrite the alp code so that it does this—it's probably wise anyway—but give that a shot in the meantime and see if it does you any good.

 

ETA: I've just pushed a change to Github that may help, but I'm not positive. Give it a shot and see how it works. If you'd be willing to e-mail me the plist file (d at daniel dot sh), that'd be great too.

Edited by phyllisstein
Link to comment

Hi there,

I'm honestly not sure, but I suspect that there's something wrong with the plist such that it's generating invalid XML. My guess, but it's only a guess, is that there's a UTF-8 character in the property list. You may have to use the codecs module to open the file, read it into memory, and then pass it to plistlib.readPlistFromString() by hand. I'll rewrite the alp code so that it does this—it's probably wise anyway—but give that a shot in the meantime and see if it does you any good.

 

ETA: I've just pushed a change to Github that may help, but I'm not positive. Give it a shot and see how it works. If you'd be willing to e-mail me the plist file (d at daniel dot sh), that'd be great too.

 

I might make a pull request soon, I figured out the issue. Apparently the default python plistlib isn't that great on some plist's. However, if you use biplist, it imports it just fine. 

 

Then again, I'm just breaking the crap out of your library :D

Link to comment

Thanks! I'll take it under advisement, though I'll confess to being less than keen on requiring all alp users to install a new module.

 

Yea that's what I figured. Easy way is to actually just bundle the .py file to not make them install it but of course that increases the module size slightly ;)

Link to comment

Yea that's what I figured. Easy way is to actually just bundle the .py file to not make them install it but of course that increases the module size slightly ;)

 

Well, since both six and biplist were small and easy to tweak so as to be imported with alp, and since it seemed like it could come up again, I decided to go ahead and do it.

 

Meaning, as a general announcement, that alp now supports reading binary plists—no changes to implementation necessary.

 

I added a link, @soulesschild, to your github profile in the code; let me know if I can be more direct with crediting you for the change in any way.

Link to comment
  • 3 weeks later...

alp has been updated to incorporate the changes to the workflow backend in v2.0.4. Any arguments with a newline character will now be enclosed in <arg></arg> tags rather than arg="" attributes. Additionally, there have been some changes to the Requests module: you must now explicitly call a download method to grab data from the server—the init method no longer does this for you. This makes it easier to flush the cache if need be.

Link to comment
  • 8 months later...

Hi Phyllisstein,

 

I just wanted to point out that the NFC normalisation alp uses in alp.core.decode won't match data returned by Python's filesystem libraries (OS X uses NFD normalisation—more or less). Data passed as script arguments by Alfred will also be NFD-normalised, but I figure that's primarily what the decode function is aimed at.

 

I've no idea whether strings returned by filesystem libraries (NFD) are more important than string literals in Python code, but I thought it worth mentioning that there's a bug waiting to happen here. A filepath with ü in it returned by os.listdir, for example, won't match unicode literals in the code or input from Alfred (if it's been NFC-normalised, as is alp's default).

 

Using NFC normalisation (Python's default), ü is \xfc, using NFD, it's u\u0308. These will not match using ==. Oddly, ü inserted into a script via "{query}" will be u\xcc\x88…

Edited by deanishe
Link to comment

Ah! Thanks for the tip. This is extraordinarily lazy of me, but as I've sort of abandoned that particular darling of mine in a basket on the broader community's doorstep, could I ask you to fork the repository and submit a pull request? I can merge it and add a note in the README crediting you for the fix, but I don't personally have the spare cycles to dig into it myself.

Link to comment

I'll be perfectly honest, no. I don't use alp myself (I just thought I'd have a peek, having fixed a similar problem with nikipore's alfred.py), and thought I'd give you a heads-up on a potential problem. I'd rather not take any responsibility, even indirectly, for a lib I don't use.

 

The way I see it, if I'm the first person to mention this—and I don't even use alp—it's only a potential problem, not a real one, so changing it might cause more trouble than leaving it as it is. At most, I'd mention the potential problem in the README (there are valid arguments for both NFC and NFD normalisation).

 

That said, would you mind if I borrowed a couple of alp's modules for my own alfred library I'm considering building? I want something much more lightweight than alp (no bundling of stuff like requests and particularly no importing of everything in __init__.py), but I really like the look of your keychain module especially. It'd all be on GitHub. I'm aiming at something that can easily be added to a workflow as a git submodule for easy updating.

 

If you're alright with my borrowing a module or two, I'll update the README to mention the potential problem with filenames. How's that sound?

Edited by deanishe
Link to comment

Hah, no worries! I appreciate the frankness. I'll have to see if I can't find some time to correct (or, as you point out, not!) the error on my own. Please do scavenge whatever seems useful for your own module. Working the whole thing in as a submodule is smart; when I was working on alp, we were all trying to keep zipped workflows under 1MB, so deleting git's voluminous histories always made more sense, but I don't think that's a concern anymore. Good luck, and thanks again for the pointer!

Link to comment

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...