Use symlinked alias file in distributed workflow?

[smarg19]

I'm wondering if someone else has tried this or does this? I want to clean up my Workflows, and the main way I'm thinking to do this is to symlink files that I use in multiple workflows (like the Alfred-Workflow library and a utils.py file I've made). While I know that this will allow the workflows to work on my machine, I'm not certain how things will work when I share a workflow with symlinked files in it. 

 

Has anyone distributed a workflow with symlinked files?

[rice.shawn]

I'm pretty sure that the symlink will exist persist but fail if the target isn't the same, which means that if you're symlinking to anything in a user's home directory, it will fail. You could just reconstruct the symlink on a "first-run" or you could implement the bundler (which is the point of the bundler).

 

Re: bundler, as you know, we're still waiting on the python version to be feature-complete before releasing Taurus. When that gets finished (and it's out of my skillset because it's python), it seems like adding 'utils.py' as a standard library would be a good idea.

[smarg19]

I've decided to go another route. I've written a quick Python script that overwrites all instances of a certain file with a canonical version. This way, I ensure that all of my workflows are using the most up-to-date version of certain key libraries (Alfred-Workflow, Alfred-Bundler, utils.py, etc), but the actual files are all real, so it's easy as pie to package and share the workflow via Packal. 

 

Also, if you post some specific target goals on GitHub for the Python Bundler, I'll give it a look. It's been a while since I looked at @deanishe's code, but from what I remember it was super well-documented and easy to follow. I also think adding utils.py could be helpful. I passed the idea of adding it to Alfred-Workflow by @deanishe, but he wasn't sure it would be worth the added bloat. But I use it in all of my workflows. It has a number of functions to simplify reading and writing to files (and directly to JSON), a collection of functions to make using AppleScript in Python a total breeze, functions to interact with Alfred, a simple but helpful mdfind wrapper for searching a user's drive, and convenience functions for working with the system clipboard (in UTF-8 encoding ). It's sort of a grab-bag, but it really helps with development, since I use these functions all of the time, but now I don't need to keep rewriting them. 

[deanishe]

Re Alfred-Workflow:

 

To be clear, I thought some of the functions were generally useful (e.g. the clipboard ones) and others were too trivial to include in the library (e.g. the JSON reading/writing ones).

I don't know about you, but I've had problems with the encoding in  pbcopy and pbpaste since I upgraded to Yosemite. I haven't yet figured out how to fix the encoding issues. As a result of that and the fact that Mavericks and earlier worked in a different way, and I don't have versions of those to test the code with, I'm not super-keen on trying to implement a clipboard library personally, even though I really like the idea. It's very much in line with the philosophy of making common, but somewhat complex, operations simple.

 

Re symlinking:

I'd be very careful about symlinking to stuff outside of your workflow. And by "very careful" I mean, "don't do it without a truly compelling reason".
 
If the symlink points to an non-existent file, your workflow is broken. So you need to include code in every workflow to check for the existence of the linked file and inform the user that it's missing and that (and how) they need to install it. And probably code to install it for them if you want your workflow to work almost as well as if you'd just included the library.

 

And what do you do when a bug crops up that breaks one of your workflows but the fix breaks the others? By using one common library, your workflows are now unnecessarily dependent on one another.

 

And even if you update them all simultaneously, what about the user that's only updated one of them?

It'd make much more sense, IMO, for you to combine all your shared functions into one library. If the library makes sense outside of your own workflows, upload it to PyPi, from where you can install and update it with pip. If it isn't that general, you can put it on GitHub, from where you can also install and update it with pip.
 
Or you can just write a script on your own machine that goes through your workflows' source directories and copies the latest version of your library into them.

 

Personally, I include Alfred-Workflow in my workflows using pip, even though I usually have a newer, better version on my HD. It's good dogfooding practice (for me as the developer). And I also do the same with other libraries. Were a workflow to require a large number of external libraries, I would write a script that installed and/or updated them (via pip or from the local filesystem or whatever).

 

Unless the code/stuff you need to install is massive (say 10+ MB), it belongs in the workflow, not symlinked to some other place that may or may not exist and is, in any case, just something else that can go wrong.

 

If you're installing 10+MB libraries, you should consider the bundler.

 

It is, essentially, just another external dependency with all the problems that entails, but it's most likely a better tested and more featureful version of any partial solution you come up with yourself.

 

(I know you wrote one of the bundlers, as did I, but they both rely on Shawn's well-tested backend code.)

[rice.shawn]

(I know you wrote one of the bundlers, as did I, but they both rely on Shawn's well-tested backend code.)

 

Just to point out a bit of cred, @deanishe, you did make me write the tests, and, thus, it is well-tested.

 

 

Re: pbcopy/pbpaste encoding problems.

 

I have two machines that are still running Mavericks as well as one running Yosemite, so if you want me to help test, I can. Also, if you really need to do testing on your machine, you can always create a VM on VMware or VirtualBox. You should be able to do that just from the "Install Mavericks" application on the app store.

[smarg19]

Re Alfred-Workflow:

 

To be clear, I thought some of the functions were generally useful (e.g. the clipboard ones) and others were too trivial to include in the library (e.g. the JSON reading/writing ones).

I don't know about you, but I've had problems with the encoding in  pbcopy and pbpaste since I upgraded to Yosemite. I haven't yet figured out how to fix the encoding issues. As a result of that and the fact that Mavericks and earlier worked in a different way, and I don't have versions of those to test the code with, I'm not super-keen on trying to implement a clipboard library personally, even though I really like the idea. It's very much in line with the philosophy of making common, but somewhat complex, operations simple.

I haven't had any problems. But I also do alot of minor magic to deal with encoding. The functions I use are: 

def to_unicode(text, encoding='utf-8'):
    """Convert `text` to unicode"""
    if isinstance(text, basestring):
        if not isinstance(text, unicode):
            text = unicode(text, encoding)
    return text

def set_clipboard(data):
    """Set clipboard to [font=courier new,courier,monospace]data[/font]"""
    os.environ['__CF_USER_TEXT_ENCODING'] = "0x1F5:0x8000100:0x8000100"
    text = to_unicode(data)
    proc = subprocess.Popen(['pbcopy', 'w'], stdin=subprocess.PIPE)
    proc.communicate(input=text.encode('utf-8'))


def get_clipboard():
    """Retrieve data from clipboard"""
    os.environ['__CF_USER_TEXT_ENCODING'] = "0x1F5:0x8000100:0x8000100"
    proc = subprocess.Popen(['pbpaste'], stdout=subprocess.PIPE)
    (stdout, stderr) = proc.communicate()
    return to_unicode(stdout)


set_clipboard('hëllœ wõrld')
x = get_clipboard()
print(type(x))
print(x.encode('utf-8'))

This works perfectly fine for me on Yosemite.

And you're right, the JSON stuff is basically unneeded with Alfred-Workflow's new serializer functionality. The AppleScript and clipboard stuff is probably the most useful (converting Python lists to AS lists is sooo helpful the one time you need it).

 

Re symlinking:

I'd be very careful about symlinking to stuff outside of your workflow. And by "very careful" I mean, "don't do it without a truly compelling reason".

 

If the symlink points to an non-existent file, your workflow is broken. So you need to include code in every workflow to check for the existence of the linked file and inform the user that it's missing and that (and how) they need to install it. And probably code to install it for them if you want your workflow to work almost as well as if you'd just included the library.

 

And what do you do when a bug crops up that breaks one of your workflows but the fix breaks the others? By using one common library, your workflows are now unnecessarily dependent on one another.

 

And even if you update them all simultaneously, what about the user that's only updated one of them?

It'd make much more sense, IMO, for you to combine all your shared functions into one library. If the library makes sense outside of your own workflows, upload it to PyPi, from where you can install and update it with pip. If it isn't that general, you can put it on GitHub, from where you can also install and update it with pip.

 

Or you can just write a script on your own machine that goes through your workflows' source directories and copies the latest version of your library into them.

 

Personally, I include Alfred-Workflow in my workflows using pip, even though I usually have a newer, better version on my HD. It's good dogfooding practice (for me as the developer). And I also do the same with other libraries. Were a workflow to require a large number of external libraries, I would write a script that installed and/or updated them (via pip or from the local filesystem or whatever).

 

Unless the code/stuff you need to install is massive (say 10+ MB), it belongs in the workflow, not symlinked to some other place that may or may not exist and is, in any case, just something else that can go wrong.

 

If you're installing 10+MB libraries, you should consider the bundler.

 

It is, essentially, just another external dependency with all the problems that entails, but it's most likely a better tested and more featureful version of any partial solution you come up with yourself.

 

(I know you wrote one of the bundlers, as did I, but they both rely on Shawn's well-tested backend code.)

I agree. I've figured it out. This was primarily aimed at keeping Alfred-Workflow up-to-date in all of my workflows, both for my own development and for packaging and sharing. I wrote a simple Python script to overwrite Alfred-Workflow with a canonical version to keep it up to date.

Edited November 26, 2014 by smarg19

[deanishe]

Good work with the clipboard functions.
 
A couple of observations:

• If you're using Alfred-Workflow, isn't Workflow.decode() enough (i.e. why do you need to_unicode())? Workflow.decode() will also normalise the Unicode for you.
• In os.environ['__CF_USER_TEXT_ENCODING'] = "0x1F5:0x8000100:0x8000100", the 0x1F5 is your user id, so that will only work properly for the first user on the system (uid 501). os.environ['LANG'] = 'en_US.UTF-8' would be more portable. Or you need to replace the 0x1F5 with hex(os.getuid()).upper().

[smarg19]

Good work with the clipboard functions.

 

A couple of observations:

• If you're using Alfred-Workflow, isn't Workflow.decode() enough (i.e. why do you need to_unicode())? Workflow.decode() will also normalise the Unicode for you.
• In os.environ['__CF_USER_TEXT_ENCODING'] = "0x1F5:0x8000100:0x8000100", the 0x1F5 is your user id, so that will only work properly for the first user on the system (uid 501). os.environ['LANG'] = 'en_US.UTF-8' would be more portable. Or you need to replace the 0x1F5 with hex(os.getuid()).upper().

 

1. I originally wrote the utils script to be Workflow independent, thus the port. 

 

2. Good call. I wasn't aware of the user-specific prefix. I will text the os.environ['LANG'] = 'en_US.UTF-8' solution.