Jump to content

Packaging Python workflows, would a setuptools command be useful?


Recommended Posts

There are currently several methods for packaging workflows, all with advantages and disadvantages of course.

For example:

It seems to me that there is a better solution though. The setuptools package is widely used within the Python community to compile, build and deploy packages to the PyPI servers but it can do more. The Sphinx-PyPI-upload-2 package for example (https://github.com/WoLpH/sphinx-pypi-upload) makes it possible to upload Sphinx documentation to the PyPI servers which can be built using the commands from Sphinx itself.

With that regard I propose a new solution for packaging Alfred workflows:

Edited by wolph
Link to post

For what it's worth, I don't use Alfred's export function myself. I use this script. Although the version I'm using is slightly more sophisticated, and adds the version number to the generated .alfredworkflow file (and is only useful for Alfred-Workflow-based workflows). I just recommend the standard Export method in the tutorial because it's simple.

My main reservation with regard to Makefiles and setup.py etc. is that they're not particularly beginner-friendly. At the very least, it's something new to learn, and trying to include non-Python files via setup.py is something I get wrong all the time (I always forget how to do it and the official documentation is confusing).
 
There are definitely advantages to using a Makefile/build script or creating a setup.py, but I would be hesitant to try to enforce that method in any way due to the additional complexity. As usual, my main reservation about doing things "properly" is that it often isn't very accessible to newbies (which is why Alfred-Workflow's default behaviour is counter to best practices in several respects).
 
How realistic do you think it would be to provide a boilerplate setup.py (or a script that generates one) that wouldn't require the user to know too much about setuptools, entry points etc.?
 
The upload_packal command is certainly very enticing, however. I could definitely see myself adding a setup.py to my own workflows.
 
How do you envisage the Alfred entry points working? How that might work is not obvious to me.

Edited by deanishe
Link to post

For what it's worth, I don't use Alfred's export function myself. I use this script. Although the version I'm using is slightly more sophisticated, and adds the version number to the generated .alfredworkflow file (and is only useful for Alfred-Workflow-based workflows). I just recommend the standard Export method in the tutorial because it's simple.

Noted and edited in the first post :)

 

My main reservation with regard to Makefiles and setup.py etc. is that they're not particularly beginner-friendly. At the very least, it's something new to learn, and trying to include non-Python files via setup.py is something I get wrong all the time (I always forget how to do it and the official documentation is confusing).

True... the documentation about extra files is lacking to say the least.

The most useful documentation to me is the Python 2 documentation. For some reason this was removed with the Python 3 release unfortunately but the Python 2 docs are still available: https://docs.python.org/2/distutils/sourcedist.html#commands

Given a bit of well documented boilerplate it might be enough for beginners. Alternatively, perhaps the cookiecutter package could help to make a simple create-a-workflow wizard: https://cookiecutter.readthedocs.org/en/latest/

 

There are definitely advantages to using a Makefile/build script or creating a setup.py, but I would be hesitant to try to enforce that method in any way due to the additional complexity. As usual, my main reservation about doing things "properly" is that it often isn't very accessible to newbies (which is why Alfred-Workflow's default behaviour is counter to best practices in several respects).

I would personally vote for a documented standard, not something that is enforced in any way but simply an easy starting point for beginners with enough documentation to get a workflow up and running without too much effort.

 

How realistic do you think it would be to provide a boilerplate setup.py (or a script that generates one) that wouldn't require the user to know too much about setuptools, entry points etc.?

Assuming we create a cookiecutter template I think it would be easy enough.

Alternatively, similar to how the django-admin command makes it possible to create a project using "django-admin startproject test" a "alfred create test" command could be created as well. Possibly even with an "alfred upload" or similar.

 

The upload_packal command is certainly very enticing, however. I could definitely see myself adding a setup.py to my own workflows.

 

How do you envisage the Alfred entry points working? How that might work is not obvious to me.

Ideally this type of installation would be supported by Alfred internally but until that time we could have the bdist_alfred create a package that adds scriptfilters and such to a automatically generated plist file.

So instead of having:

setup(
    ...
    entry_points={
        'distutils.commands': [
            'upload_sphinx = sphinx_pypi_upload:UploadDoc',
        ],
    },
)
It could be something like:

setup(
    ...
    entry_points={
        'alfred.scriptfilters': [
            'command = workflow_script:Command',
            'optional_arg_command * = workflow_script:OptionalArgCommand',
            'required_arg_command + = workflow_script:OptionalArgCommand',
        ],
    },
)
Note that all of this is purely theoretical at this point. But I do think it would make for an easier and more reusable system.
Link to post

It's worth mentioning that in v3, the built in export does a few more tasks such as stripping out specified workflow variables (i.e. if you've filled in an API key you don't want exporting)... Just keep that in mind if you are exporting workflows outside of Alfred's prefs :)

 

workflow%20vars%20for%20export.png

Link to post

Noted and edited in the first post :)

 

True... the documentation about extra files is lacking to say the least.

The most useful documentation to me is the Python 2 documentation. For some reason this was removed with the Python 3 release unfortunately but the Python 2 docs are still available: https://docs.python.org/2/distutils/sourcedist.html#commands

That's where I always refer to. Problem is I can never remember when I'm supposed to use package_data and when MANIFEST.in. (And I usually end up publishing a broken version of Alfred-Workflow as a result, which is why there are a lot of gaps in its history on the Cheeseshop.)

There must be a clearer tutorial out there somewhere…

 

Given a bit of well documented boilerplate it might be enough for beginners. Alternatively, perhaps the cookiecutter package could help to make a simple create-a-workflow wizard: https://cookiecutter.readthedocs.org/en/latest/

 

I would personally vote for a documented standard, not something that is enforced in any way but simply an easy starting point for beginners with enough documentation to get a workflow up and running without too much effort.

 

Assuming we create a cookiecutter template I think it would be easy enough.

 

This is the cookiecutter template I use, but it's closely tied to the way I write workflows (docopt-based CLI programs that happen to output XML).

 

I'm not sure if that's broadly useable, but a cookiecutter template is definitely a good idea.

 

Alternatively, similar to how the django-admin command makes it possible to create a project using "django-admin startproject test" a "alfred create test" command could be created as well. Possibly even with an "alfred upload" or similar.

 

Ideally this type of installation would be supported by Alfred internally but until that time we could have the bdist_alfred create a package that adds scriptfilters and such to a automatically generated plist file.

So instead of having:

setup(
    ...
    entry_points={
        'distutils.commands': [
            'upload_sphinx = sphinx_pypi_upload:UploadDoc',
        ],
    },
)
It could be something like:

setup(
    ...
    entry_points={
        'alfred.scriptfilters': [
            'command = workflow_script:Command',
            'optional_arg_command * = workflow_script:OptionalArgCommand',
            'required_arg_command + = workflow_script:OptionalArgCommand',
        ],
    },
)
Note that all of this is purely theoretical at this point. But I do think it would make for an easier and more reusable system.

 

 

I'm still not following regarding entry points, I'm afraid. Is this based on Alfred supporting entry points natively? So, like Alfred loads your module/package and sets keywords appropriately. Or would there be some framework responsible for receiving input from Alfred and dispatching it appropriately, like a setuptools-based plugin manager?

 

I can definitely see the benefit in something like an alfred tool in the vein of django-admin. I've thought about writing something to replace the workflow-build.py script I linked to above and its workflow-install.py counterpart. (Its most useful feature being the ability to symlink the source code directory to Alfred's workflow folder, so your repo isn't hidden somewhere in the depths of ~/Dropbox or ~/Library.)

I've not yet been convinced that its utility would outweigh the effort involved (if we're talking about features like adding new Script Filters, Actions etc. to existing workflows), however. I think even more so now, as Alfred 3 looks like it will add a whole bunch of new stuff. install, build and publish (to Packal, possibly GitHub, too) are no-brainers. Beyond that, I dunno. Perhaps create, too, as a simple wrapper around a cookiecutter call?

I've largely avoided the need for any intelligence in my scripts, again because the actual workflow is entirely contained in a subdirectory. I only need to symlink/copy that directory to the right place to install, or zip its contents minus *.pyc files to build.

This seems to work well enough for Python (for my purposes, at least), but I had to use a proper build script when I tried Go.

 

That said, something that a sophisticated alfred tool would probably need that I think would be extremely useful for a lot of workflows is a library that can understand and manipulate info.plist. There's a lot of cool stuff you can do when your workflow can rewrite its own code. I'm currently rewriting Searchio! to be more easily configurable, and it will need to be able to rewrite its own info.plist.

 

Fuzzy Folders messes with its own info.plist, but the code is neither good nor portable.

 

With regard to Packal, we (Shawn, I and a few others) were mulling over a "standard" workflow.ini file that Packal and Alfred-Workflow and any other libraries could use. That's all a bit up in the air at the moment, as it's not clear how much of the information Packal needs will be in the Alfred 3 info.plist.

Link to post

One more thing:
 

 
The bundler is a different beast to what we've been talking about. It's purpose is to be able to fetch dependencies post-install and keep them all in once place, so you don't have 20 copies of cocoaDialog or requests or terminal-notifier etc. spread out across your workflows.
 
It is, AFAIK, abandonware (at least for the time being). There are a few workflows out there that use it, but I'm fairly confident it isn't being actively developed and I'm not sure Shawn would recommend using it at this time.

Link to post

It's worth mentioning that in v3, the built in export does a few more tasks such as stripping out specified workflow variables (i.e. if you've filled in an API key you don't want exporting)... Just keep that in mind if you are exporting workflows outside of Alfred's prefs :)

That's indeed interesting information, thanks :)

 

I suppose that would have to be included somewhere as well but I'm not sure what would be the best location. I don't think the entry_points would be a great fit.

 

That's where I always refer to. Problem is I can never remember when I'm supposed to use package_data and when MANIFEST.in. (And I usually end up publishing a broken version of Alfred-Workflow as a result, which is why there are a lot of gaps in its history on the Cheeseshop.)

There must be a clearer tutorial out there somewhere…

Personally I would recommend the usage of MANIFEST.in alone and simply ignore package_data. They largely overlap in functionality and I would argue that the MANIFEST.in system is easier to interpret as it doesn't offer any mapping/rewrite options.

 

As for a better tutorial, chapter 15 of my book covers the usage of MANIFEST.in but I'm not sure if it's that much clearer... I'll send you the paragraph soon so you can judge for yourself :)

 

This is the cookiecutter template I use, but it's closely tied to the way I write workflows (docopt-based CLI programs that happen to output XML).

 

I'm not sure if that's broadly useable, but a cookiecutter template is definitely a good idea.

 

Looks like a very nice starting point :)
 
 

I'm still not following regarding entry points, I'm afraid. Is this based on Alfred supporting entry points natively? So, like Alfred loads your module/package and sets keywords appropriately. Or would there be some framework responsible for receiving input from Alfred and dispatching it appropriately, like a setuptools-based plugin manager?

 

Ideally Alfred would support the entry points natively which would even make it possible to do a pip install -e workflow for development purposes. That would break backward compatibility however so I don't think that should be the first or only option.

 

What I would like to propose is a setuptools build command which generates the entire workflow including the Alfred XML from the setup.py file.

 

To ease development I think it should also contain a alfred_install or alfred_develop command to symlink the package so Alfred sees the workflow straight away.

 

The added benefit of the setuptools entry points approach is that it would allow for a plugin manager to easily detect all Alfred plugins and show a list of available entry_points. That would allow for easy documentation as well since everything will be automatically generated :)

 

 

I can definitely see the benefit in something like an alfred tool in the vein of django-admin. I've thought about writing something to replace the workflow-build.py script I linked to above and its workflow-install.py counterpart. (Its most useful feature being the ability to symlink the source code directory to Alfred's workflow folder, so your repo isn't hidden somewhere in the depths of ~/Dropbox or ~/Library.)

I've not yet been convinced that its utility would outweigh the effort involved (if we're talking about features like adding new Script Filters, Actions etc. to existing workflows), however. I think even more so now, as Alfred 3 looks like it will add a whole bunch of new stuff. installbuild and publish (to Packal, possibly GitHub, too) are no-brainers. Beyond that, I dunno. Perhaps create, too, as a simple wrapper around a cookiecutter call?

I've largely avoided the need for any intelligence in my scripts, again because the actual workflow is entirely contained in a subdirectory. I only need to symlink/copy that directory to the right place to install, or zip its contents minus *.pyc files to build.

This seems to work well enough for Python (for my purposes, at least), but I had to use a proper build script when I tried Go.

 

That said, something that a sophisticated alfred tool would probably need that I think would be extremely useful for a lot of workflows is a library that can understand and manipulate info.plist. There's a lot of cool stuff you can do when your workflow can rewrite its own code. I'm currently rewriting Searchio! to be more easily configurable, and it will need to be able to rewrite its own info.plist.

 

Fuzzy Folders messes with its own info.plist, but the code is neither good nor portable.

 

The single strongest argument I can think of to go for a smart utility is consistency. A single silly mistake in a workflow can break everything which is especially annoying for inexperienced developers.

Not having to worry about packaging, uploading and the writing of a plist file at least removes that part of the equation.

 

Everything is open for discussion of course, as long as it makes it things easier for developers :) 

 

With regard to Packal, we (Shawn, I and a few others) were mulling over a "standard" workflow.ini file that Packal and Alfred-Workflow and any other libraries could use. That's all a bit up in the air at the moment, as it's not clear how much of the information Packal needs will be in the Alfred 3 info.plist.

 

That could be interesting, and generally easier to write than setup.py files :P

 

One more thing:

 

 

The bundler is a different beast to what we've been talking about. It's purpose is to be able to fetch dependencies post-install and keep them all in once place, so you don't have 20 copies of cocoaDialog or requests or terminal-notifier etc. spread out across your workflows.

 

It is, AFAIK, abandonware (at least for the time being). There are a few workflows out there that use it, but I'm fairly confident it isn't being actively developed and I'm not sure Shawn would recommend using it at this time.

 

Ok, I'll edit the start post :)

Link to post

Personally I would recommend the usage of MANIFEST.in alone and simply ignore package_data. They largely overlap in functionality and I would argue that the MANIFEST.in system is easier to interpret as it doesn't offer any mapping/rewrite options.

Thanks! I'll ditch package_data from AW's setup.py.

 

Ideally Alfred would support the entry points natively which would even make it possible to do a pip install -e workflow for development purposes. That would break backward compatibility however so I don't think that should be the first or only option.

 

What I would like to propose is a setuptools build command which generates the entire workflow including the Alfred XML from the setup.py file.

 

To ease development I think it should also contain a alfred_install or alfred_develop command to symlink the package so Alfred sees the workflow straight away.

 

The added benefit of the setuptools entry points approach is that it would allow for a plugin manager to easily detect all Alfred plugins and show a list of available entry_points. That would allow for easy documentation as well since everything will be automatically generated :)

I can't imagine Andrew adding native Python support (in the sense of embedding it, like Sublime or vim).

 

The thing with a non-standard installation method, such as pip install …, is that anything that isn't installed in Alfred's workflow directory (well in the Alfred.alfredpreferences bundle to be precise) doesn't get synced. If you start putting stuff in site-packages, you need to do it on all your machines (and potentially worry about making sure they all have the same version installed).

 

For any installation method, you also need to strip the workflow's Hotkeys on installation (and re-instate the ones the user has set). Alfred and Packal both do this. There may be other things too (Packal has some kind of workflow signing, but I don't know the details).

 

I'd be tempted to punt on that one, so python setup.py install would build the .alfredworkflow file and then tell Alfred to open (i.e. install) it itself. This is what AW's update mechanism does.

 

There's almost never a need to edit info.plist by hand. In 99.5% of cases, you're better off editing it indirectly via the workflow editor in Alfred Preferences. On occasion, I'll edit a word or two in my editor because I have that open in front of me, but I'd never consider trying to add an object to the workflow that way. There are several cases where you'd want to edit it programatically, however.

 

Again, perhaps I've misunderstood, but I'm not sure that generating XML from a setup.py would be a great help. The vast majority of the time, the XML needs to be generated dynamically based on the user query and your workflow's data, and it always has to be fed to Alfred via the STDOUT of a Script Filter process (at least in Alfred 2).

 

The single strongest argument I can think of to go for a smart utility is consistency. A single silly mistake in a workflow can break everything which is especially annoying for inexperienced developers.

Not having to worry about packaging, uploading and the writing of a plist file at least removes that part of the equation.

Agree 100% except regarding the plist file: Alfred already has that covered, imo.

Link to post
  • 4 weeks later...

I can't imagine Andrew adding native Python support (in the sense of embedding it, like Sublime or vim).

 

The thing with a non-standard installation method, such as pip install …, is that anything that isn't installed in Alfred's workflow directory (well in the Alfred.alfredpreferences bundle to be precise) doesn't get synced. If you start putting stuff in site-packages, you need to do it on all your machines (and potentially worry about making sure they all have the same version installed).

 

For any installation method, you also need to strip the workflow's Hotkeys on installation (and re-instate the ones the user has set). Alfred and Packal both do this. There may be other things too (Packal has some kind of workflow signing, but I don't know the details).

 

I'd be tempted to punt on that one, so python setup.py install would build the .alfredworkflow file and then tell Alfred to open (i.e. install) it itself. This is what AW's update mechanism does.

That's a very good point. I think you are right, that is most likely the most convenient option here.

 

There's almost never a need to edit info.plist by hand. In 99.5% of cases, you're better off editing it indirectly via the workflow editor in Alfred Preferences. On occasion, I'll edit a word or two in my editor because I have that open in front of me, but I'd never consider trying to add an object to the workflow that way. There are several cases where you'd want to edit it programatically, however.

 

Again, perhaps I've misunderstood, but I'm not sure that generating XML from a setup.py would be a great help. The vast majority of the time, the XML needs to be generated dynamically based on the user query and your workflow's data, and it always has to be fed to Alfred via the STDOUT of a Script Filter process (at least in Alfred 2).

Fully generating the files might not be that useful but some part could be. To make the creation of new packages easy it would be very nice if developers could do something like:

@alfred.scriptfilter(keyword='spam')
def some_function(query):
    pass
That would automatically link that function to a scriptfilter using the specified keyword. Once it's generated there's no real need anymore so everything can be edited by Alfred after that but it can make the link between Alfred and the Python scripts easier :)
Link to post
@alfred.scriptfilter(keyword='spam')
def some_function(query):
    pass
That would automatically link that function to a scriptfilter using the specified keyword. Once it's generated there's no real need anymore so everything can be edited by Alfred after that but it can make the link between Alfred and the Python scripts easier :)

 

 

What do you mean by this exactly? The decorator would capture the output of some_function then call Alfred via AppleScript with the query spam <output>?

Link to post

Well, it's just one possible option and not necessarily we should include. But I mean the other way around.

This could be a shortcut to definining Alfred scriptfilters from the Python code itself. Instead of having to modify the plist yourself by adding a scriptfilter through Alfred you could have it automatically add a scriptfilter to the plist file using this decorator.

Link to post

It is, AFAIK, abandonware (at least for the time being). There are a few workflows out there that use it, but I'm fairly confident it isn't being actively developed and I'm not sure Shawn would recommend using it at this time.

 

I might revisit it at some point soon. I do think it is useful, and v2 was close to being done. I have a few other things on my plate before I get back to it.

Link to post

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

×
×
  • Create New...