defunct ZotQuery: an Alfred workflow for Zotero

[kithairon]

Here's the error from Terminal after running the above config...py-file:

execution error: File Macintosh HD:Users:amw:icon.png wasn’t found. (-43)
Traceback (most recent call last):
  File "/Users/amw/Library/Application Support/Alfred 2/Alfred.alfredpreferences/workflows/user.workflow.069AB234-E6B0-4354-B204-D3381AEF38D6/config_install-python-dependencies.py", line 40, in <module>
    if res[0] == '1':
IndexError: string index out of range
amw-5:~ amw$

[smarg19]

Ok. For whatever reason, the AS dialog boxes where I try to show the icon don't like your machine. But that's fine because its an unnecessary addition. When I went and fixed the action AS config scripts, I forgot about the AS dialogs in this script. So, I went in and added the error routing with these dialog boxes as well. If it can't find the icon, then it will show simple dialogs instead. This should resolve the problem. 

 

I've just uploaded v. 5.1 to Packal with the bug fix. Install and let me know. 

[kithairon]

Trying to export a citation here the action.log

Traceback (most recent call last):
  File "action_export-citation.py", line 6, in <module>
    from dependencies.pyzotero import zotero
  File "/Users/amw/Library/Application Support/Alfred 2/Alfred.alfredpreferences/workflows/user.workflow.069AB234-E6B0-4354-B204-D3381AEF38D6/dependencies/pyzotero/zotero.py", line 55, in <module>
    import pytz
ImportError: No module named pytz

[smarg19]

Did you run the configurator first? That should see that the module pytz isn't downloaded and try to download it. If you didn't config, delete the config files and run z:config. Then see if the dialogs appear, then try the export. 

[kithairon]

Have deleted the .json files and run the configurator. I saw the install python dialog the first time – and installed and configured everything as before.

When exporting still no happiness. Here the action.log:

Traceback (most recent call last):
  File "action_export-ref.py", line 5, in <module>
    from dependencies.pyzotero import zotero
  File "/Users/amw/Library/Application Support/Alfred 2/Alfred.alfredpreferences/workflows/user.workflow.069AB234-E6B0-4354-B204-D3381AEF38D6/dependencies/pyzotero/zotero.py", line 55, in <module>
    import pytz
ImportError: No module named pytz

Edited February 10, 2014 by kithairon

[smarg19]

Okay. Well easy fix. Open Terminal and type "sudo easy_install pytz". Press return. Report the result and then try the exporter. 

[kithairon]

Installed /Library/Python/2.7/site-packages/pytz-2013.9-py2.7.egg
Processing dependencies for pytz
Finished processing dependencies for pytz
 

Action.log upon export:

Traceback (most recent call last):
  File "action_export-ref.py", line 5, in <module>
    from dependencies.pyzotero import zotero
  File "/Users/amw/Library/Application Support/Alfred 2/Alfred.alfredpreferences/workflows/user.workflow.069AB234-E6B0-4354-B204-D3381AEF38D6/dependencies/pyzotero/zotero.py", line 56, in <module>
    import feedparser
ImportError: No module named feedparser

[smarg19]

Okay. do the exact same thing for "feedparser" (instead of pytz). 

[kithairon]

Installed /Library/Python/2.7/site-packages/feedparser-5.1.3-py2.7.egg
Processing dependencies for feedparser
Finished processing dependencies for feedparser

 

You've done it. Happy to report that I have successfully exported references, citations and a bibliography of one of my collections! This will be immensely useful. Thanks for hanging in with me and being incredibly patient at every step of this.

Edited February 10, 2014 by kithairon

[smarg19]

Well thank you for giving me so much great feedback on small issues that weren't appearing on my machine. That feedback makes the workflow as a whole much stronger. 

[kithairon]

Glad to help and cheer on your work on a great work-flow.

Kithairon

[smarg19]

Well I'm glad it's working fully for you now, and hopefully it will be a great boon to your workflow. Also, if you do come to have feature requests, let me know. I obviously can't make promises, but I'm always interested in making things better. 

[smarg19]

Well this kerfuffle has led me to rethink the configurator's dependencies installer. I didn't like piping tasks from Python to Applescript to the shell. I also think I had a weak set-up, so I've completely rewritten the config_install-deps script. It is now entirely Applescript. It still checks to see what modules need to be installed, and prompts the user to install the necessary ones. But, it will now actually list the modules to be installed. This is in addition to hopefully being less buggy or more consistently helpful. 

 

PLEASE let me know if you run the new configurator (v. 5.2) and things work PROPERLY. I know that typically only bugs are reported, but I would sleep easier knowing it is working as I imagine it will. 

[deanishe]

Nice idea for an installer.

 

However, you should remove things like "os", "re", "shutil", "datetime" and all the other standard Python modules from the script: it just makes the script run unnecessarily slowly (shelling out and starting Python 30+ times instead of ~4 times). There's zero chance any of them are missing. At a glance, I'd say only "pytz", "httpretty", "requests" and "feedparser" are necessary out of the modules you test for.

 

As noted in the Alp thread, installing libraries in the system Python is not the optimal solution.

Edited February 11, 2014 by deanishe

[smarg19]

Good point on the list. I ran a script to just grap all of the import statements in the whole workflow. I'll go through and shorten the list. 

 

I am currently researching and testing to teach myself more about Python package management. Firstly, I didn't know that installing one module (pyzotero in this case) via pip will automatically install its dependencies. This means it would actually be easier not to bundle pyzotero with the workflow. The problems I found when trying to bundle packages/modules within the module always centered on explicit import statements. I was trying to use relative imports to specify the package/module in the workflow folder I had bundled. Given my general ignorance, this had tons of problems. I see now that adding a package folder to the sys.path will probably be easier. All of this new knowledge is leading me to rethink ZotQuery set-up. 

 

One question though: Let’s say I change the configurator to install pyzotero with its dependencies (is this unique to pip, or does easy_install automatically install dependent packages/modules as well? I ask, since not all users will have pip, but easy_install comes standard. If only pip, I will need to install pip first). Let’s say I also tell pip (or easy_install) to download all of these packages to ZotQuery’s Workflow Data folder. I will then need to add that folder to sys.path (can I do this in the bash script for the configurator, such that it is universal? Or will I need to add a line to each Python script to add the ZotQuery Workflow Data folder to sys.path before any imports?). Assuming all this, will the basic import statements in pyzotero function properly? So that, if pyzotero.zotero imports pytz, and both pyzotero and pytz are stached in the Workflow Folder, since this larger folder is in sys.path, things should just work as is, right? 

 

I’m sure this is a bit dumb, but I’m only now getting into the skeleton of Python. So the details of package management are currently beyond me (tho I am assiduously learning).

[deanishe]

If you install the packages to the workflow root directory, everything should work fine without fiddling with sys.path. The directory of the running script is automatically added to sys.path.

 

If you install the packages to a subdirectory within the workflow, you'll need to add the subdirectory to sys.path in every script that gets run before importing pyzotero.

 

Personally, given the way ZotQuery is structured, I think you should use pip to install pyzotero and its dependencies in the root workflow directory before you distribute the workflow, so your users don't have to worry about dependency problems and you don't have to waste time debugging them. Messing with the system Python is a bad idea, even installing pip. What if a user, like me, already has pip installed, but it belongs to, say, a non-system Python 3 installation? It might install an incompatible version of a library. If pip is installed, you can ask it which Python it belongs to (or just include your own runner script in your workflow—/usr/local/bin/pip is a very simple wrapper), but that's all an awful lot of work that has little to do with actually improving your workflow.

 

Integrating an installer into a workflow is a very good idea, but it's also a complicated one. It requires a lot of thought and testing to get right.

 

If I were you, I'd just include the packages in the workflow (using pip install --target) and concentrate on improving the workflow proper.

 

If I can help with the Python side of things, drop me a PM.

[smarg19]

Honestly, that is brilliant. I deleted pyzotero and did a fresh install directly into the dependencies folder. Pip automatically downloaded all necessary dependent packages, and a test run ran smoothly. I think you're 100% correct that dealing with user system stuff is unnecessarily painful. I've had way too much trouble with my initial set-up. What I've found as well is that if you delete the .pyc files before uploading, the workflow still isn't very large. Its less than 2 MB now. At one point, I had a bloated version around 6 MB. I would say that having the workflow creator pip install any dependencies into a `dependencies` folder is probably the easiest, simplest solution. 

 

Thank you so much for your help. 

 

I'm currently working on another large update that will add alot of functionality that revolves around Zotero's write API. I'll def send you a PM if I have any questions, but I'm pretty close now. 

 

Once again, thanks. 

[deanishe]

You're welcome. That's one hell of a workflow you're building there. Glad I could be of help

[mark]

Hello, ZotQuery export problems...

 

Running OSX 10.8.5, Alfred 2 and ZotQuery 5.3. z:config seemed to go ok and I was not prompted to install any python modules.

 

After setup found ZotQuery not exporting citations and references to clipboard. So attempted to troubleshoot.

 

* Trying to troubleshoot by following posts #101 to #110 at [2].

 

* Found that python module requests wasn’t installed (as in [2] #104). 

Ruddy$ python action_export-citation.py "2S94Z4BH"
Traceback (most recent call last):
  File "action_export-citation.py", line 6, in <module>
    from dependencies.pyzotero import zotero
  File "/Users/markruddy/Library/Application Support/Alfred 2/Alfred.alfredpreferences/workflows/user.workflow.BFFC5A57-46F4-460C-B476-17F66C339C09/dependencies/pyzotero/zotero.py", line 31, in <module>
    import requests
ImportError: No module named requests

* ‘requests’ was installed successfully using:

easy_install requests

* Still couldn’t export from ZotQuery. 

* New terminal error:

Ruddy$ python action_export-citation.py "2S94Z4BH"
Traceback (most recent call last):
  File "action_export-citation.py", line 6, in <module>
    from dependencies.pyzotero import zotero
  File "/Users/markruddy/Library/Application Support/Alfred 2/Alfred.alfredpreferences/workflows/user.workflow.BFFC5A57-46F4-460C-B476-17F66C339C09/dependencies/pyzotero/zotero.py", line 33, in <module>
    import feedparser
ImportError: No module named feedparser

* Attempted to install python module feedparser using:

python config_install-python-dependencies.py

* This produced a ZotQuery box promt to install basic python modules. But clicking ‘install’ had no effect, still get:

ImportError: No module named feedparser

...and am still unable to export from ZotQuery.

 

Appreciate any help

 

thanks

 

Mark

 

Sources

 

1. http://hackademic.postach.io/zotquery-an-alfred-workflow-for-zotero#fnref:1 
2. http://www.alfredforum.com/topic/3708-zotquery-an-alfred-workflow-for-zotero/page-6

[smarg19]

Sorry for the problems. These damned pyzotero dependencies have been the bane of me. But thanks to some help from @deanishe, I've finally squished this bug. I've just uploaded version 5.5 to Packal: http://www.packal.org/workflow/zotquery

 

This version no longer has Python module downloading as a part of the config. The necessary dependencies (requests, feedparser, pytz, and pyzotero) are all bundled with the workflow. All scripts in the workflow will use these modules (even if you have them installed). This should finally squish any bugs from dependencies. So upgrade the workflow and get back to me. 

 

stephen

[mark]

Stephen,

 

upgraded to 5.5 and now exports work fine. Thanks to you and @deanishe, but esp you for putting the workflow together.

 

PS

I'm new to Alfred. If I wanted to modify ZotQuery to export markdown [#citekey] and pandoc [@citekey] links where would I look? Within ZotQuery or is this part of Zotero prefs?

 

best wishes

 

Mark

[smarg19]

Well, that depends on what you are going to use to process the citekeys. For example, will pandoc process the document? If so, what will it look at to convert the citekeys? 

 

Currently, you can select BibTeX as an export option. Simply run `z:settings`, and bibtex will be the last option. But these citekeys are only meaningful relative to some database. So you will need to know exactly how whatever processor you use translates citekeys to formatted citations. 

 

FWIW, I actually created ZotQuery partially to avoid having to deal with citekeys. Why insert a citekey that will be blindly transformed into a formatted citation when I can directly insert the formatted citation just as easily? If, like me, you write in Markdown, you can export full citations and short references that are already formatted as easily as you would export citekeys, so why add the extra step. Especially when that step is less easy to tweak and debug. If, for instance, you have a small problem in your metadata, and the formatted references is slightly wrong, so you export your Markdown text to PDF and the PDF with the formatted references has an error, you can't fix the MD document, you have to find your database (in whatever format its in), find the corrupted item, fix it, then re-run the whole process. Whereas, with ZotQuery, you can insert the full citation in the MD document, see the error, fix it in the paper, and then later go and fix it in Zotero itself. 

 

So, all that to say, I can add CiteKey functionality, but I would need more information about your use-case, and I would also just recommend against it. I think the power of ZotQuery is to have the ease of citekeys alongside the ease of actual citations and references. But those are just my two cents.

[kithairon]

Thanks for the latest update. It's an enormous workflow! Delighted to say it works smoothly. Small feedback: Upon exporting a bibliograpy from a collection – the export works perfectly – I have, however, this from the action.log:

Traceback (most recent call last):
  File "action_read-restart-bib.py", line 51, in <module>
    with open(alp.cache(join="temp_bibliography.html"), 'r') as f:
IOError: [Errno 2] No such file or directory: '/Users/amw/Library/Caches/com.runningwithcrayons.Alfred-2/Workflow Data/com.hackademic.zotquery/temp_bibliography.html'

 

Despite your reasoning behind ZotQuery as a way to avoid citation keys – here a request: I'm using the odf/rtf-scan plug-in with Zotero and would appreciate the export option of "Scannable cite" (the rtf-scan is a lot less reliable). Since my draft writing and finishing don't happen in the same app (and I need live citations) pasting the finished citations in my document isn't an option. This begs the next question: how easy would it be to make the export configurable beyond the current few styles? A "scannable cite"-option would in some way make the 2nd request less needed, since with live connections the user could do all this via Zotero's plugins.
 

Edited February 19, 2014 by kithairon

[mark]

Interesting points @smarg19. I’m not sure whether this conversation belongs here or in a general workflow-esque forum but anyway…

 

 

Workflow

 

I come from a LaTeX tradition, only lately to markdown. Most of my library is in BibDesk, but I’ve started using Zotero recently because it’s ability to scrape refs is brilliant and is cross-platform.

 

My writing can be for academic, technical, journalistic, or web-based end points. So i need to keep my options open for formats and csl styles. The workflow i’ve gravitated to is:

1. Write in MMD/pandoc
   
2. Generally write into Scrivener
   
3. Export to LaTeX, .pdf, .odt, .doc, .md via pandoc or MMD+LaTeX, which calls a mybiblio.bib BibDesk file for bibliography
   

This gives me the ability to cite while writing and let pandoc/LaTeX sort out the reference style. But it only works semi-fluidly in BibDesk at the moment because Zotero doesn’t export MMD or pandoc citekeys or MMD referenced links for the bibliography. The zotero-better-bibtex (ZBB) add-on for Zotero by Emiliano heyns (friflaj) [2] does come tantilisingly close in exporting LaTeX citekeys and BibTex bibliographies from Zotero. I’ve tried to hack around with a fork of ZBB but the json is a bit too much for me at the moment.

 

 

The need for citekeys

 

I must admit I hadn’t thought about doing away with citekeys althogether. I take the very good points you make @smarg19. The problem seems to me (as you also say) is stable citekeys in the database that ensure the reference is maintained between citekey int he document and the bibliodb in Zotero. The developers at Zotero seem to have been asked for this type of functionality for a long time but avoided it because of the points you make.

 

One could circumnavigate the citekey integrity issue by dumping a snapshot of the bibliodb at the time of writing ones document – ZBB can do this. This would preserve the citekey-bibliodb links. You might say this is just the same as dispensing with citekeys altogether and doing what ZotQuery does now. Well sort of, but citekey-less writing prevents the ability to do good stuff with csl and automatically generate new bibliographies from the text if citekeys are dropped/added from the body text.

 

I have thought about how to modify my workflow to avoid citekeys. But the need for ‘live citations’ (as @kithairon puts it) is a killer for me. The availability of citekeys in Zotero would go someway to achieving a ‘scholarly markdown’ [3].

 

So, yes for me – and it sounds like @kithairon as well as other who at the moment are not using Alfred/ZotQuery – citekeys are pretty important. Maybe provide the option for people to use citekeys if they like. If the citekey generated in the db is stable – and the user knows this needs to be case – there should no problem.

 

You may be aware of these Zotero Firefox extensions as examples of the sort of functionality I’m getting at. ZBB [2] has the ability to specify citekey format and cite command – but the cite command is slightly inflexible, while zotero-autoexport [4] exports the library to different filetypes, but with citekey inflexibility.

 

I’d be happy to work with you /others to get the range of citekey options / bibliodb exports correct. But without inflating the workflow to levels that mae ZotQuery difficult to manage.

 

You’ve done so much of the heavy-lifting here already, it’s really impressive. I bought the powerpack on the basis of ZotQuery alone.

best wishes

 

Mark

 

Sources

1. http://www.alfredforum.com/topic/3708-zotquery-an-alfred-workflow-for-zotero/page-6 
   
2. https://github.com/friflaj/zotero-better-bibtex 
   
3. http://blog.martinfenner.org/2013/06/19/citations-in-scholarly-markdown/ 
   
4. http://rokdd.de/b/zotero-autoexport
   

[smarg19]

To both @kithairon and @mark, 

 

I know that many, many academics use CiteKeys and I think that adding fuller support in ZotQuery is a healthy feature addition. Since, however, I do not use them, it's not at the top of my list. But, let me give the state of affairs as they stand now and suggest some possible moves forward. 

 

First, ZotQuery currently has *some* support for CiteKeys. If you run the `z:settings` keyword, you can select "bibtex" as your CSL style. If you set this as your export format, you will get a result similar to this for a full citation export:

 

`@inbook{Todes_1975, place={The Hague}, title={Shadows in Knowledge: Plato's Misunderstanding of Shadows and of Knowledge as Shadow-Free}, booktitle={Selected Studies in Phenomenology and Existential Philosophy}, author={Todes, Samuel}, editor={Zaner, R. M. and Ihde, D.Editors}, year={1975}, pages={94-113}}`

 

This is a full BibTeX entry for the chosen item. It uses Zotero's own CSL style to generate the BibTeX, so I cannot speak to its accuracy, as I know there are many plugins that aim to improve Zotero's BibTeX export. 

 

If you export a short reference, you will get a result like this: `Todes_1975`. This is obviously the bones of a CiteKey. The question/issue is whether you could easily link this CiteKey to a specific item in your .bib file. Since this uses the Zotero API, there's no way to change the CiteKey that ZotQuery receives, so the only options would be to somehow alter the CiteKey returned in some way.

 

As to the RTF/Scannable Cites format, I actually started a script to generate these items. Its basically done, but since I don't use this feature I wasn't able to test it, and so left it out. Tho it remains in the code. You can see it in the GitHub repo here: https://github.com/smargh/ZotQuery/blob/master/old_and_test/test_scannable-cites-export.py . If you want this feature, add it the workflow and test it out, if it works well for you, I can add it as a permanent feature. 

 

So, steps forward?? I'm pretty busy with school currently, and I'm also working on some specific new features, so I don't have the time to properly tackle these issues here and now. But, a few small steps can easily be taken:

 

* It would be trivial to format the Zotero CiteKey in either pandoc or MMD format. Simply wrap it in either "[#...]" or "[@...]". So, if you can associate the Zotero citekey with that item in your .bib file, the problem is easily solved. 

* Since I've already started the RTF/ODT scannable cites script, it shouldn't be too difficult to get that up to snuff. As far as I can remember, it should currently generate the basic scannable cite for the chosen item. I just was never able to sufficiently test it. So if you test it and it consistently works, then that's all set.

* It is also trivially easy to add new CSL styles as export options, so if there is another CSL style in the Zotero repository that exports what you want, let me know and I can add it. 

 

Aside from these options, there is a slightly *hackier* solution that has the benefit of being easily implemented. If you want to use citekeys and bibtex, you *could* just roll with Zotero's basic implementation. So, for example, create a collection for your current writing project. When you've finished research, export a full bibliography of that collection in BibTeX format. This will be your .bib database for that particular paper. Then export short references in BibTeX format as you write and insert them into your text. Once written, process the paper as usual, but point your processor (MMD or pandoc) to the ZotQuery created .bib file, which will naturally have the citekeys match up. Now, since I don't use BibTeX with Zotero, I'm not aware of the issues with that integration. So I'm not certain how cleanly this would work, but you should try it out and let me know. I'm inclined to think that it will probably work more times than not. 

 

All of this to say, we should collectively explore the features that ZotQuery currently has, and if there are features missing, I would need to "crowdsource" their development, both because I don't have the time or the expertise in such things. All of the code is up on GitHub, and I will gladly receive pull requests, so let's jump into this together.

 

Let me know your thoughts, how some of these experiments go, etc. 

stephen