defunct ZotQuery: an Alfred workflow for Zotero

[rice.shawn]

That makes sense with the item key. I replaced it with one of mine, and test_space.py worked, but I'm getting the can't locate and clone database.

 

I've got the app and the firefox add-on installed.

 

--- paused for five minutes ---

 

Actually, I just tested something: I had my zotero library installed in a custom place ~/Documents/Zotero/, which was the problem. 

 

user_pref("extensions.zotero.lastDataDir", "/Users/Sven/Documents/Zotero");

 

There is another one that's just DataDir as well. You might consider, on a setup, to grab the datadir and place it in the/a settings file.

 

I'm not sure how you'd like to do it in python, but if you want to grab the directory in bash then use this:

echo $data | egrep -E "^user_pref\(\"extensions.zotero.dataDir\", \"([A-Za-z0-9 /]{1,})\"\);$" ~/Library/Application\ Support/Zotero/Profiles/*/prefs.js|grep -oE "\"([A-Za-z0-9 /]{1,})\""

[rice.shawn]

That one will return 

"/path/to/library"

with the quotation marks. It might be good to do this as a search and then grab it and throw it in the settings.

 

Also, the directory names where the "*" is in the regex seem to be randomly generated. You might run into some problems if there is more than one Zotero profile stored on the machine, so I'm not perfectly sure how to do that, except, maybe use the "lastDataDir" pref instead, which might work if we assume that we want to search the last available zotero one.

[rice.shawn]

Okay, now the library import seems to work, but the search queries don't. I went into the directory and ran `python filter_zot-general.py "test"` and I got an error telling me that it can't format the results, although test_space.py still works.

[smarg19]

Ok. To the directory issue. I actually had a version that did that (run regex on prefs.js to find path), but it got lost in updates (honestly don't remember how). Will add it. As for the randomly generated profile ID, that is also set in the prefs.js file, and I currently have a function that grabs it (in _zotquery). 

 

To debug the filters, you will need to open up the script (let's stick with zot-general) in a program that can execute Python (Sublime, TextMate, CodeRunner). Put this code into the test_space.py script and run it, report the errors. the problem (from the error above) is in the section from `for item in clean`... where the script generates the Alfred feedback. :

#!/usr/bin/env python
# -*- coding: utf-8 -*-
import alp
import json
import sys
from _zotquery import zot_string, info_format

"""
This script queries the JSON cache of your Zotero database for any matches of the query.
""" 

# Get Zotero data from JSON cache
cache = alp.cache(join='zotero_db.json')
json_data = open(cache, 'r')
zot_data = json.load(json_data)
json_data.close()

# Get the user input
#query = sys.argv[1]
query = 'griff'

# Search the Zotero data for matches
results = alp.fuzzy_search(query, zot_data, key=lambda x: zot_string(x))

# Clean up any duplicate results
if not results == []:
	clean = []
	l = []
	for item in results:
		if item['id'] not in l:
			clean.append(item)
			l.append(item['id'])


	xml_res = []
	for item in clean:
		# Format the Zotero match results
		info = info_format(item)
		
		# Prepare data for Alfred
		title = item['data']['title']
		sub = info[0] + ' ' + info[1]

		# Create dictionary of necessary Alred result info.
		res_dict = {'title': title, 'subtitle': sub, 'valid': True, 'uid': str(item['id']), 'arg': str(item['key'])}
		
		# If item has an attachment
		if item['attachments'] != []:
			res_dict.update({'subtitle': sub + ' Attachments: ' + str(len(item['attachments']))})
		
		# Export items to Alfred xml with appropriate icons
		if item['type'] == 'article-journal':
			if item['attachments'] == []: 
				res_dict.update({'icon': 'icons/n_article.png'})
			else:
				res_dict.update({'icon': 'icons/att_article.png'})
		elif item['type'] == 'book':
			if item['attachments'] == []:
				res_dict.update({'icon': 'icons/n_book.png'})
			else:
				res_dict.update({'icon': 'icons/att_book.png'})
		elif item['type'] == 'chapter':
			if item['attachments'] == []:
				res_dict.update({'icon': 'icons/n_chapter.png'})
			else:
				res_dict.update({'icon': 'icons/att_book.png'})
		elif item['type'] == 'paper-conference':
			if item['attachments'] == []:
				res_dict.update({'icon': 'icons/n_conference.png'})
			else:
				res_dict.update({'icon': 'icons/att_conference.png'})
		else:
			if item['attachments'] == []:
				res_dict.update({'icon': 'icons/n_written.png'})
			else:
				res_dict.update({'icon': 'icons/att_written.png'})

		res_item = alp.Item(**res_dict)
		xml_res.append(res_item)
			
	alp.feedback(xml_res)
			

[rice.shawn]

 

I actually had a version that did that (run regex on prefs.js to find path), but it got lost in updates (honestly don't remember how).

 

I hate it when that happens. To me: all too often.

 

So, I'm not quite sure about the above. I put that in test_space.py and ran it via the command line, and it just output the search information correctly, although not in xml format. Not sure what to tell you on this. I'm looking through all the files.

 

It looks like alp saved the database json file into the volatile storage where I mentioned above, and there is also the regular sqlite database there as well.

 

When I run "python filter_zot-general.py" from the command line, then I get "Error! Could not read/find JSON cache." When I put in a query after it (python filter_zot-general.py "test"), I get "Error! Could not format results."

 

Does that help?

[smarg19]

Ahhhhhhh......... That helps a ton. 

 

I had a stupid tab error in the filter. I will check the other filters as well. Basically, after it runs a search for the input, I had an if block (if results not empty...). I then had a try block, try to format the results. The try block was supposed to be under the if block, but it was in fact parallel, so even if the query didn't have any results it was trying to format the None. Dumb on my part. I'll update soon. 

 

Also just finishing a better prefs scraper based on suggestions. I'm certain this will fix your issues. BUT, do any of the other queries have problems? Does the workflow work for you at all right now?

[rice.shawn]

Well, because of the tab error, I don't think it works.

 

The test_space.py does grab the results, and the library caches work. I can't test the actual searches until the update!

[rice.shawn]

Here's a quick applescript that can do the settings file for you, although you might need to tweak it.

on run argv
	
	--- set the path to the preferences file
	set path_ to (path to "cusr" as text) & "Library:Application Support:Alfred 2:Workflow Data:com.hackademic.zotquery:settings.json" as text
	try
		set the file_ to open for access file path_
		set text_ to read the contents of file_
		close access the file_
	on error errMsg
		--- Maybe make the file right here. Or something.
	end try
	
	display dialog "Please enter your User ID. It should be seven characters." default answer "" buttons {"Cancel", "Where do I find my User ID?", "Set User ID"} default button 3 cancel button 1 with title "ZotQuery Configuration"
	set response to button returned of the result
	if response is "Where do I find my User ID?" then open location "https://www.zotero.org/settings/keys"
	set userid_ to text returned of the result
	
	display dialog "Please enter your API key. It should be 24 characters." default answer "" buttons {"Cancel", "Where do I find my key?", "Set Key"} default button 3 cancel button 1 with title "ZotQuery Configuration"
	set key_ to text returned of the result
	set response to button returned of the result	
	if response is "Where do I find my key?" then open location "https://www.zotero.org/settings/keys"
	
	---write the new file
	set the file_ to open for access file path_ with write permission
	set eof of file_ to 0
	write "{'type': 'user','user_id': '" & userid_ & "','api_key': '" & key_ & "'}" to file_
	close access the file_
	
end run

Maybe make it run on a "first run" or if the file isn't found. However, you might need to include a command to make the folder and the file first.

 

If you want to fancify it more, then you could put in a check to make sure that the text response is the right number of characters, and, if not, then launch the dialog again with an error message. It'd probably be easiest to write those as functions so you can just call them again on a bad result.

Edited January 14, 2014 by Shawn Rice

[smarg19]

That's awesome. Thank you. 

 

Noobish question tho. I've seen a number of workflows that set things up on first run. Is there some setting or way to have something run on a first run, or do you simply need to state in the README to run a certain command when first installed?

[rice.shawn]

What I'll usually do is to create a file in the NV storage called "first-run" when the first-run is complete. Then, whenever I invoke a script (either called by a keyword of a script filter), I just check to see if the file exists. If it doesn't, then I abort the query and launch the first run script. If it does, then I just continue with the query.

 

If you want to make changes on updates, then you could just have a different file (first-run, update-one, etc...), or you could have the contents of the first-run file to be a version# or something unique. I like to just have it as an empty file so I can just check to see if it exists and don't have to read and parse it

 

Instead of the first-run file, however, you could just do it with the settings file. Although, it would also be nice to have a "config" keyword that would just re-run the first-run script. The applescript provided overwrites the file each time.

Edited January 14, 2014 by Shawn Rice

[kithairon]

Back again for about for an hour. I run both Zotero and Standalonde, they both share a database, which is – I should have thought of this – not in the standard FF-profile (I have half a dozen them) but in /Users/amw/Documents/Zettelbank/Zotero

I feel we're getting closer.

Edited January 14, 2014 by kithairon

[smarg19]

We're def getting closer. I'm writing a much smarter prefs scraper that will (hopefully) find even non-standard database paths (the previous version assumed a standard path). I'm also re-writing the scripts to make debugging easier in the future (letting Python level errors pass through, instead of catching them). I'm also re-working the storage system (following Shawn's sage advice), and creating a `z:config` setup command that will run on first run and get things set up (using, among other things, Shawn's applescript from above). 

 

I'm learning a ton (which is great), but hopefully I will have learned enough soon to have a stable (actually stable) release out soon. Classes start up for me tomorrow, so that will cut into my time, but weekends are good still (at the beginning). I will push out an update to Packal soon (tomorrow or the day after), and we will go from there. 

 

Sorry for all of this, having never developed (A) something in Python, ( something of this scope, and © something for Alfred, I've made a few mis-steps. It sucks that others pay for that and not me (the workflow has always been working on my end). But I know that this will be very helpful to a lot of people, so I'm committed to getting it right. You initial debuggers have been immensely helpful to me, so thank you. 

[kithairon]

Glad to help – although you'd be better off with a more savvy user. What you're doing is great and will be so useful for many folks using Alfred and Zotero. Will board my plane in a few hours and be teaching flat out the next week, but hope to check in here and stay tuned. This is just too promising. Thanks for your patience.

Kithairon

[rice.shawn]

I've made a few mis-steps. It sucks that others pay for that and not me (the workflow has always been working on my end).

 

That's the fun thing about releasing something into the wild: it breaks in ways that you would never expect it to. And it doesn't suck for us. Instead think of it this way: you're giving us a free piece of software that will make our lives better and faster, and, quite possibly, the time we spent debugging will be less than what we save. Plus, I love trying to track down bugs because it makes me feel like I accomplished something, unlike writing...

 

Luckily I don't have to start teaching for another two weeks.

[smarg19]

Shawn, 

 

Quick question, what exactly is the difference between the volatile and non-volatile storage locations when dealing with upgrading workflows? Does volatile (in the Caches directory) get wiped with new installs of workflows while non-volatile remains? Or what exactly happens? 

[rice.shawn]

Upgrading workflows actually has nothing to do with it. It has to do more at the system level. The volatile cache is in ~/Library/Caches/, meaning that whenever the system caches are wiped, it's wiped. Depending on the configuration of the user's machine, that might happen when certain maintenance scripts run or at every reboot. On my machine, it's not cleared very often. On others', it might be.

 

If you're worried about having to keep things fresh, then it's fine to keep the cache in the cache. After all, if someone is actively updating their Zotero and doesn't update their ZotQuery cache, then new items won't be there. You want to keep it fresh.

 

You could make it so that the ZotQuery caches are generated automatically and kept fresh by doing a check that compares the file size of the zotero sqlite database with the copy that zotquery makes. If they're different, then you could just run the update cache script before proceeding. It would make things run slower if someone is actively updating their library, but it seems that people would be using this more when writing, so actual performance won't really take a hit. Just run that check every time the workflow is invoked.

 

Another note: on the first run script, you should probably make sure that the python dependencies are installed. Perhaps just run a check for all of them, and, if they aren't there, then throw up an Applescript dialog telling them that they need to install additional dependencies, if they so "install" then run a bash script with those commands that I posted, but from Applescript use "do shell script theScript_ with administrator privileges" -- that should take care of the "sudo" problem. If they press "cancel" on the dialog box, then just put up another one that says they won't be able to use ZotQuery unless they install those and exit the first run. You should be able to adapt the applescript that I wrote above with the dialogs and responses based on the above.

[smarg19]

Ah, that's what I thought about the storage. Thanks for the clarification. ZotQuery already auto-updates the cache. After every execution (there are 5 action options when you select an item, after any one of them), it will check if the cache needs to be updated, and if so, will update it in the background. Plus, I always keep `z:cache` around so that the user to force an update. 

 

I've just finished a new beta version of ZotQuery that has two step configuration (Zotero API settings, then Zotero paths) and better error reporting. In my earlier versions I was wrapping nearly everything in try/except blocks (a habit from my Applescripting days), which was actually more troublesome than helpful. Python has much better error reporting, so having raw Python errors report, rather than my little "Error! xxx..." line will greatly help in debugging. I've also added checks for configuration on all filters and keyword actions, so that nothing runs until you configure the settings. Finally, I radically improved the script for finding the various Zotero paths required (path to .sqlite database, path to internal storage, path to linked attachments). So that's no longer an issue. All in all, this is a much improved version. Will publish on Packal soon. 

 

I like the idea of checking for Python modules. I will work something up later. I actually have a beta-tester friend here who I'm going to ship this version to and see how it holds up on a "fresh environment" and if it holds, I'll push it out. 

[rice.shawn]

Applescript really is buggers when it comes to error reporting. I really hate that language, but it is the easiest way to do certain things.

 

Adding in the checks for the python modules will avert other support problems in the future. I'm also a big fan of not putting any data on a user's computer outside of the workflow folder and the NV and V storages without letting them know (which is why I suggested the Applescripts to let them confirm the installation of the modules).

 

But it all sounds great! You've been awesome pushing out these updates so quickly. 

[smarg19]

Version 3.0 upgrade now live on Packal. 

 

Major new feature is the Configurator. Now, configuring all the necessary settings for ZotQuery is (1) much more user-friendly, (2) much smarter, (3) much more beautiful. I must give a shout out to Shawn Rice, who offered me much help (and code) on the configurator. Please visit the ZotQuery page on my blog for full documentation of the configuration process.

 

In addition, I have cleaned up the workflow greatly as well. Things run more smoothly, more cohesively, and the code is more readable. There are also new error icons (I love making pretty icons). There's probably too much to explain here, but the page on my blog covers it all. 

 

Please download and let me know how it goes.

Edited January 15, 2014 by smarg19

[smarg19]

New major release! Number of big features. 

 

First, ZotQuery now can export in both Markdown and Rich Text. This means all academics on the Mac who use Zotero can take advantage of ZotQuery's quick access to their citation data. Export individual references or citations in rich text, or export full bibliographies from Collections or Tags in rich text. 

 

Second, I've added some exporting preferences to the configurator. Now you can select out of 5 possible CSL styles for export. Instead of the default Chicago (author-date), you can now also choose APA, MLA, BibTeX, or Zotero's own RTF-Scan cite key format. In addition, you will also then choose which format to export in: Markdown or Rich Text. At any point, you can also change these settings using the z:settings command. 

 

Also, I've added the ability to search only items with attachments. This works effectively like an attachments search, since you only need to press return (not shift+return as with the other queries) in order to open the attachment.

 

There are also a few small changes:

• now requires at least 3 characters to search (not 4)
• new titles and sub-titles for queries
• cleaner view when looking at workflow in Alfred's settings
• all queries are "loose" (query in data, not query begins data)
• fixed small bug in caching script
• fixed bug in configurator

 

This is a major update, and aside from any bug fixes is likely to be the last major release. I honestly can't think of any other features. With rich text support, ZotQuery now covers all the necessary bases.

 

I hope you enjoy it.

Edited January 18, 2014 by smarg19

[smarg19]

A couple of bug fixes and small edits pushed to Packal. From what I can tell, ZotQuery is stable now. Please let me know if you have any issues.

[smarg19]

Update.

 

I've added your notes to the data that is cached. This means that your general searches now search within your notes. I've also added a zot:n query type that will allow you search only within the notes of your items. So, if you make important notes about items, and want to find that one item where you made that one note, this feature is for you. 

 

I've also made a few bug fixes. 

 

Download or update via Packal: http://packal.org/workflow/zotquery

[smarg19]

Fixed a few small bugs. Updating on Packal now.

 

As a side-note, can people let me know what version of OS X you have ZotQuery working on? And what your Zotero set-up is? (Standalone? Firefox? Both?). I want to upgrade my README with testing conditions. Any info is helpful. Thank you.

[rice.shawn]

Mavericks, Both.

[zfolgerl]

The workflow will not export references, citations or bibliographies. 

 

I have followed the provided instructions, and confirmed in settings.json that my userID and key were correct. 

 

I'm running Mavericks and have both Firefox and standalone Zotero installed. 

 

Are there other steps to installing the workflow that were not explicitly stated? I'm relatively new to Alfred 2 and have limited coding knowledge.