defunct ZotQuery: an Alfred workflow for Zotero

[smarg19]

Here's a user question/poll:
 
Having switched to @deanishe's library, I've completely reworked the querying procedures to utilize his very helpful filter method. This now allows for ZotQuery to work with multiple search terms and use algorithms similar to Alfred's himself. But here's the question: This change has gotten me thinking about how results are ordered. As things currently stand, ZotQuery allows Alfred to remember which results you act on, and so these results will be given primacy over time, regardless of their position in the query relevancy. I could change this, but should I?
 
So there are two options:

• Have Alfred remember chosen items and have them filter towards to top of results, or
• Keep all query results ordered by the searching algorithm?

What say ye? Any hard preferences? If not, I'm just going to keep things as they are.

Edited March 3, 2014 by smarg19

[deanishe]

Good point. Thanks for the reminder. I've changed the workflow accordingly, but I won't push out an update till I finish at least one more of my desired goals.

One of my long-term, don't mess with now goals is to see whether I can port pyzotero away from its requests dependency to your web module. But that's for another month

I had a look at the module myself today with the same idea. Afraid it won't work as it uses DELETE HTTP calls. I'd have to look at the actual API to see if POST is also okay, but my web.py module doesn't do DELETE or PUT. I'd like to add them, but I'd have to override some of Python's built-in libraries to do that. I'll look into the feasibility of that at some point, but for now, I don't think it's possible to replace requests with my web module.

[smarg19]

That makes sense. And luckily the general cleanup brings the workflow size down a good bit. And your sys.path appending method allows me to hide the eye-sore folders anyway. So far, your library works like a gem. I haven't encountered any bugs, tho I have posted 2 issues to the GitHub repo. Both are essentially feature requests, but I have workarounds in place now, so its nothing urgent.

Edited March 3, 2014 by smarg19

[deanishe]

I've closed your "method arguments" issue as resolved.

You can pass a dict that contains the named arguments to a function/method simply by prepending ** to the name of the dict as you pass it in: 

title = 'a spoon'
subtitle = 'my favourite spoon'
arg = 'more spoons'
uid = 'spoon-mafia'
add_item(title, subtitle, arg, uid)

is equivalent to 

d = {'title': 'a spoon', 'subtitle': 'my favourite spoon', 'arg': 'more spoons', 'uid': 'spoon-mafia'}

add_item(**d)

You just have to be sure that your dict contains all required arguments (those without default values).

This is a common Python idiom and works with any function/method.

[smarg19]

Magic! Thanks for the info. Sorry for an "Issue" based on ignorance. But, this change-over has provided an impetus for me to start learning some about Object Oriented Programming. While I'm still in the basics stage, it makes a ton of sense and I can see how it could prove useful to ZotQuery. But right now, its one step at a time.

[deanishe]

Don't worry about it. I've done the same thing many times myself. I even managed to ask Guido van Rossum himself a really dumb question about Python. The shame

I wouldn't say that ZotQuery would necessarily benefit from classes as such, but it could do with refactoring under the DRY (don't repeat yourself) principle.

_mappings.py, for example, contains three functions that are largely the same. You could look at how you could change those to a couple of lines each and put all the common code into other functions they can all use. That way, when you find a whoopsie, you only have to change it in one place, not three or four (which increases the likelihood of missing one or two).

[smarg19]

Good points. I'll go thru at some point and clean more up. 

 

For now, I pushed a quick update to Packal (v. 6.1.1) which cleans up the filter.py script with the dict passed as a var to wf.add_item. More importantly, however, I've utilized @deanishe's thoroughly helpful "magic arguments" feature to add a new keyword: z:bug which will automatically open ZotQuery log file, which the user can then send my way for easier debugging. This is actually a crazy helpful feature (as the logs of this thread show) and I tip my hat to @deanishe for thinking up such a simple, yet to-date unactualized feature. To any user's who post bug reports in this thread, Please use the z:bug keyword to get your logs and post them alongside your bug report (either as a Dropbox link or something similar). This will greatly aid and speed-up my ability to help you.

 

As a side promo: there is MUCH to like about his Python-Alfred library. I personally love the error handling. If an error is thrown, the workflow automatically parses its type, and displays an error report in Alfred's viewbar with an appropriate icon and the error message. It's a small touch (visually, not code-wise), but I appreciate the polish. 

Edited March 3, 2014 by smarg19

[deanishe]

The error catching is an absolutely genius idea, and not the slightest bit mine

 

I copied it from zhaowu's Alfred2-Ruby-Template. That's where I got the idea to include a bunch of OS X system icons from, too.

 

The log-file-opening thing was all me, however, and I take fullest credit for it

 

 

 

[smarg19]

As a scholar-in-training, I appreciate the citations. My complete lack of knowledge of Ruby kept me blinded to that library/gem/template's existence. Whatever the source, tho, it is genius and I appreciate having it in ZotQuery.

[kithairon]

Thanks for recent updates. It's hard to catch up with ZotQuery's update speed The recent changes also mean that my neat odf-scan option built in via your test_scannable_scannable-cites-export.py (you explained in #178) don't work anymore. I see you've not included the file in the latest version and also the filter_zot_general.py it relied on has gone. I wonder whether you plan to bring this functionality back? (Possibly handling umlauts and indic diacritica?) I copied the old test...export.py back in the worflow's folder tried changing the script filter to rely on the filter.py instead – a sheer act of hope – with no luck though.

If you decide not to I can go back to an older version of ZQuery that still supports the above tweak. Thanks.

Edited March 4, 2014 by kithairon

[deanishe]

Stephen has rolled all the filter_* scripts into one: filter.py. You should be able to get the old behaviour of filter_zot_general.py "{query}" with filter.py "{query}" "general".

Note, I haven't tested this, I just looked at the code.

[kithairon]

Thanks for helping. Have replaced the old scrip filter with this: [python filter.py "{query}" "general"]. A step further, definitely. I seem to get loads more search results – not always the most straightforward ones. Also, the export produces this error:

 

Traceback (most recent call last):
  File "test_scannable-cites-export.py", line 3, in <module>
    import alp
ImportError: No module named alp

 

(z:bug only produces this – which I believe to be the confirmation of it actually working.)

14:44:47 workflow.py:371 INFO     Opening workflow log file

Please let me know if you have more ideas.
 

[smarg19]

@kithairon, 

 

I'm sorry for the newest version breaking functionality you use. However, the ODF-RTF Scannable Cites export shall return. I want to roll it into the Export Styles tho. This means that it will be the 6th option in the configurator's choice of export styles (mla, apa, bibtex, etc.)

 

In order to move the whole workflow over to @deanishe's library, I had to remove alot of beta stuff and clean the whole workflow up a bit. But now that I've done that, I'm adding those beta features back as fully functional features. I really appreciate you beta test the ODF export, and I will have it back soon, I promise. 

 

(note: the old script fails because it relies upon the alp library, which is what I migrated away from)

[kithairon]

 

However, the ODF-RTF Scannable Cites export shall return

Great. I understand you're knocking down walls and removing floors to redesign the house while I try to straighten the pictures on the wall. Take your time.

My beta-testing is as much greed and distraction as sheer selflessness

[smarg19]

Well it is a small fix, so I just pushed an update to Packal for version 6.2. Now, "odt-scannable-cites" is a new option in the z:set export options. If you export a full Citation or a short Reference, you will get a base form Scannable Cite for that item. 

 

@kithairon, try it out and let me know how it works for you. 

[kithairon]

Excellent! The new function looks good. Odt-scannable-cites seem to work well. I threw about 20 citations and my worst accented authors at ZQ and the endproduct after the scan looks flawless. The actual exported cites can look a bit grim (eg: { | Ñāṇavīra, 1987 | | |zu:17490:G5HT7458} and { | Śāntideva, 2002 | | |zu:17490:7XN5U2A6} ) but after Zotero's odt scan they are rendered perfectly.

 

There is something else though: I have many authors with multiple items in my db and ZotQuery – after a "za" author search – lists far fewer items than are actually stored in Zotero. (One author has 76 items, ZQ consistently only finds 2 of those). This happens irrespective of language, i.e. diacritica in the author's name. I've re-cached a few times tough this doesn't seem to change anything. (The log stays empty). I'm fairly confident that this was different a few versions ago. Any idea?

Edited March 4, 2014 by kithairon

[smarg19]

I'm actually in the middle of working on a similar issue. In short, I completely changed the way that ZotQuery searches its database cache. In the past version, I used simple in searches to see if the query was in any of the places ZotQuery was looking. So for zot:a searches, it would look thru every item's creators' last names and see if the query was a part of that last name. This worked fine, but was a bit simplistic. 

 

In the new version (post 6.0), I use @deanishe's fuzzy search function (workflow.filter) for all of my queries, regardless of type). While his method has many benefits over my original method, and in some ways is much looser, in other ways it is much stricter. I have actually found it, in my experience to be looser than I would want (let in more results than necessary). I would love to test my improvements on your data (where you get less results than you want). Could you send me your zotero_db.json file (there won't be anything confidential in there, only your Zotero data. And I want be looking at it, just testing against it). It is found in /Users/[username]/Library/Application Support/Alfred 2/com.hackademic.zotquery/. Then, could you tell me a few of these author last names that aren't getting enough results via ZQ?

I'll send you a PM here on the forums with my email address, so that you don't have to post it publicly. 

 

PS. I'll fix the funky ODT stuff too

Edited March 4, 2014 by smarg19

[smarg19]

There is something else though: I have many authors with multiple items in my db and ZotQuery – after a "za" author search – lists far fewer items than are actually stored in Zotero. (One author has 76 items, ZQ consistently only finds 2 of those). This happens irrespective of language, i.e. diacritica in the author's name. I've re-cached a few times tough this doesn't seem to change anything. (The log stays empty). I'm fairly confident that this was different a few versions ago. Any idea?

 

As you know, I have been working on the `Skimmer` workflow this week (and it's been a busy week in general), but I haven't forgotten about this. There were actually a few bugs in @deanishe's code as well as in mine. I've been patching those problems and will have an update to ZotQuery out soon. The goal is to have great coverage in results, regardless of input. For example, from your data, if you put 'Anālayo' into ZotQuery, you will get 78 results. Also, if you simply put 'Analayo' into ZotQuery, you will also get 78 results. 

 

This was a great bug catch and I greatly appreciate your continued deep interaction with these workflows. It really helps to take things to the next level. 

[deanishe]

Hey, that's not a bug, it's a feature

 

The search wasn't designed for folks to start throwing whole paragraphs at it: you're meant to streamline your search keys first!

 

No, but seriously, you're right, it does need some fine-tuning of the mathematical sort. You should also consider stripping common works like "the", "of", "on", "with", "from", "a", "an" etc. from such long search keys. Here's a good stop list.

[smarg19]

Hey, that's not a bug, it's a feature

 

The search wasn't designed for folks to start throwing whole paragraphs at it: you're meant to streamline your search keys first!

 

Oh I know. I was actually referencing the math bug as well as the new bug that I just commented on at the GitHub repo.

I think that as a free, easy-to-use fuzzy search, the filter method is fantastic (which is why I moved all of my filtering to it), but the heavy usage that such a move creates also stresses it in ways that help it grow  

 

I, for one, am learning alot just by tinkering with it so much.

Edited March 17, 2014 by smarg19

[smarg19]

Excellent! The new function looks good. Odt-scannable-cites seem to work well. I threw about 20 citations and my worst accented authors at ZQ and the endproduct after the scan looks flawless. The actual exported cites can look a bit grim (eg: { | Ñāṇavīra, 1987 | | |zu:17490:G5HT7458} and { | Śāntideva, 2002 | | |zu:17490:7XN5U2A6} ) but after Zotero's odt scan they are rendered perfectly.

 

There is something else though: I have many authors with multiple items in my db and ZotQuery – after a "za" author search – lists far fewer items than are actually stored in Zotero. (One author has 76 items, ZQ consistently only finds 2 of those). This happens irrespective of language, i.e. diacritica in the author's name. I've re-cached a few times tough this doesn't seem to change anything. (The log stays empty). I'm fairly confident that this was different a few versions ago. Any idea?

Version 6.4, now up on Packal, fixes both of these issues. This update should also allow you to search for most Unicode-lettered strings simply by using the proper corresponding ASCII character. For example, `za noel` will match the item whose creator's last name is "Nöel". 

 

Please upgrade and test.

[deanishe]

True enough. You've found a good few bugs I missed. You're a good fella to have using the code.

Edited March 17, 2014 by deanishe

[smarg19]

True enough. You've found nearly a good few bugs I missed. You're a good fella to have using the code.

You're a better fella for writing 99% of it!

[kithairon]

Thanks for the latest improvements; tried 6.4 with scannable cites and my accented names – it works great. When writing I've started pasting my cites with ZQ now. When doing an author search, I'm particularly grateful for getting at the accented characters by using the corresponding 'vanilla' ASCII character; this makes the searches much more convenient. I'm also getting the full number of entries for authors which have more than one item in my Zotero library – thus the new filter seems to do a great job. And I appreciate the aesthetic improvement of names in the scans itself  – i.e. "NÃÉaÃÑnãaviÃÑra" now looking like "Ñāṇavīra". Will get back when I have tried some of the other functions. This, however, is already a productive tool for me.

[smarg19]

This, however, is already a productive tool for me.

 

Really happy to hear it.