defunct ZotQuery: an Alfred workflow for Zotero

[smarg19]

UPDATE: Version 6.5 now on Packal (tho currently I can't change version num, so still says 6.4; but I've changed the title to show new version)

 

New Features: Choose primary Zotero client in preferences (z:set command). When you open an item (return on selected item after any query), will open in your chosen client (Firefox or Standalone). This thus adds Firefox functionality when opening items in Zotero. 

 

Update at Packal.

[rice.shawn]

UPDATE: Version 6.5 now on Packal (tho currently I can't change version num, so still says 6.4; but I've changed the title to show new version)

 

Hm. I did something and a JS library that's very nice and is supposed to control "select" fields only decided to take over text fields... wtf? Anyway, I disabled it for now, so that means any select boxes will look not as nice. However, the text fields work as text fields again. I went ahead and changed the title back to ZotQuery and updated the version # for you on Packal. Sorry about that strange hiccup.

[kithairon]

Tried v6.5. Nice addition to open an item directly in FF. (Like probably a few Zotero users I use both Standalone and FF for different tasks.)

 

Some observations:

When I do an authorsearch and open the item in Zotero proper, I seem to get an additional empty tab for every opened item.

The highlighting in the middle pane is erratic, i.e. sometimes there and sometimes not. The item is always found, the info tab displays the looked for item's info, the middle pane shows the segment of my library that contains the looked for item – it's just not highlited in the usual grey. Have not found out any obvious difference from what I'm doing.

The tag function – after a ztg search – brings up my tags, however the in-tag-search doesn't produce my items (tested a few and couldn't find any authors in any of my tags)

Finally, sometimes, after a couple of searches I have what looks like a runaway process claiming between 90 and 100% of my CPU. I killed it initially but it seems to disappear after  2-3 mins on its own accord.

Any hope for pre- and suffix options coming back in the odf-cites output?

Edited March 20, 2014 by kithairon

[smarg19]

UPDATE: Version 6.6 up on Packal with a bug fix.

 

- - - 

 

When I do an author search and open the item in Zotero proper, I seem to get an additional empty tab for every opened item.

If you mean Zotero Firefox, there is nothing I can do to change that behavior. Firefox isn't really scriptable, and I have basically no control of its actions. All I can do is tell it to open the Zotero extension to that item.

 

 

The highlighting in the middle pane is erratic, i.e. sometimes there and sometimes not. The item is always found, the info tab displays the looked for item's info, the middle pane shows the segment of my library that contains the looked for item – it's just not highlited in the usual grey. Have not found out any obvious difference from what I'm doing.

 

This also goes back to having basically no control over Firefox. I know its buggy (which I why I didn't add it originally), but if you really want Firefox Zotero to open, this is what you have to deal with.

 

 

The tag function – after a ztg search – brings up my tags, however the in-tag-search doesn't produce my items (tested a few and couldn't find any authors in any of my tags)

 

Bug fixed in version 6.6 live on Packal now. 

 

 

Finally, sometimes, after a couple of searches I have what looks like a runaway process claiming between 90 and 100% of my CPU. I killed it initially but it seems to disappear after  2-3 mins on its own accord.

 

 

Please describe this process more. What is it named? It's probably the update cache function. I'll investigate when I get more info.

 

 

 

Any hope for pre- and suffix options coming back in the odf-cites output?

 

No. But you can manually add it. I think it deters from the clean user-experience. None of the other export options have 2 dialog boxes pop up. Plus, not everyone uses the prefix and suffix in the ODF cites, and those who do don't always use it. Its seems simpler and cleaner to let users add that info manually once the raw form is pasted into their doc.

 

 

 

[smarg19]

UPDATE: Version 6.6.1 alter how creators are listed in the Sub-Title. Instead of truncating creators to only 2 last names, ZotQuery now lists every creator's last name along with type.
 
Download and upgrade at Packal.
 

[greghoch]

Hi,

 

thanks for the great work on zotquery! I have one suggestion:

 

It would be great if zotquery supports fuzzy search as the main search with 'zot'. So if I want to search for the zotero item

Einstein 1935 Can quantum-mechanical description of physical reality be considered complete? 

I can simply type "Ein1935" to find it. My suggestion would be that the fuzzy search is applied to 

author year titile

So I could find the item also with "Einsteinquantum" or "Einquantum". But this could also be user configurable.

 

I think fuzzy search would be much nicer!

 

Alp even has a fuzzy search algorithm build in, which I contributed so I am happy to help with it. Just look at the fuzzy.py file. There is an example at the bottom of the code.

[deanishe]

There is already fuzzy search in ZotQuery, as Stephen has mentioned a few times.

 

Try putting a space between the separate words of your query.

 

If you search for "Ein1935", then the search will look for the substring "Ein1935". It will still match any search key containing the same characters in the same order (as Alp does), but such results will typically be ranked very low by the algo, quite likely below the cut-off rank, depending on what Stephen has set in ZotQuery and how far apart the words are.

 

If you want your query to match separate words, separate the respective parts of the query with spaces. "ein 1935", "ein 35" or "ein quant" will get you the results you're looking for. Heck, "e q" or "qmd" (quantum-mechanical description) would possibly suffice: the algo prioritises matches across first characters.

Edited April 5, 2014 by deanishe

[smarg19]

@deanishe is right, ZotQuery uses a fuzzy search algorithm. I'm a fan of this implementation, not least of all because it allows for a few different fine-grained controls. I currently have it set to be a bit stricter in its searches, so that performance stays top notch. However, if you want it as fuzzy as possible, it's a quick fix in the code. 

 

Open filter.py and run a simple Find and Replace:

Find match_on=MATCH_ALL ^ MATCH_ALLCHARS and Replace it with match_on=MATCH_ALL

 

Try that configuration for your searches and see if it meets your needs.

[greghoch]

Oh, that is great! I still have three question:

 

1) "Ein 1935" as a search string does not work for me. I tried it with multiple items (beginning of author name and year). Is it possible that year does not get searched? 

2) It is possible to exclude group libraries from the search? The reason is that a lot of my items show up twice because they are both in my personal and in my group libraries. Another reason is that there is a lot of crap in my group libraries, outdated entries or whatever. 

3) For me, it does not work to reveal items in group libraries (my preference would be to exclude them). When I select them, nothing happens. I think the library is indicated by the part before the underscore (e.g. 0_ENDQ22KT refers to an item in the personal library). 

 

Thanks!

[deanishe]

There's no mention of "year" in the source code that generates the search key.

Edited April 6, 2014 by deanishe

[smarg19]

@deanishe is right, the date currently isn't searched, but I can easily add that. 

 

I don't have or use group libraries, so I haven't tested this use case. Could you PM me your .json database so that I can use it for testing? I will try to find a way to easily exclude items in group libraries (if its easily done).

 

stephen

[greghoch]

Sending it via PM right now. Thanks!

 

For the search: I would prefer that the date gets searched or even better a user setting that allows you to configure what is searched by 'zot' (and the order). My personal preference would be "first_author_last_name year title".

[smarg19]

I am in the midst of refactoring the code base now, and will be adding these features. Group libraries will be supported, and I will add a "power user" feature to set the key for the `zot` query. I would recommend, however, using the more specific queries. I'm in the middle of some school work, so I can't say when ZotQuery will be updated, but it should alleviate most of these problems.

 

stephen

[smarg19]

UPDATE: Version 7.0 is now live on Packal. I have completely refactored the code for this version, opting for an object-oriented approach (which now allows me to have unit tests, yay!). However, the functionality is exactly the same, plus some. Nothing has been lost, only added.

• ZotQuery now deals with Group Libraries. By default, ZQ indexes and caches items in Group Libraries that you are a member of, you can, however, disable this feature. Items in Group Libraries will now open appropriately from ZotQuery as well.
• Dates have been added to the searches of the "general", "titles", and "creators" scope.
• Better, cleaner support of multi-term queries
• smaller code footprint

For those power-users who want more fine-grained control, I will post later how to alter the code to change what all is searched with the zot/z keyword, as well as how to disable indexing of Group Library Items. 
 
Till then, go and update!
stephen

Edited April 8, 2014 by smarg19

[greghoch]

That was fast! Thanks for the great update!

[smarg19]

Ok. Version 7.0.1 is now up. It fixes a small bug in the zot query affecting some users. 

 

Also, here's how to change the scope of the zot query. This will be a "power-user" feature, which will require altering the code. So know that up front. If you aren't comfortable with that, I would recommend getting more familiar and comfortable with the myriad of specific queries. But, to the issue:

 

Open the zq_utils.py file in a text editor or code editor (I use Sublime Text, tho TextMate will also work. As will many others no doubt). You need to find the zot_string function (it should be at line 112). It will look like this (in version 7.0.1):

def zot_string(d, scope='general'):

    """Convert key values of item into string for fuzzy filtering"""

    def get_datum(d, key, val):

        try:

            return [d[key][val]]

        except KeyError:

            return []

        except TypeError:

            return [x[val] for x in d[key]]


    _list = []

    if scope == 'general': 

        _list += get_datum(d, 'data', 'title')

        _list += get_datum(d, 'creators', 'family')

        _list += get_datum(d, 'data', 'collection-title')

        _list += get_datum(d, 'data', 'container-title')

        _list += get_datum(d, 'data', 'issued')

        _list += get_datum(d, 'name', 'zot-collections')

        _list += get_datum(d, 'name', 'zot-tags')

        _list += d['notes']

    elif scope == 'titles':

        _list += get_datum(d, 'data', 'title')

        _list += get_datum(d, 'data', 'collection-title')

        _list += get_datum(d, 'data', 'container-title')

        _list += get_datum(d, 'data', 'issued')

    elif scope == 'creators':

        _list += get_datum(d, 'creators', 'family')

        _list += get_datum(d, 'data', 'issued')

    elif scope == 'notes':

        _list += d['notes']

    elif scope == 'in-collection':

        _list += get_datum(d, 'name', 'zot-collections')

    elif scope == 'in-tag':

        _list += get_datum(d, 'name', 'zot-tags')

    elif scope == 'attachments':

        _list += get_datum(d, 'name', 'attachments')


    _list = [unicode(x) for x in _list]

    _str = ' '.join(_list)

    uni = to_unicode(_str)

    return uni

You will be changing the code under the if scope == 'general': section. This defines what data is included in the zot/z query. As you can see, it currently grabs most of the important data: 

• title
  

• author(s) last name
  

• book/journal title
  

• pub date
  

• Zotero collection(s) name
  

• Zotero tag(s) name
  

• any notes
  

 

This is clearly a broad swath of data. If you have alot of long notes, for example, this will decrease the searching speed, but might not be helpful to you. To change, you need only delete the lines that add those items. You can limit the scope simply by deleting any line(s) that add data you don't want. 

Edited April 9, 2014 by smarg19

[deanishe]

FWIW, it might be a better idea to prepend unwanted lines with # (which will make them inactive) rather than deleting them: makes it easier to turn them back on if you made a whoopsie.

[smarg19]

FWIW, it might be a better idea to prepend unwanted lines with # (which will make them inactive) rather than deleting them: makes it easier to turn them back on if you made a whoopsie.

Good point. Agreed.

[greghoch]

Thanks for the details! I am still having problems with searching with 'author year'. To find this zotero item 

 

Einstein 1935 Can quantum-mechanical description of physical reality be considered complete? 

 

I can use 'Einstein 1935' but 'Ein 1935' does not work. My understanding of the fuzzy search was that it should work, no?

[deanishe]

Stephen doesn't have the "fuzziest" option (i.e. mimicking Alp's behaviour) turned on, so it currently won't work with multi-word queries.

 

"Einstein 1935" presumably matches because it's an exact substring.

 

We're looking at how best to handle multi-word queries, but for now, you can turn full fuzziness back on yourself.

 

You'll need to extract the alfred-workflow.zip file in ZotQuery's root directory, edit line 979 to match_on=MATCH_ALL (i.e. remove ^ MATCH_ALLCHARS), and zip it back up again.

[smarg19]

Ok. I've just posted version 7.1 to Packal. This should fix the multi-term query issues. Essentially, I "strain" the results for each term in the query string. So, to keep things as fast as possible, I would recommend putting the more specific term up front. I will think over ways to determine which term is most important, and maybe ZotQuery can reorder words for you. But for now, it will at least match multi-word queries with fuzziness. 

 

For example, the query "ein qmd" would match against "Einstein 1935 Can quantum-mechanical description of physical reality be considered complete?" using different rules on different runs. 

 

I'll continue to try to improve performance, but this should work well for a majority of users.

 

stephen

[greghoch]

Works like a charm! I also checked the group thing and I am not sure whether it's working correctly. Is it possible that it was fine in 7.0 but not in 7.1? Here are two things I observed:

1) Reveal for group items does not seem to work and I think it did in 7.0. At least it worked once when I tried and now it doesn't anymore.

2) Looking at the json file, the group or library info seems to be saved as part of the "zot-collections" array. Each of the objects in the list contain an entry for "library_id" and "group". This doesn't really match the zotero database and creates some problems. The group and the associated library id is a property of the item (maybe also of a collection but that is irrelevant here). So it would make more sense to save it together with "id" and "key" as item properties. This matters particularly when an item is in no collection. For these items your cached data doesn't contain any information on the group/library. Does that make sense or am I missing something?

 

 

Also as a side note: I am happy with the solution to modify the code and change the search term. The annoying thing is that you have to do it each time you update zotquery. A better approach would be to add a setting to the 'settings.json' file, which allows the user to chance what is included in the search with zot and potentially other things like what is displayed in the title and subtitle of the Alfred feedback (e.g. someone might wants to add the group or journal title the displayed feedback). These settings could be only accessible by modifying the 'settings.json' file and not through the interactive interface you provide for the location of the database and the API access (similar to ST settings). 

[deanishe]

Unless your workflow/app is aimed at coders, asking users to edit JSON files all too often ends in tears.

 

It's very easy to get the syntax wrong, even if you're using a programmer's editor with syntax highlighting, and if you open a JSON file in TextEdit, it will be broken by default thanks to TextEdit's Smart Quotes.

[greghoch]

It was more a suggestion for the "power users" targeted by the description to change the python code. The solution to change works and I am fine with it but everyone who makes that change has to redo it for each update. The settings file would solve that problem. But I agree that it is not a good solution for the average user (as is changing the python code).

Edited April 10, 2014 by LeChuck

[deanishe]

Moving such things into the settings file is probably a good idea given how often Stephen updates the workflow

 

But it should probably remain an undocumented "power-user" feature until there's a UI, or use a simpler file format than JSON. INI, perhaps.

Edited April 10, 2014 by deanishe