Alfred stopped finding files on Google Drive File Stream

[alfredpanda]

17 minutes ago, vitor said:

 

Should be fixed in the new one. Also added a :gdlaunchd which you only need to run once and it will add a launchd agent that will rebuild your cache every hour.

You've nailed it! 🙌🙌🙌

 

Please let us know if you put this on Github or somewhere we can promote it. There are a lot of people that would like this functionality 😄

[physiltant]

37 minutes ago, vitor said:

 

Should be fixed in the new one. Also added a :gdlaunchd which you only need to run once and it will add a launchd agent that will rebuild your cache every hour.

 

I'm using your new one, but I'm having the same issue as alfredpanda: some searches aren't showing up (that are showing up in your old one). It is super quick now though!

[alfredpanda]

2 minutes ago, physiltant said:

 

I'm using your new one, but I'm having the same issue as alfredpanda: some searches aren't showing up (that are showing up in your old one). It is super quick now though!

Did you actually download the new new version from his post?

 

Use this one → https://transfer.sh/1xeVpJI/Google Drive.alfredworkflow

[physiltant]

59 minutes ago, alfredpanda said:

Did you actually download the new new version from his post?

 

Use this one → https://transfer.sh/1xeVpJI/Google Drive.alfredworkflow

 

Yes, that's the one I'm using. I tried before and after running :gdlaunchd and same issue.

[physiltant]

7 minutes ago, physiltant said:

 

Yes, that's the one I'm using. I tried before and after running :gdlaunchd and same issue.

 

In fact, after reinstalling Google Drive (I waited a while first), it's now not finding anything at all...

 

Edited August 17, 2021 by physiltant

[vitor]

1 hour ago, alfredpanda said:

You've nailed it! 🙌🙌🙌

 

Please let us know if you put this on Github or somewhere we can promote it. There are a lot of people that would like this functionality 😄

 

Great! It will be on GitHub, yes, and it will have it’s own forum post. Just need to write the README, fill the URL in the Workflow, and add the auto update. I’ll post here when that happens.

 

@physiltant Use :gdlaunchd to remove the agent, then run it again to install the agent, and wait a bit (you have a lot of files). Then try again.

[physiltant]

1 hour ago, vitor said:

 

Great! It will be on GitHub, yes, and it will have it’s own forum post. Just need to write the README, fill the URL in the Workflow, and add the auto update. I’ll post here when that happens.

 

@physiltant Use :gdlaunchd to remove the agent, then run it again to install the agent, and wait a bit (you have a lot of files). Then try again.

 

Thanks for the reply. It's been just over 30m and still nothing..

 

Note that, to emulate a brand new user, I fully uninstalled both Google Drive and Alfred, restarted my Macbook, reinstalled Alfred + Powerpack, then reinstalled Google Drive from scratch, then waited 30m (just in case), then installed the workflow, removed the agent and reinstalled it as instructed, waited another 30m, and it's still not finding anything at all. Can anyone else replicate this? I wonder if it only works for those who used your older versions first (which might have done some indexing or whatever that's helped your latest version work). Anyway I'll wait another 30m and see if anything happens. Thanks for all your efforts.

Edited August 17, 2021 by physiltant

[vitor]

1 minute ago, physiltant said:

and if it only works for those who used your older versions first (which might have done some indexing or whatever that's helped your latest version work).

 

No, that would have no relation.

 

What does the debugger (set the log output to “All Information”) say?

[physiltant]

4 minutes ago, vitor said:

 

No, that would have no relation.

 

What does the debugger (set the log output to “All Information”) say?

 

I ran a couple of searches. Here's what it outputted. Sorry if I've not done it properly.

 

[00:32:26.595] Logging Started...
[00:33:16.161] Google Drive[Script Filter] Queuing argument 'd'
[00:33:16.250] Google Drive[Script Filter] Queuing argument 'de'
[00:33:16.386] Google Drive[Script Filter] Queuing argument 'dee'
[00:33:16.411] Google Drive[Script Filter] Script with argv 'de' finished
[00:33:16.412] Google Drive[Script Filter] {"items":[{"title":"Nothing found","subtitle":"Try searching something else"}]}
[00:33:16.452] Google Drive[Script Filter] Queuing argument 'deen'
[00:33:16.611] Google Drive[Script Filter] Script with argv 'deen' finished
[00:33:16.611] Google Drive[Script Filter] {"items":[{"title":"Nothing found","subtitle":"Try searching something else"}]}
[00:33:21.675] Google Drive[Script Filter] Queuing argument 'l'
[00:33:21.783] Google Drive[Script Filter] Script with argv 'l' finished
[00:33:21.785] Google Drive[Script Filter] {"items":[{"title":"Nothing found","subtitle":"Try searching something else"}]}
[00:33:22.799] Google Drive[Script Filter] Queuing argument 'b'
[00:33:22.865] Google Drive[Script Filter] Queuing argument 'bo'
[00:33:23.003] Google Drive[Script Filter] Queuing argument 'boo'
[00:33:23.024] Google Drive[Script Filter] Script with argv 'bo' finished
[00:33:23.031] Google Drive[Script Filter] {"items":[{"title":"Nothing found","subtitle":"Try searching something else"}]}
[00:33:23.136] Google Drive[Script Filter] Queuing argument 'book'
[00:33:23.186] Google Drive[Script Filter] Script with argv 'boo' finished
[00:33:23.195] Google Drive[Script Filter] {"items":[{"title":"Nothing found","subtitle":"Try searching something else"}]}
[00:33:23.347] Google Drive[Script Filter] Script with argv 'book' finished
[00:33:23.358] Google Drive[Script Filter] {"items":[{"title":"Nothing found","subtitle":"Try searching something else"}]}
[00:33:23.768] Google Drive[Script Filter] Queuing argument 'books'
[00:33:23.856] Google Drive[Script Filter] Queuing argument 'booksh'
[00:33:23.945] Google Drive[Script Filter] Queuing argument 'bookshe'
[00:33:24.059] Google Drive[Script Filter] Queuing argument 'bookshel'
[00:33:24.105] Google Drive[Script Filter] Script with argv 'bookshe' finished
[00:33:24.117] Google Drive[Script Filter] {"items":[{"title":"Nothing found","subtitle":"Try searching something else"}]}
[00:33:24.125] Google Drive[Script Filter] Queuing argument 'bookshelf'
[00:33:24.285] Google Drive[Script Filter] Script with argv 'bookshelf' finished
[00:33:24.300] Google Drive[Script Filter] {"items":[{"title":"Nothing found","subtitle":"Try searching something else"}]}
[00:33:34.837] Google Drive[Script Filter] Queuing argument 'm'
[00:33:34.949] Google Drive[Script Filter] Queuing argument 'my'
[00:33:35.005] Google Drive[Script Filter] Script with argv 'm' finished
[00:33:35.021] Google Drive[Script Filter] {"items":[{"title":"Nothing found","subtitle":"Try searching something else"}]}
[00:33:35.038] Google Drive[Script Filter] Queuing argument 'my '
[00:33:35.129] Google Drive[Script Filter] Queuing argument 'my d'
[00:33:35.196] Google Drive[Script Filter] Queuing argument 'my dr'
[00:33:35.241] Google Drive[Script Filter] Queuing argument 'my dri'
[00:33:35.376] Google Drive[Script Filter] Queuing argument 'my driv'
[00:33:35.404] Google Drive[Script Filter] Script with argv 'my dri' finished
[00:33:35.405] Google Drive[Script Filter] {"items":[{"title":"Nothing found","subtitle":"Try searching something else"}]}
[00:33:35.443] Google Drive[Script Filter] Queuing argument 'my drive'
[00:33:35.604] Google Drive[Script Filter] Script with argv 'my drive' finished
[00:33:35.624] Google Drive[Script Filter] {"items":[{"title":"Nothing found","subtitle":"Try searching something else"}]}
[00:33:58.509] Google Drive[Script Filter] Queuing argument 'f'
[00:33:58.596] Google Drive[Script Filter] Queuing argument 'fi'
[00:33:58.757] Google Drive[Script Filter] Queuing argument 'fin'
[00:33:58.783] Google Drive[Script Filter] Script with argv 'fi' finished
[00:33:58.784] Google Drive[Script Filter] {"items":[{"title":"Nothing found","subtitle":"Try searching something else"}]}
[00:33:58.824] Google Drive[Script Filter] Queuing argument 'fina'
[00:33:58.911] Google Drive[Script Filter] Queuing argument 'finan'
[00:33:58.979] Google Drive[Script Filter] Queuing argument 'financ'
[00:33:59.024] Google Drive[Script Filter] Queuing argument 'finance'
[00:33:59.183] Google Drive[Script Filter] Queuing argument 'finances'
[00:33:59.207] Google Drive[Script Filter] Script with argv 'finance' finished
[00:33:59.208] Google Drive[Script Filter] {"items":[{"title":"Nothing found","subtitle":"Try searching something else"}]}
[00:33:59.355] Google Drive[Script Filter] Script with argv 'finances' finished
[00:33:59.381] Google Drive[Script Filter] {"items":[{"title":"Nothing found","subtitle":"Try searching something else"}]}

 

Edited August 17, 2021 by physiltant

[vitor]

Run :gdrebuildcache once. What does the debugger say? Wait until you get the notification and try another search.

Check the file at ~/Library/Caches/com.runningwithcrayons.Alfred/Workflow Data/com.vitorgalvao.alfred.googledrive/cache.txt. Does it exist? Does it contain a list of you Google Drive Files?

 

Also, are you sure your Google Drive directory is at /Volumes/GoogleDrive/My Drive?

[physiltant]

6 minutes ago, vitor said:

Run :gdrebuildcache once. What does the debugger say? Wait until you get the notification and try another search.

 

How long does the notification take? It's been a couple of minutes and I haven't seen one. This is what the debugger says anyway:

 

[00:47:20.327] Google Drive[Keyword] Processing complete
[00:47:20.331] Google Drive[Keyword] Passing output '' to Arg and Vars
[00:47:20.331] Google Drive[Arg and Vars] Processing complete
[00:47:20.332] Google Drive[Arg and Vars] Passing output '' to Run Script
[00:47:20.494] Google Drive[Run Script] Processing complete
[00:47:20.499] Google Drive[Run Script] Passing output '{"items":[{"title":"Nothing found","subtitle":"Try searching something else"}]}
' to Conditional
[00:47:20.499] Google Drive[Conditional] Processing complete
[00:47:20.500] Google Drive[Conditional] Passing output '{"items":[{"title":"Nothing found","subtitle":"Try searching something else"}]}
' to Post Notification

 

7 minutes ago, vitor said:

Check the file at ~/Library/Caches/com.runningwithcrayons.Alfred/Workflow Data/com.vitorgalvao.alfred.googledrive/cache.txt. Does it exist? Does it contain a list of you Google Drive Files?

 

Yes the cache.txt file exists via that path but no it is blank

 

9 minutes ago, vitor said:

Also, are you sure your Google Drive directory is at /Volumes/GoogleDrive/My Drive?

 

Very sure

[mishamsk]

15 hours ago, alfredpanda said:

I'm not convinced this is a workaround - when I tried this it started downloading files offline. What we're looking for is for Spotlight/Alfred to be able to index the structures and files even when streamed.

What did you execute exactly when it started downloading files? I can confirm that since running mdutil -i on /path/to/googledrive, Spotlight indexed the folder and search works for offline and streamed files (see screenshot below). And so far it has survived more than 24 hours, although it is clear that any Google Drive restart (or mac restart for that matter) will break it again.

 

I think using @vitor is a great temporary workaround, but as I've said, I am assuming Google will fix this soon too.

[Henry31]

17 hours ago, vitor said:

Fantastic! Now just need to wait for confirmation from @Henry31 and @alfredpanda

 

 

@vitor you are amazing! I read through all the post that were posted here, I installed the latest version, and it works pretty well. In the beginning, I thought that it was not working, but I think it was building up the cache in the background. After a while, all files and folders will show up close to 1 second after typing in.

 

What I usually do are two things:

• "searching for a customer project folder" this is working now ✅ 
• "searching for our current google doc file like 'SEO-Sitemap - domain.com.gsheet'" but as you can see it's a .gsheet file. So this is not a file, this is physical on my disk. And this is working as well ✅

I had once a very big lag after typing in "seo" I have a huge amount of files including this keyword. The interface from Alfred freezes and I couldn't close it. After some seconds, 10 or 20 it disappears. But yeah, maybe an issue because it is still building up a cache in the background? Or because the json file in the background was getting to big, because of the huge amount of files including this keyword.

 

16 hours ago, physiltant said:

I changed the keyword to ' (which I assume many will want to eventually do)

 

I'm definitely one of these guys. 

Currently, it is a bit unusual for me. I also changed the keyword to ' , because you can activate Alfred in the "Quick File Search mode". If you then use the space bar in the Alfred window, ' is entered, and you search the whole computer.

Before it was native everything, of course. Now I have to go down once with the arrow key, or I just type the space bar a second time.

I guess you have to get used to that because the solution now is using workflow. But I think this works for me pretty well.

 

I attached two screenshots about the settings and how it look like for me when typing in the [spacebar] or keyword ' (single apostrophe/single quotation mark)

[tylerw]

19 hours ago, vitor said:

 

Great! It will be on GitHub, yes, and it will have it’s own forum post. Just need to write the README, fill the URL in the Workflow, and add the auto update. I’ll post here when that happens.

 

@physiltant Use :gdlaunchd to remove the agent, then run it again to install the agent, and wait a bit (you have a lot of files). Then try again.

 

Hey all,

So I used to use BOTH Backup and Sync AND Drive File Stream, and now that I've been forced by Google to transition and consolidate I have major Spotlight issues.

 

@vitor Your solution seems great. However, I was wondering if you could add exclusions? I have dev folders that have thousands of tiny files that I used to exclude from spotlight using Privacy before GDFS stopped working with spotlight.

[vitor]

@Henry31 Thank you for the feedback.

 

@physiltant Try this new version and do a :gdrebuildcache. See if that works to build the cache.txt file. Everyone else can skip this version, though if you switch to it everything else will work the same.

 

6 hours ago, tylerw said:

Your solution seems great. However, I was wondering if you could add exclusions?

 

Let’s worry about that when this is closer to completion.

[physiltant]

3 hours ago, vitor said:

@physiltant Try this new version and do a :gdrebuildcache. See if that works to build the cache.txt file. Everyone else can skip this version, though if you switch to it everything else will work the same.

 

Great so the cache.txt seems to get built out just fine from what I can see (I can't check every line cos I have thousands of files but it looks OK at a glance), and results show up.

 

However, the actual search has some issues. For folders with a smaller number of files, all seems fine and consistent (though the order of results is questionable and non-intuitive). But for folders with lots of files, it's not very consistent or quick: e.g. searching the same thing 3 times delivered 3 different sets of results (one of which was no results at all) - though I can't replicate this again so it was probably just the first couple of times for that search. For larger folders it also takes a few seconds (up to 10-15 seconds), and often freezes (rainbow wheel) - this I've replicated a few times. For me personally right now this is an issue because I use my Bookshelf folder (formerly Library) a lot and it has a lot of files so this will mean freezing often, but I think for many others with smaller folders this is a perfect solution, at least temporarily.

 

Again, it goes without saying we all massively appreciate your work. The above isn't a complaint, it's just feedback!

 

EDIT:

Additional point RE results. So using "Bookshelf" as an example, when your latest version does eventually list results, it seems to list the Bookshelf folder and then EVERYTHING in that folder (hundreds of files) even if the file does not have "Bookshelf" in it, which of course isn't desirable. Your original version (before the caching feature) is therefore actually quicker for this keyword and presents better results for my Bookshelf folder (it just shows files/folders with the actual keyword in it). This applies to other searches as well: it shows any folders with the keyword in it and then ALL constituent files even if they don't have the keyword in it - whereas it's best to only show files with the keyword in it.

Edited August 19, 2021 by physiltant

[vitor]

New version. This one if for everyone.

Do force a cache rebuild via :gdrebuildcache before using, because there are changes there and I need confirmation if it can handle large numbers of files.

Files are now added to the cache based on access date, and when searching only the top 100 matches will show. This is configurable via a new Workflow Environment Variable, and the lower the number, the faster results will return.

It also fixes the case mentioned above where too many files were showing.

[millerstevew]

@vitor, I've been following these updates for the past day. This newest update works great for me. (Each release has been returning results faster than the previous one.) Thank you for all of your hard work. I'll keep my eye out on future release and post back if I have any issues. 

[physiltant]

12 hours ago, vitor said:

New version. This one if for everyone.

Do force a cache rebuild via :gdrebuildcache before using, because there are changes there and I need confirmation if it can handle large numbers of files.

Files are now added to the cache based on access date, and when searching only the top 100 matches will show. This is configurable via a new Workflow Environment Variable, and the lower the number, the faster results will return.

It also fixes the case mentioned above where too many files were showing.

 

Amazing! Much faster (still 2-5 seconds for some searches but it's good enough for me and I think I just have too many files) and no longer freezes. Results are much more intuitive and relevant now as well. Thank you!

[deanishe]

17 hours ago, vitor said:

I need confirmation if it can handle large numbers of files

 

Alfred can handle filtering many tens of thousands of results without issue. You could perhaps use “Alfred filters results” and store the cache as Alfred JSON, then just pass it through in the Script Filter.

 

I can see a lot of people having so many files in their Google Drive that filtering it in Ruby will be sluggish.

[vitor]

2 hours ago, deanishe said:

Alfred can handle filtering many tens of thousands of results without issue. You could perhaps use “Alfred filters results” and store the cache as Alfred JSON, then just pass it through in the Script Filter.

That was my first approach. It’s the solution I use in CardSearch and RecentDownloads and it works great there. Definitely agree it’s the best.

 

But if you go back to the first posts, it wasn’t working for anyone testing it, the Script Filter was never finishing. At the time there was no caching but it still should’ve returned, and it only did when I asked people to try the raw script (no changes, but running it in a Terminal). That worked. I went from 15 minutes without results to a few seconds. Some have so many files, even the script took minutes. Changing to filtering results beforehand, even without caching, worked.

 

I have a vague memory (maybe you can help there, either to confirm or deny) of a post where someone bumped into an Alfred JSON size limit, so I guessed that might’ve been the problem.

 

Even when caching was introduced, for one person it didn’t build. So I switched from writing it in Ruby (getting the full path list and writing the cache) to bash find with a redirect and it worked.

 

The current version doesn’t load the full cache file before searching; it does it sequentially and stops at the (user-modifiable) limit, which is why the cache is now built in order of access time (making it closer to default Alfred behaviour). Every script optimisation is trying to get it closer to the behaviour of Alfred doing the filtering—though it can never reach that speed—because that option turned out to not be viable right with the first batch of testers.

[alfredpanda]

23 hours ago, vitor said:

New version. This one if for everyone.

Do force a cache rebuild via :gdrebuildcache before using, because there are changes there and I need confirmation if it can handle large numbers of files.

 

It definitely took a while to rebuild the cache this time - over 15 minutes (but I do have 500GB in there. Although mostly media, not a huge volume of individual files.)

 

Interestingly the quality of search has changed for me. Using my ongoing example of the folder "Courses" which sits at /Volumes/GoogleDrive/My Drive/Courses - it can no longer be seen:

 

 

EDIT:

 

What's weird is given some time it now seems to be working. Fewer results, but more relevant - it's finding the folder I want

 

 

Edited August 20, 2021 by alfredpanda

[deanishe]

2 hours ago, vitor said:

I have a vague memory (maybe you can help there, either to confirm or deny) of a post where someone bumped into an Alfred JSON size limit, so I guessed that might’ve been the problem.

 

Dunno about a JSON size limit, but STDERR/the debugger used to be limited to the size of the pipe buffer.

 

I know Andrew tested filtering with upwards of 50K results.

[vitor]

It occurred to me the issue might not be with Alfred hitting a limit in displaying the JSON, but Ruby hitting a limit in generating it.

So I did some tests, and Alfred was able to display a 1.2GB JSON with eight million items (it beachballs for almost a minute before displaying it and the scrollbar is invisible). But the JSON was generated by Ruby. I’m unable to reproduce the initial reported issues at all, even under ridiculous pressure.

This is (probably) the last test.

• Google Drive FIND
• Google Drive JSON
• Google Drive SQLITE (added later)

Only one can be installed at a time, but once you build the cache with each, you can try one or the other (they use different cache files, so there will be no conflict). On each, please do :gdrebuildcache (and wait for the notification) once before trying gd.

 

Most important considerations would be: does it work at all, and is it fast? The one you choose will be the final implementation. Any subsequent changes (which will still happen) will use that code base.

 

Pinging @alfredpanda @Henry31 @physiltant @millerstevew @tylerw

Edited August 24, 2021 by vitor

[millerstevew]

@vitor, trying both out now, I'm finding the "Google Drive FIND" workflow more performant. With the JSON workflow, I will find there is a brief 1-2 second delay at times when I invoke the workflow command. This is not an issue with the FIND workflow. 

 

A question about both: If I am searching for a file titled "2021.22.08 bulletin- final.pdf" and type "2021 bulletin", this does not return the file. Is this expected behavior?

 

I ask because previously, Alfred search would return files. I'm assuming that there is something about the logic of your workflow that doesn't treat spaces as an AND operator. (My CS knowledge is limited, so please forgive me.)