Alfred stopped finding files on Google Drive File Stream

[vitor]

Are any of you comfortable enough with the terminal to try running the script directly?

I can’t reproduce your hanging, and perhaps running directly will provide some insight. A normal output would be it showing a JSON output of your Drive files.

[physiltant]

So initially Alfred just found nothing. I then installed this workflow which seems to solve things for the most part. Now the behaviour I'm experiencing is that some files/folders simply aren't being found by Alfred, even when I make them available for offline access. Many other files are being found fine, even if online only. I've added the Volumes/GoogleDrive to the search scope. Troubleshooting File Search in Alfred doesn't reveal anything. Even mdutil -i on /Volumes/GoogleDrive via Terminal doesn't make a difference. It all seems a bit arbitrary, and is very frustrating. I can't even revert to Backup & Sync which was perfectly fine for me.

[vitor]

@Yiching @physiltant What you’re both reporting has already been discussed in this thread and others. We’re all aware of the problems; repeating them without new information makes the conversation harder to follow.

If you can answer my questions and try the solutions, perhaps we’ll be able to get new insights. What’s needed now is focus on what might work.

Edited August 16, 2021 by vitor

[mishamsk]

@vitor I think this https://apple.stackexchange.com/a/424445/399192 explains what is happening. Based on the investigation there is a good chance that Google can fix it that on their end (enable the fixed SMB port). I can at least confirm that manually enabling the spotlight index works (although it's been 30 minutes and it hasn't built the full index for me yet). I understand that it won't last, but at least I am hopeful it is not one of those things that google takes eternity to fix

[Henry31]

17 hours ago, vitor said:

Are any of you comfortable enough with the terminal to try running the script directly?

I can’t reproduce your hanging, and perhaps running directly will provide some insight. A normal output would be it showing a JSON output of your Drive files.

 

I changed the Gdrive_path variable to the path of my Gdrive Folder.

 

The script takes a lot of time... Maybe we should add a timeout somewhere...

 

But after 10 or 20 minutes I got a long output with all of my files. I give you an excerpt of that:

 

{"uid":"/Users/henryauffahrt/Google Drive - AufBau/Shared drives/50_Kunden-Projekte/P21010_domain.net - Herr XXXXXX/Sourcecode/templates/OM-Aufbau/themes/my-evo/bootstrap.css","type":"file","title":"bootstrap.css","subtitle":"/Users/henryauffahrt/Google Drive - AufBau/Shared drives/50_Kunden-Projekte/P21010_domain.net - Herr XXXXXX/Sourcecode/templates/OM-Aufbau/themes/my-evo/bootstrap.css","icon":{"path":"/Users/henryauffahrt/Google Drive - AufBau/Shared drives/50_Kunden-Projekte/P21010_domain.net - Herr XXXXXX/Sourcecode/templates/OM-Aufbau/themes/my-evo/bootstrap.css","type":"fileicon"},"arg":"/Users/henryauffahrt/Google Drive - AufBau/Shared drives/50_Kunden-Projekte/P21010_domain.net - Herr XXXXXX/Sourcecode/templates/OM-Aufbau/themes/my-evo/bootstrap.css"},
{"uid":"/Users/henryauffahrt/Google Drive - AufBau/Shared drives/50_Kunden-Projekte/P21010_domain.net - Herr XXXXXX/Sourcecode/templates/OM-Aufbau/themes/my-evo/sourcemap.map","type":"file","title":"sourcemap.map","subtitle":"/Users/henryauffahrt/Google Drive - AufBau/Shared drives/50_Kunden-Projekte/P21010_domain.net - Herr XXXXXX/Sourcecode/templates/OM-Aufbau/themes/my-evo/sourcemap.map","icon":{"path":"/Users/henryauffahrt/Google Drive - AufBau/Shared drives/50_Kunden-Projekte/P21010_domain.net - Herr XXXXXX/Sourcecode/templates/OM-Aufbau/themes/my-evo/sourcemap.map","type":"fileicon"},"arg":"/Users/henryauffahrt/Google Drive - AufBau/Shared drives/50_Kunden-Projekte/P21010_domain.net - Herr XXXXXX/Sourcecode/templates/OM-Aufbau/themes/my-evo/sourcemap.map"},
{"uid":"/Users/henryauffahrt/Google Drive - AufBau/Shared drives/50_Kunden-Projekte/P21010_domain.net - Herr XXXXXX/Sourcecode/templates/OM-Aufbau/productdetails","type":"file","title":"productdetails","subtitle":"/Users/henryauffahrt/Google Drive - AufBau/Shared drives/50_Kunden-Projekte/P21010_domain.net - Herr XXXXXX/Sourcecode/templates/OM-Aufbau/productdetails","icon":{"path":"/Users/henryauffahrt/Google Drive - AufBau/Shared drives/50_Kunden-Projekte/P21010_domain.net - Herr XXXXXX/Sourcecode/templates/OM-Aufbau/productdetails","type":"fileicon"},"arg":"/Users/henryauffahrt/Google Drive - AufBau/Shared drives/50_Kunden-Projekte/P21010_domain.net - Herr XXXXXX/Sourcecode/templates/OM-Aufbau/productdetails"},
{"uid":"/Users/henryauffahrt/Google Drive - AufBau/Shared drives/50_Kunden-Projekte/P21010_domain.net - Herr XXXXXX/Sourcecode/templates/OM-Aufbau/productdetails/stock_OLD_B11.tpl","type":"file","title":"stock_OLD_B11.tpl","subtitle":"/Users/henryauffahrt/Google Drive - AufBau/Shared drives/50_Kunden-Projekte/P21010_domain.net - Herr XXXXXX/Sourcecode/templates/OM-Aufbau/productdetails/stock_OLD_B11.tpl","icon":{"path":"/Users/henryauffahrt/Google Drive - AufBau/Shared drives/50_Kunden-Projekte/P21010_domain.net - Herr XXXXXX/Sourcecode/templates/OM-Aufbau/productdetails/stock_OLD_B11.tpl","type":"fileicon"},"arg":"/Users/henryauffahrt/Google Drive - AufBau/Shared drives/50_Kunden-Projekte/P21010_domain.net - Herr XXXXXX/Sourcecode/templates/OM-Aufbau/productdetails/stock_OLD_B11.tpl"}

 

For data protection reasons, I changed one folder name using search and replace.

And I've added a new line. So you can see a list of items according to UID

[alfredpanda]

19 hours ago, vitor said:

Are any of you comfortable enough with the terminal to try running the script directly?

I can’t reproduce your hanging, and perhaps running directly will provide some insight. A normal output would be it showing a JSON output of your Drive files.

 

@vitor I ran the script on my MacBook Pro (13-inch, M1, 2020) and it executed in seconds, showing everything as expected ✅

 

8 hours ago, mishamsk said:

@vitor I think this https://apple.stackexchange.com/a/424445/399192 explains what is happening. Based on the investigation there is a good chance that Google can fix it that on their end (enable the fixed SMB port). I can at least confirm that manually enabling the spotlight index works (although it's been 30 minutes and it hasn't built the full index for me yet). I understand that it won't last, but at least I am hopeful it is not one of those things that google takes eternity to fix

I'm not convinced this is a workaround - when I tried this it started downloading files offline. What we're looking for is for Spotlight/Alfred to be able to index the structures and files even when streamed.

[vitor]

4 hours ago, Henry31 said:

But after 10 or 20 minutes I got a long output with all of my files.

 

Minutes?! That’s an insane amount of time. It would mean a monstrous number of files in there.

 

Which is going to be the difference between your setups and mine—because I’m only testing, I don’t have as many files. My bet is that you’re bumping against Alfred’s limit for the JSON size, leading to the hanging.

 

So let’s try a different approach. Instead of showing everything and letting Alfred filter, we’ll do the search in the Workflow. This means you’ll have to type something to search, but it should work.

 

New version. This one is easier for you to change the path @Henry31; I’ve added a Workflow Environment Variable.

[physiltant]

22 hours ago, vitor said:

@Yiching @physiltant What you’re both reporting has already been discussed in this thread and others. We’re all aware of the problems; repeating them without new information makes the conversation harder to follow.

If you can answer my questions and try the solutions, perhaps we’ll be able to get new insights. What’s needed now is focus on what might work.

 

Running 

ls -R "/Volumes/GoogleDrive/My Drive"

reveals everything as expected.

 

I tried your workflow as well and it leaves me hanging like the others (I have almost 1 TB of files, Google Drive is in the default location, I have a MBP 16", I waited over 30m)

 

I have a couple more behaviours to report if it adds any insight (remember I am using this workflow) :

 

1. I noticed that files if files show up as having synced (even if "online only") in the menu icon window (see below), they'll show up in an Alfred search. Before syncing they don't show up in Alfred's search.

 

 

2. I also noticed that there is one folder that is stubbornly refusing to show up in search: a folder called "Library" which has a bunch of folders and PDFs. Alfred seems to have an issue with the word "Library". When I rename this Library folder to anything else, it doesn't work. If I create a new folder called "Library", it doesn't work. However, if I create a *new folder* called Library OLD (or anything else), it shows up as synced in the menu bar (as above) and shows up in search. If I then rename this new folder to "Library" instead of "Library OLD" or "Libraryyy", it stays synced, but Alfred will pretend it's not called "Library". See below. Notice how the path is to "Library" as it should be, but it still calls it "Libraryyy" in search. Renaming this new folder to anything but "Library" is absolutely fine. This was my only problematic folder (as far as I've noticed so far), so I guess for me I now have a fix: create a new folder that is not called Library, and copy all my files into this new folder. This seems to work for now.

 

 

EDIT: OK I can't rename it back to Library. Only the parent "Library" folder will show up in search (with the wrong name but correct path), but none of its constituent files sync. I have to rename it to something else for it and its constituent files to sync.

Edited August 17, 2021 by physiltant
Addition

[vitor]

Thank you @physiltant. Appreciate the detailed information.

 

3 minutes ago, physiltant said:

I tried your workflow as well and it leaves me hanging like the others

 

Try the new one. I’m also interested in how long it takes to run (approximately, no need to time it exactly).

[physiltant]

8 minutes ago, vitor said:

Try the new one. I’m also interested in how long it takes to run (approximately, no need to time it exactly).

 

This seems to work! Even with "Library" haha. It takes about 5-7 seconds to find the files (I have around 1TB of files).

[vitor]

Just now, physiltant said:

This seems to work!

 

Fantastic! Now just need to wait for confirmation from @Henry31 and @alfredpanda

 

Just now, physiltant said:

Even with "Library" haha.

 

Because it’s doing a “dumb” directory traversal (no optimisation), everything will show up but it will also be slower.

 

2 minutes ago, physiltant said:

It takes about 5-7 seconds to find the files (I have around 1TB of files).

 

Could be worse, but still long. I’m thinking of saving the full file list to a file to make it faster to search.

[physiltant]

5 minutes ago, vitor said:

Could be worse, but still long. I’m thinking of saving the full file list to a file to make it faster to search.

 

Great. One nitpicky thing maybe to consider for the final version: I changed the keyword to ' (which I assume many will want to eventually do), but it shows duplicate results (I assume one from the standard search and one from your workflow). See example below. (Also not sure why Alfred is suddenly showing results without the use of a workflow - yesterday it wasn't finding anything by itself at all!)

 

[physiltant]

3 minutes ago, physiltant said:

(Also not sure why Alfred is suddenly showing results without the use of a workflow - yesterday it wasn't finding anything by itself at all!)

 

Ok all of a sudden Alfred is working absolutely fine by itself (bar the "Library" issue - which only your workflow solved). I have disabled all workflows. What's happened?

Edited August 17, 2021 by physiltant

[vitor]

47 minutes ago, physiltant said:

One nitpicky thing maybe to consider for the final version: I changed the keyword to ' (which I assume many will want to eventually do), but it shows duplicate results (I assume one from the standard search and one from your workflow).

 

The Workflow is a stopgap until Google gets its act together. It’s anyone’s choice to change the keyword, but it’s different on purpose.

 

45 minutes ago, physiltant said:

Ok all of a sudden Alfred is working absolutely fine by itself (bar the "Library" issue - which only your workflow solved). I have disabled all workflows. What's happened?

 

I’m not a file systems expert, so all I can offer is conjecture. Perhaps the fact that the Workflow traverses that directory structure makes the OS “aware” the files exist and thus they become indexable. But that should mean that even the ls -R command would have been enough for the files to show up in Alfred (perhaps in conjunction with the mdutil you did earlier).

[alfredpanda]

@vitor the indexing works! Unfortunately it's unusable for me because the search takes ~8 seconds before showing a result, and selecting a result doesn't actually open it anywhere.

 

I really appreciate your help to find a solution.

 

[vitor]

2 minutes ago, alfredpanda said:

the indexing works!

 

Great! Which means we have a path forward. Did you check if Alfred started indexing those files as well, as it did for @physiltant above?

 

4 minutes ago, alfredpanda said:

the search takes ~8 seconds before showing a result

 

Yep, next step is to try some caching. New Workflow should be done soon, but likely not today.

 

4 minutes ago, alfredpanda said:

selecting a result doesn't actually open it anywhere.

 

All your File Actions will work if you press the shortcut on the result. But yes, I still haven’t added anything on ↵ and variants. Open to specific suggestions.

[alfredpanda]

@vitor I can't see that Alfred is indexing them yet, but I'll keep trying.

 

Sorry - I misunderstood the functionality.

 

Thank you so much for your help!

[millerstevew]

@vitor, your workflow works wonders for me. I've been following this issue and just stumbled on your work around. One thing that would be tremendously helpful for me is to include the "Shared Drives" folder as well. Most of my organization's files are located in shared drives and not "my drive." 

 

I was able to modify your original workflow to make this happen. 

 

 

When I tried a similar edit in the shell script of your most recent workflow, it didn't work. I, however, am no shell script expert so a lot of other things could be going on here that I don't know about. 

 

Edited August 17, 2021 by millerstevew
improperly tagged someone

[alfredpanda]

3 hours ago, physiltant said:

 

Ok all of a sudden Alfred is working absolutely fine by itself (bar the "Library" issue - which only your workflow solved). I have disabled all workflows. What's happened?

@physiltant Is Alfred indexing your Drive folder now, since trying vitor's new workflow?

 

Are your folders that are indexed streamed, or downloaded locally?

 

Mine are all streamed, and this is what I see:

 

 

[vitor]

52 minutes ago, millerstevew said:

When I tried a similar edit in the shell script of your most recent workflow, it didn't work.

 

On the most recent version, you no longer need to (or should) edit the (ruby, not shell) script directly. Instead, change the Workflow Environment Variable.

 

39 minutes ago, alfredpanda said:

Mine are all streamed, and this is what I see

 

Seems you forgot the ' (as in 'courses) to search for files. Or does your configuration not need that?

[vitor]

New version! This one already supports caching, so you should see a major increase in speed.

 

The first time you run it, it will still be slow, because it will be building the cache. I’m not sure I’ll be able to squeeze more speed out of it, so I hope subsequent runs are fast enough.

 

Alternatively, you can run :gdrebuildcache to build the cache. A notification will show when it’s done. If it works well, in the next version I’ll add something to periodically rebuild the cache so it’s always at the ready.

 

This version also includes niceties on ↵ or ⌘↵ a result.

[physiltant]

6 hours ago, physiltant said:

 

Ok all of a sudden Alfred is working absolutely fine by itself (bar the "Library" issue - which only your workflow solved). I have disabled all workflows. What's happened?

 

Update: restarted Google Drive and Alfred no longer works on its own. Doesn't find Google Drive or any of its files at all.

[alfredpanda]

1 hour ago, vitor said:

New version! This one already supports caching, so you should see a major increase in speed.

 

The first time you run it, it will still be slow, because it will be building the cache. I’m not sure I’ll be able to squeeze more speed out of it, so I hope subsequent runs are fast enough.

 

Alternatively, you can run :gdrebuildcache to build the cache. A notification will show when it’s done. If it works well, in the next version I’ll add something to periodically rebuild the cache so it’s always at the ready.

 

This version also includes niceties on ↵ or ⌘↵ a result.

Wow - the speed after the cache is built is fantastic! This is so close to being perfect.

 

I am however now facing an issue where the same search results don't appear anymore. It seems to be finding files but not folders.

 

For example continuing with my "courses" example (I have a folder at /Volumes/GoogleDrive/My Drive/Courses) - this folder itself no longer shows. The results are from a different folder - /Volumes/GoogleDrive/My Drive/Personal - where there are files that contain the name "course" somewhere or other. There are 18 results and they all show the "Personal" folder.

 

 

If I leave a partial search in there then you can see files from the folder "Courses" show, but not the folder itself.

 

 

Thanks so much!

[vitor]

38 minutes ago, alfredpanda said:

I am however now facing an issue where the same search results don't appear anymore.

 

Should be fixed in the new one. Also added a :gdlaunchd which you only need to run once and it will add a launchd agent that will rebuild your cache every hour.

[physiltant]

12 minutes ago, vitor said:

 

Should be fixed in the new one. Also added a :gdlaunchd which you only need to run once and it will add a launchd agent that will rebuild your cache every hour.

 

Looking forward to trying this. It's not working at the moment but I think it's because I just reinstalled Google Drive so I have to wait a little for it to load files or something and I should try again later - is that right?