clipboard viewer Text search inside clipboard history images

[Mr Pennyworth]

Clipboard History OCR for Alfred

Make images in Alfred's clipboard history searchable by their text content.

 

This post might be outdated: up-to-date readme.
Download the latest release.

Alfred's clipboard history includes images too. However, there's no way to search through them. For example, in the screenshots below, are two images in the clipboard history, but the only "searchable" information about them is dimensions and size.

This tool runs OCR every time an image is copied to clipboard, and makes the image searchable using that text.

Setup

1. Download the latest release.
2. Type .clipboard-history-ocr-install
3. Done!

Uninstall

1. Run .clipboard-history-ocr-uninstall in Alfred.
2. Delete the workflow from Alfred.

Credits

• Tesseract OCR
• Mac dilyb bundler
• Icon made by combining icons from flaticon by Pixel Perfect and Dimitry Miroliubov.

Edited November 29, 2020 by Mr Pennyworth

[deanishe]

That’s a really clever idea.

 

Unfortunately, you’ve hardcoded your own system’s paths in the Python script (e.g. /Users/sujeet), so it won’t work as-is for anyone else.

 

[Mr Pennyworth]

53 minutes ago, deanishe said:

Unfortunately, you’ve hardcoded your own system’s paths in the Python script (e.g. /Users/sujeet), so it won’t work as-is for anyone else.

Edit:

If my specific ocr.py file is included, the first invocation of the launch agent will fail. That's because the launch agent is loaded first and python file is overwritten next. I updated the workflow in two ways to fix this:

1. first create the python file and then load the launch agent
2. remove the bundled ocr.py file from the workflow (just to be safe)

Thanks so much for spotting it!

 

Original Reply:

Whoops! Didn't mean to include the `ocr.py` file.
However, it'll get overwritten when `.clipboard-history-ocr-install` is run.
Before posting I tested on another mac with a different username and it worked.

 

Here's the part where the ocr.py file gets overwritten with user-specific settings:

Edited November 5, 2020 by Mr Pennyworth
added clarification

[deanishe]

15 hours ago, Mr Pennyworth said:

Whoops! Didn't mean to include the `ocr.py` file.

 

Right, I see how it works now. I saw ocr.py was referenced in the launch agent, and there it was in the workflow folder… I didn't realise that was only your personalised copy.

 

I’m not sure you need a custom generated Python file, do you? You could get the path to the clipboard database with os.path.expanduser('~/...'), and the path to ocr.app via sys.path[0], seeing as it’s in the same directory as the Python script. Or set the launch agent’s working directory to the workflow directory.

 

But whatever.

 

This workflow is awesome!

[Mr Pennyworth]

53 minutes ago, deanishe said:

I’m not sure you need a custom generated Python file, do you? You could get the path to the clipboard database with os.path.expanduser('~/...'), and the path to ocr.app via sys.path[0], seeing as it’s in the same directory as the Python script. Or set the launch agent’s working directory to the workflow directory.

Hadn't even stopped to think about this! Just because I was generating the plist like a template in bash, I did the same with the python file. Your suggestions make so much more sense! Thanks!

 

I've updated the workflow accordingly.

[deanishe]

👍 Easier than always having to remember to remove ocr.py before exporting the workflow.

[JJJJ]

Hello,

 

It looks very useful for me. I installed it  with the keyword ".clipboard-history-ocr-install", but I still cannot find images by words in the clipboard history.

 

I am on 10.15.7 and I get the debugger output below when I try to use ".clipboard-history-ocr-install" keyword again. Is there any more information that I can provide so someone can help me to solve the issue?

 

[15:21:26.791] Logging Started...
[15:25:38.629] Clipboard History OCR[Keyword] Processing complete
[15:25:38.630] Clipboard History OCR[Keyword] Passing output '' to Run Script
[15:25:39.253] STDERR: Clipboard History OCR[Run Script] mkdir: /Users/Ceyhun/Library/LaunchAgents: File exists
xattr: ./ocr.app: No such xattr: com.apple.quarantine
/Users/Ceyhun/Library/LaunchAgents/alfred-clipboard-history-ocr.plist: service already loaded

[Mr Pennyworth]

Can you post the contents of /tmp/alfred-clipboard-history-ocr.out ?

Edited November 7, 2020 by Mr Pennyworth

[JJJJ]

6 hours ago, Mr Pennyworth said:

Can you post the contents of /tmp/alfred-clipboard-history-ocr.out ?

This is what is inside when I open the /tmp/alfred-clipboard-history-ocr.out file with xcode.

 

/System/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python: can't open file '/Users/Ceyhun/Documents/Alfred.alfredpreferences/workflows/user.workflow.256D9BE8-8BC6-4E35-8640-5D4D5EEDFEF6/ocr.py': [Errno 1] Operation not permitted
/System/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python: can't open file '/Users/Ceyhun/Documents/Alfred.alfredpreferences/workflows/user.workflow.256D9BE8-8BC6-4E35-8640-5D4D5EEDFEF6/ocr.py': [Errno 1] Operation not permitted

 

[Mr Pennyworth]

These discussions seem relevant:

https://stackoverflow.com/questions/58479686/permissionerror-errno-1-operation-not-permitted-after-macos-catalina-update

https://stackoverflow.com/a/60500021

 

Especially the second link, just the way that answer adds /bin/bash to full disk access, you could try adding /usr/bin/python to full disk access...

[JJJJ]

1 hour ago, Mr Pennyworth said:

These discussions seem relevant:

https://stackoverflow.com/questions/58479686/permissionerror-errno-1-operation-not-permitted-after-macos-catalina-update

https://stackoverflow.com/a/60500021

 

Especially the second link, just the way that answer adds /bin/bash to full disk access, you could try adding /usr/bin/python to full disk access...

I think that works. Thank you for your help. But I am not sure OCR works as it should because I get some mixed results for image names. You can find the new content in /tmp/alfred-clipboard-history-ocr.out and clipboard history for images ss below.

 

 

/System/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python: can't open file '/Users/Ceyhun/Documents/Alfred.alfredpreferences/workflows/user.workflow.256D9BE8-8BC6-4E35-8640-5D4D5EEDFEF6/ocr.py': [Errno 1] Operation not permitted
/System/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python: can't open file '/Users/Ceyhun/Documents/Alfred.alfredpreferences/workflows/user.workflow.256D9BE8-8BC6-4E35-8640-5D4D5EEDFEF6/ocr.py': [Errno 1] Operation not permitted
/System/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python: can't open file '/Users/Ceyhun/Documents/Alfred.alfredpreferences/workflows/user.workflow.256D9BE8-8BC6-4E35-8640-5D4D5EEDFEF6/ocr.py': [Errno 1] Operation not permitted
ObjectCache(0x101b55370)::~ObjectCache(): WARNING! LEAK! object 0x7ff1feed5a60 still has count 1 (id /Users/Ceyhun/Documents/Alfred.alfredpreferences/workflows/user.workflow.256D9BE8-8BC6-4E35-8640-5D4D5EEDFEF6/ocr.app/Contents/Resources/tessdata/eng.traineddatapunc-dawg)
ObjectCache(0x101b55370)::~ObjectCache(): WARNING! LEAK! object 0x7ff1fee121b0 still has count 1 (id /Users/Ceyhun/Documents/Alfred.alfredpreferences/workflows/user.workflow.256D9BE8-8BC6-4E35-8640-5D4D5EEDFEF6/ocr.app/Contents/Resources/tessdata/eng.traineddataword-dawg)
ObjectCache(0x101b55370)::~ObjectCache(): WARNING! LEAK! object 0x7ff1feed5ac0 still has count 1 (id /Users/Ceyhun/Documents/Alfred.alfredpreferences/workflows/user.workflow.256D9BE8-8BC6-4E35-8640-5D4D5EEDFEF6/ocr.app/Contents/Resources/tessdata/eng.traineddatanumber-dawg)
ObjectCache(0x101b55370)::~ObjectCache(): WARNING! LEAK! object 0x7ff2061620d0 still has count 1 (id /Users/Ceyhun/Documents/Alfred.alfredpreferences/workflows/user.workflow.256D9BE8-8BC6-4E35-8640-5D4D5EEDFEF6/ocr.app/Contents/Resources/tessdata/eng.traineddatabigram-dawg)
ObjectCache(0x101b55370)::~ObjectCache(): WARNING! LEAK! object 0x7ff1feed5b20 still has count 1 (id /Users/Ceyhun/Documents/Alfred.alfredpreferences/workflows/user.workflow.256D9BE8-8BC6-4E35-8640-5D4D5EEDFEF6/ocr.app/Contents/Resources/tessdata/eng.traineddatafreq-dawg)
ObjectCache(0x1009ce370)::~ObjectCache(): WARNING! LEAK! object 0x7fa005228690 still has count 1 (id /Users/Ceyhun/Documents/Alfred.alfredpreferences/workflows/user.workflow.256D9BE8-8BC6-4E35-8640-5D4D5EEDFEF6/ocr.app/Contents/Resources/tessdata/eng.traineddatapunc-dawg)
ObjectCache(0x1009ce370)::~ObjectCache(): WARNING! LEAK! object 0x7fa0052286f0 still has count 1 (id /Users/Ceyhun/Documents/Alfred.alfredpreferences/workflows/user.workflow.256D9BE8-8BC6-4E35-8640-5D4D5EEDFEF6/ocr.app/Contents/Resources/tessdata/eng.traineddataword-dawg)
ObjectCache(0x1009ce370)::~ObjectCache(): WARNING! LEAK! object 0x7f9ffde09340 still has count 1 (id /Users/Ceyhun/Documents/Alfred.alfredpreferences/workflows/user.workflow.256D9BE8-8BC6-4E35-8640-5D4D5EEDFEF6/ocr.app/Contents/Resources/tessdata/eng.traineddatanumber-dawg)
ObjectCache(0x1009ce370)::~ObjectCache(): WARNING! LEAK! object 0x7f9ffdcafb30 still has count 1 (id /Users/Ceyhun/Documents/Alfred.alfredpreferences/workflows/user.workflow.256D9BE8-8BC6-4E35-8640-5D4D5EEDFEF6/ocr.app/Contents/Resources/tessdata/eng.traineddatabigram-dawg)
ObjectCache(0x1009ce370)::~ObjectCache(): WARNING! LEAK! object 0x7f9ffde09710 still has count 1 (id /Users/Ceyhun/Documents/Alfred.alfredpreferences/workflows/user.workflow.256D9BE8-8BC6-4E35-8640-5D4D5EEDFEF6/ocr.app/Contents/Resources/tessdata/eng.traineddatafreq-dawg)
Error in pixCreateHeader: depth must be {1, 2, 4, 8, 16, 24, 32}
Error in pixCreateNoInit: pixd not made
Error in pixCreate: pixd not made
Error in pixFreeData: pix not defined
Error in pixSetData: pix not defined
Error in pixSetYRes: pix not defined
Error in pixGetSpp: pix not defined
Error in pixGetDimensions: pix not defined
Error in pixGetColormap: pix not defined
Error in pixCopy: pixs not defined
Error in pixGetDepth: pix not defined
Error in pixGetWpl: pix not defined
Error in pixGetYRes: pix not defined
Error in pixClone: pixs not defined
Please call SetImage before attempting recognition.
Please call SetImage before attempting recognition.

[Mr Pennyworth]

Great to see it running!
Can you tell me what exactly did you do to make the permissions error go away? (So that I can add those steps to "troubleshooting" instructions in the original post)

 

About the "Warning! Leak!" errors, it is a long standing issue with the OCR library, and I couldn't find any easy way to fix it. However, as the program is not a long-running one, but it runs every time an image is added and then quits, the leak warnings aren't something to worry about.

 

About the quality of OCR, yep, it is rough
However, I wouldn't worry too much about what is seen in the visible part of of clipboard history viewer.

Take a look at the example below: the OCR has incorrectly "recognized" weird characters like ="... Also, the actual words "member", "38", "posts" aren't even visible in the title.

But as can bee seen from the example, those words exist, and you can search using them.

For further explanation, see the image below:
This is the screenshot of the relevant entry in Alfred's clipboard database.
As you can see, the OCR library almost correctly recognized the words, but also recognized garbage characters in the avatar.

Alfred's built-in viewer (in the above screenshot) shows only the first line of each database entry, hence, in the titles that you mention, you won't be able to see the rest of the text. But that's okay IMO. When you search for "member" or "38 posts", it shows up, and that's good for rudimentary searching

 

I'm sure the tesseract library can be fine-tuned for things like "only output text that has a confidence level of more than 90%" or something like that. I just haven't really found the need to spend time to figure it out yet

 

Reminder: Can you tell me what exactly did you do to make the disk-access permissions error go away? (So that I can add those steps to "troubleshooting" instructions in the original post)

[JJJJ]

Thank you for the detailed explanation. I just wanted to be sure if it works as expected or not.

 

I gave full disk access to python as you said by drag and drop python into the full disk access window in Security&Privacy in System Preferences. I tried it for python3 for the first time but the error was about python 2.7 so tried it with 2.7 and then it worked. You can find the path of the file below.

 

/System/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7

[flome]

Getting an error when installing:

 

[09:56:41.907] STDERR: Clipboard History OCR[Run Script] mkdir: /Users/xxxxxxxx/Library/LaunchAgents: File exists
xattr: ./Tesseract.app: No such xattr: com.apple.quarantine
Load failed: 5: Input/output error

 

Running Big Sur

[Mr Pennyworth]

@flome

Seems like BigSur has some weird issues: https://developer.apple.com/forums/thread/665661

Try this command in terminal: launchctl load -wF ~/Library/LaunchAgents/alfred-clipboard-history-ocr.plist
(It is a one-time thing)

Edited December 5, 2020 by Mr Pennyworth

[alvint]

This worked until March 17 (think latest update to Monterey where they removed Python. I home-brewed back python 3 but still getting error 

STDERR: Clipboard History OCR[Run Script] mkdir: /Users/xxxxxxxx/Library/LaunchAgents: File exists
xattr: ./Tesseract.app: No such xattr: com.apple.quarantine
Load failed: 5: Input/output error

Tried the luanchctl unload and load. 

Still no luck. 

 

Hope for some advise as this is really useful to search the images in clipboard!

[TomBenz]

On 4/4/2022 at 9:34 AM, alvint said:

This worked until March 17 (think latest update to Monterey where they removed Python. I home-brewed back python 3 but still getting error 

STDERR: Clipboard History OCR[Run Script] mkdir: /Users/xxxxxxxx/Library/LaunchAgents: File exists
xattr: ./Tesseract.app: No such xattr: com.apple.quarantine
Load failed: 5: Input/output error

Tried the luanchctl unload and load. 

Still no luck. 

 

Hope for some advise as this is really useful to search the images in clipboard!

Exactly same issue. Request for an update. @Pennyworth @vitor Thanks in advance