Jump to content

Text search inside clipboard history images

Mr Pennyworth

By Mr Pennyworth
in Share your Workflows

Recommended Posts

Mr Pennyworth Member

Mr Pennyworth

Posted
Posted (edited)

Clipboard History OCR for Alfred

Make images in Alfred's clipboard history searchable by their text content.

 

This post might be outdated: up-to-date readme.
Download the latest release.


Alfred's clipboard history includes images too. However, there's no way to search through them. For example, in the screenshots below, are two images in the clipboard history, but the only "searchable" information about them is dimensions and size.
before-1.png
before-2.png


This tool runs OCR every time an image is copied to clipboard, and makes the image searchable using that text.
after-1.png
after-2.png
after-3.png


Setup

  1. Download the latest release.
  2. Type .clipboard-history-ocr-install
  3. Done!


Uninstall

  1. Run .clipboard-history-ocr-uninstall in Alfred.
  2. Delete the workflow from Alfred.


Credits

Edited by Mr Pennyworth
Link to comment
Mr Pennyworth Member

Mr Pennyworth

Posted
  • Author
Posted (edited)
53 minutes ago, deanishe said:

Unfortunately, you’ve hardcoded your own system’s paths in the Python script (e.g. /Users/sujeet), so it won’t work as-is for anyone else.

Edit:

If my specific ocr.py file is included, the first invocation of the launch agent will fail. That's because the launch agent is loaded first and python file is overwritten next. I updated the workflow in two ways to fix this:

  1. first create the python file and then load the launch agent
  2. remove the bundled ocr.py file from the workflow (just to be safe)

Thanks so much for spotting it! :)

 

Original Reply:

Whoops! Didn't mean to include the `ocr.py` file.
However, it'll get overwritten when `.clipboard-history-ocr-install` is run.
Before posting I tested on another mac with a different username and it worked.

 

Here's the part where the ocr.py file gets overwritten with user-specific settings:Untitled.png.ba3cb78129c2b7683c125b83f9d0aa19.png

image.thumb.png.55cbfa408daa1ef7ee90f6c79cedeef8.png

Edited by Mr Pennyworth
added clarification
Link to comment
deanishe Advanced Member

deanishe

Posted
Posted
15 hours ago, Mr Pennyworth said:

Whoops! Didn't mean to include the `ocr.py` file.

 

Right, I see how it works now. I saw ocr.py was referenced in the launch agent, and there it was in the workflow folder… I didn't realise that was only your personalised copy.

 

I’m not sure you need a custom generated Python file, do you? You could get the path to the clipboard database with os.path.expanduser('~/...'), and the path to ocr.app via sys.path[0], seeing as it’s in the same directory as the Python script. Or set the launch agent’s working directory to the workflow directory.

 

But whatever.

 

This workflow is awesome!

Link to comment
Mr Pennyworth Member

Mr Pennyworth

Posted
  • Author
Posted
53 minutes ago, deanishe said:

I’m not sure you need a custom generated Python file, do you? You could get the path to the clipboard database with os.path.expanduser('~/...'), and the path to ocr.app via sys.path[0], seeing as it’s in the same directory as the Python script. Or set the launch agent’s working directory to the workflow directory.

Hadn't even stopped to think about this! Just because I was generating the plist like a template in bash, I did the same with the python file. Your suggestions make so much more sense! Thanks! :)

 

I've updated the workflow accordingly.

Link to comment
JJJJ Member

JJJJ

Posted
Posted

Hello,

 

It looks very useful for me. I installed it  with the keyword ".clipboard-history-ocr-install", but I still cannot find images by words in the clipboard history.

 

I am on 10.15.7 and I get the debugger output below when I try to use ".clipboard-history-ocr-install" keyword again. Is there any more information that I can provide so someone can help me to solve the issue?

 

[15:21:26.791] Logging Started...
[15:25:38.629] Clipboard History OCR[Keyword] Processing complete
[15:25:38.630] Clipboard History OCR[Keyword] Passing output '' to Run Script
[15:25:39.253] STDERR: Clipboard History OCR[Run Script] mkdir: /Users/Ceyhun/Library/LaunchAgents: File exists
xattr: ./ocr.app: No such xattr: com.apple.quarantine
/Users/Ceyhun/Library/LaunchAgents/alfred-clipboard-history-ocr.plist: service already loaded

Link to comment
JJJJ Member

JJJJ

Posted
Posted
6 hours ago, Mr Pennyworth said:

Can you post the contents of /tmp/alfred-clipboard-history-ocr.out ?

This is what is inside when I open the /tmp/alfred-clipboard-history-ocr.out file with xcode.

 

/System/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python: can't open file '/Users/Ceyhun/Documents/Alfred.alfredpreferences/workflows/user.workflow.256D9BE8-8BC6-4E35-8640-5D4D5EEDFEF6/ocr.py': [Errno 1] Operation not permitted
/System/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python: can't open file '/Users/Ceyhun/Documents/Alfred.alfredpreferences/workflows/user.workflow.256D9BE8-8BC6-4E35-8640-5D4D5EEDFEF6/ocr.py': [Errno 1] Operation not permitted

 

Link to comment
JJJJ Member

JJJJ

Posted
Posted
1 hour ago, Mr Pennyworth said:

These discussions seem relevant:

https://stackoverflow.com/questions/58479686/permissionerror-errno-1-operation-not-permitted-after-macos-catalina-update

https://stackoverflow.com/a/60500021

 

Especially the second link, just the way that answer adds /bin/bash to full disk access, you could try adding /usr/bin/python to full disk access...

I think that works. Thank you for your help. But I am not sure OCR works as it should because I get some mixed results for image names. You can find the new content in /tmp/alfred-clipboard-history-ocr.out and clipboard history for images ss below.

 

3goVhzs.jpg

 

/System/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python: can't open file '/Users/Ceyhun/Documents/Alfred.alfredpreferences/workflows/user.workflow.256D9BE8-8BC6-4E35-8640-5D4D5EEDFEF6/ocr.py': [Errno 1] Operation not permitted
/System/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python: can't open file '/Users/Ceyhun/Documents/Alfred.alfredpreferences/workflows/user.workflow.256D9BE8-8BC6-4E35-8640-5D4D5EEDFEF6/ocr.py': [Errno 1] Operation not permitted
/System/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python: can't open file '/Users/Ceyhun/Documents/Alfred.alfredpreferences/workflows/user.workflow.256D9BE8-8BC6-4E35-8640-5D4D5EEDFEF6/ocr.py': [Errno 1] Operation not permitted
ObjectCache(0x101b55370)::~ObjectCache(): WARNING! LEAK! object 0x7ff1feed5a60 still has count 1 (id /Users/Ceyhun/Documents/Alfred.alfredpreferences/workflows/user.workflow.256D9BE8-8BC6-4E35-8640-5D4D5EEDFEF6/ocr.app/Contents/Resources/tessdata/eng.traineddatapunc-dawg)
ObjectCache(0x101b55370)::~ObjectCache(): WARNING! LEAK! object 0x7ff1fee121b0 still has count 1 (id /Users/Ceyhun/Documents/Alfred.alfredpreferences/workflows/user.workflow.256D9BE8-8BC6-4E35-8640-5D4D5EEDFEF6/ocr.app/Contents/Resources/tessdata/eng.traineddataword-dawg)
ObjectCache(0x101b55370)::~ObjectCache(): WARNING! LEAK! object 0x7ff1feed5ac0 still has count 1 (id /Users/Ceyhun/Documents/Alfred.alfredpreferences/workflows/user.workflow.256D9BE8-8BC6-4E35-8640-5D4D5EEDFEF6/ocr.app/Contents/Resources/tessdata/eng.traineddatanumber-dawg)
ObjectCache(0x101b55370)::~ObjectCache(): WARNING! LEAK! object 0x7ff2061620d0 still has count 1 (id /Users/Ceyhun/Documents/Alfred.alfredpreferences/workflows/user.workflow.256D9BE8-8BC6-4E35-8640-5D4D5EEDFEF6/ocr.app/Contents/Resources/tessdata/eng.traineddatabigram-dawg)
ObjectCache(0x101b55370)::~ObjectCache(): WARNING! LEAK! object 0x7ff1feed5b20 still has count 1 (id /Users/Ceyhun/Documents/Alfred.alfredpreferences/workflows/user.workflow.256D9BE8-8BC6-4E35-8640-5D4D5EEDFEF6/ocr.app/Contents/Resources/tessdata/eng.traineddatafreq-dawg)
ObjectCache(0x1009ce370)::~ObjectCache(): WARNING! LEAK! object 0x7fa005228690 still has count 1 (id /Users/Ceyhun/Documents/Alfred.alfredpreferences/workflows/user.workflow.256D9BE8-8BC6-4E35-8640-5D4D5EEDFEF6/ocr.app/Contents/Resources/tessdata/eng.traineddatapunc-dawg)
ObjectCache(0x1009ce370)::~ObjectCache(): WARNING! LEAK! object 0x7fa0052286f0 still has count 1 (id /Users/Ceyhun/Documents/Alfred.alfredpreferences/workflows/user.workflow.256D9BE8-8BC6-4E35-8640-5D4D5EEDFEF6/ocr.app/Contents/Resources/tessdata/eng.traineddataword-dawg)
ObjectCache(0x1009ce370)::~ObjectCache(): WARNING! LEAK! object 0x7f9ffde09340 still has count 1 (id /Users/Ceyhun/Documents/Alfred.alfredpreferences/workflows/user.workflow.256D9BE8-8BC6-4E35-8640-5D4D5EEDFEF6/ocr.app/Contents/Resources/tessdata/eng.traineddatanumber-dawg)
ObjectCache(0x1009ce370)::~ObjectCache(): WARNING! LEAK! object 0x7f9ffdcafb30 still has count 1 (id /Users/Ceyhun/Documents/Alfred.alfredpreferences/workflows/user.workflow.256D9BE8-8BC6-4E35-8640-5D4D5EEDFEF6/ocr.app/Contents/Resources/tessdata/eng.traineddatabigram-dawg)
ObjectCache(0x1009ce370)::~ObjectCache(): WARNING! LEAK! object 0x7f9ffde09710 still has count 1 (id /Users/Ceyhun/Documents/Alfred.alfredpreferences/workflows/user.workflow.256D9BE8-8BC6-4E35-8640-5D4D5EEDFEF6/ocr.app/Contents/Resources/tessdata/eng.traineddatafreq-dawg)
Error in pixCreateHeader: depth must be {1, 2, 4, 8, 16, 24, 32}
Error in pixCreateNoInit: pixd not made
Error in pixCreate: pixd not made
Error in pixFreeData: pix not defined
Error in pixSetData: pix not defined
Error in pixSetYRes: pix not defined
Error in pixGetSpp: pix not defined
Error in pixGetDimensions: pix not defined
Error in pixGetColormap: pix not defined
Error in pixCopy: pixs not defined
Error in pixGetDepth: pix not defined
Error in pixGetWpl: pix not defined
Error in pixGetYRes: pix not defined
Error in pixClone: pixs not defined
Please call SetImage before attempting recognition.
Please call SetImage before attempting recognition.

Link to comment
Mr Pennyworth Member

Mr Pennyworth

Posted
  • Author
Posted

Great to see it running!
Can you tell me what exactly did you do to make the permissions error go away? (So that I can add those steps to "troubleshooting" instructions in the original post)

 

About the "Warning! Leak!" errors, it is a long standing issue with the OCR library, and I couldn't find any easy way to fix it. However, as the program is not a long-running one, but it runs every time an image is added and then quits, the leak warnings aren't something to worry about.

 

About the quality of OCR, yep, it is rough :(
However, I wouldn't worry too much about what is seen in the visible part of of clipboard history viewer.

Take a look at the example below: the OCR has incorrectly "recognized" weird characters like ="... Also, the actual words "member", "38", "posts" aren't even visible in the title.

But as can bee seen from the example, those words exist, and you can search using them.

image.thumb.png.bd3fd858f661469786c9c43982253564.png

For further explanation, see the image below:
This is the screenshot of the relevant entry in Alfred's clipboard database.
As you can see, the OCR library almost correctly recognized the words, but also recognized garbage characters in the avatar.

Alfred's built-in viewer (in the above screenshot) shows only the first line of each database entry, hence, in the titles that you mention, you won't be able to see the rest of the text. But that's okay IMO. When you search for "member" or "38 posts", it shows up, and that's good for rudimentary searching :)

image.png.1d832261a7fe956bc0b3ac4f52226c03.png

 

I'm sure the tesseract library can be fine-tuned for things like "only output text that has a confidence level of more than 90%" or something like that. I just haven't really found the need to spend time to figure it out yet :)

 

Reminder: Can you tell me what exactly did you do to make the disk-access permissions error go away? (So that I can add those steps to "troubleshooting" instructions in the original post)

Link to comment
JJJJ Member

JJJJ

Posted
Posted

Thank you for the detailed explanation. I just wanted to be sure if it works as expected or not. :)

 

I gave full disk access to python as you said by drag and drop python into the full disk access window in Security&Privacy in System Preferences. I tried it for python3 for the first time but the error was about python 2.7 so tried it with 2.7 and then it worked. You can find the path of the file below.

 

/System/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7

Link to comment
flome Member

flome

Posted
Posted

Getting an error when installing:

  

[09:56:41.907] STDERR: Clipboard History OCR[Run Script] mkdir: /Users/xxxxxxxx/Library/LaunchAgents: File exists
xattr: ./Tesseract.app: No such xattr: com.apple.quarantine
Load failed: 5: Input/output error

 

Running Big Sur

Link to comment
  • 1 year later...
alvint Helping Hand

alvint

Posted
Posted

This worked until March 17 (think latest update to Monterey where they removed Python. I home-brewed back python 3 but still getting error  

STDERR: Clipboard History OCR[Run Script] mkdir: /Users/xxxxxxxx/Library/LaunchAgents: File exists
xattr: ./Tesseract.app: No such xattr: com.apple.quarantine
Load failed: 5: Input/output error

Tried the luanchctl unload and load. 

Still no luck. 

 

Hope for some advise as this is really useful to search the images in clipboard!

Link to comment
  • 2 months later...
TomBenz Advanced Member

TomBenz

Posted
Posted
On 4/4/2022 at 9:34 AM, alvint said:

This worked until March 17 (think latest update to Monterey where they removed Python. I home-brewed back python 3 but still getting error  

STDERR: Clipboard History OCR[Run Script] mkdir: /Users/xxxxxxxx/Library/LaunchAgents: File exists
xattr: ./Tesseract.app: No such xattr: com.apple.quarantine
Load failed: 5: Input/output error

Tried the luanchctl unload and load. 

Still no luck. 

 

Hope for some advise as this is really useful to search the images in clipboard!

Exactly same issue. Request for an update. @Pennyworth @vitor Thanks in advance

Link to comment

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
Go to topic listing
×
×
  • Create New...