Keep clipboard history forever

[DevMan]

The ability to keep clipboard history forever or set limit to chosen count of stored records instead of time (3 months) is highly needed.

Edited November 19, 2017 by DevMan

[deanishe]

On 19/11/2017 at 1:31 AM, DevMan said:

set limit chosen count of stored record

 

+1.

 

This is the way most similar software works, and makes more sense in terms of any potential downsides of a large history.

 

A significantly higher limit for plain text items than for images also seems reasonable and useful.

 

Edited November 27, 2017 by deanishe
Typo

[DevMan]

On 19/11/2017 at 1:48 AM, deanishe said:

A significantly higher limit for plain text items than for images also seems reasonable and useful

+1
actually this is the most needed part: limiting time to store images/files/etc is almost ok. but storing plain text is not a problem at all.

Edited November 27, 2017 by deanishe
Abusing mod powers to correct my typos

[mawek]

+1

clipboard history is the most useful feature in alfred for me and 3 months constraint is kind of limiting

[Andrew]

While I'm not completely closed to the idea of increasing potential capacity of the clipboard history (search performance implications aside), I'd like to understand when 3 months of text isn't enough.

 

If you use an item from the clipboard history, it moves to the top of the list, effectively resetting the 3 month timer on that item. So really, the 3 month limit is "items in the clipboard history which I haven't used in the last 3 months".

 

Anything you need longer than this would surely be better suited to a snippet? (It's extremely easy to convert a clipboard item to a snippet by using cmd+s on it in the clipboard history view).

 

Cheers,

Andrew

[mawek]

I use snippets for most frequently used texts (account number, basic commands that for some reason i don't want to create bash aliases for, etc...).

But for time to time (and this happens more than i'd expect), I find myself searching in clipboard some text I know i copied some time ago. This might be some sql script i was working on some time ago, or some shell command.. It's not frequently used 'snippet', its just some text, that now i have to google again.

For example couple of months ago I was working a lot with docker so I copied&pasted docker commands a lot. It didn't have value for me to create snippet for docker at that time because it was in my history all the time. But then i switched projects. And now, I occasionally need to exec some command in docker and i know i should have that command somewhere in the clipboard history but it's not there anymore. Now I might create snippet for that text (since apparently I will be using it probably more often), but I don't want to create snippets for such texts in advance..

Another case is that couple of months ago i was working on some sql script. Now i need something similar so I just want to search it in the clipboard history, paste and refactor it. Again - I don't think that this is the case for snipped. I will create snippet for it once i see that I use it too often. But I don't know that in advance.

[deanishe]

1 hour ago, Andrew said:

would surely be better suited to a snippet?

 

From my reading of the situation, a snippet is too formalised and permanent (if only perceptually).

 

I mean, you may often run similar, but not the same commands. No concrete command incantation suggests itself as worthy of being permanently memorialised in a snippet, and trying to create a generalised snippet around that command may require a fairly substantial investment of time and cognitive effort.

 

Often, just trying to think of a trigger (which a lot of people mentally treat as "forever") is too much in that regard. (Which triggers should you assign to the dozen or so different rsync commands you use semi-regularly?)

 

I'm not trying to make a case for anything in particular here (I find my shell history is fine for things like this), just explaining my understanding of what users might be getting at.

 

[DevMan]

Snippet is something you use often. But history is just a history - your trip in the past, your memory, etc. And I can’t find any reasonable point to limit it by time. 

 

Storing/searching plain text does not require a lot of space and not a problem for any more or less modern mac. And it could be user option how many mb/gb use for storage. 

If you have any concerns about search performance you can do the instant search within last X months and the show “search X months older” and then show “search X months older” and so on. 

[Cody]

+1 for storing text history forever. I frequently search for something I remember having on my history more than 3 months ago but wouldn't want to save as a snippet.

[Vero]

On 27/11/2017 at 1:40 AM, deanishe said:

No concrete command incantation suggests itself as worthy of being permanently memorialised in a snippet, and trying to create a generalised snippet around that command may require a fairly substantial investment of time and cognitive effort.

@deanishe Just coming back to this to add something that might be beneficial to @Cody and others reading the thread too...

 

If there's something in your clipboard you want to keep, press Cmd + S to save it as a snippet, give it a name and hit save; There's no need to set a trigger for *every* snippet you create.

 

I've got a collection of non-expanding snippets just called "links and stuff" that are various links to posts, snips of code I need every so often, but have no intention of remembering a snippet trigger for.

 

As of 3.6 (currently in pre-release), it's possible to switch the matching to include snippet content, so you'll need to set a name, but it doesn't need to be mind-blowingly complicated. Once you're in the Clipboard Viewer, type part of the snippet text and it'll appear in your results.

 

 

With this option, there's less friction in storing a snippet (which also has the benefit of it being synced/backed up if syncing is enabled) without the risk of storing an infinitely ballooning clipboard history database.

 

Cheers,
Vero

[mawek]

Does this mean that 3 months max restriction won't be eliminated ?

[TheBloke]

On 26/11/2017 at 11:59 PM, Andrew said:

While I'm not completely closed to the idea of increasing potential capacity of the clipboard history (search performance implications aside), I'd like to understand when 3 months of text isn't enough.

 

If you use an item from the clipboard history, it moves to the top of the list, effectively resetting the 3 month timer on that item. So really, the 3 month limit is "items in the clipboard history which I haven't used in the last 3 months".

 

Anything you need longer than this would surely be better suited to a snippet? (It's extremely easy to convert a clipboard item to a snippet by using cmd+s on it in the clipboard history view).

 

Cheers,

Andrew

 

I'm a long term Alfred user, but somewhat on-and-off-again due to my coming in and out of using macOS.  I'm now back in macOS and hopefully for a good while now.  And I signed up at this forum specifically to post this feature request, which has been one thing that's bugged me about Alfred's Clipboard feature since I first stated using it in 2012 or so.

 

On Windows I use Ditto and I have my expiration set at 12 months, but that was only for space considerations as  Ditto doesn't have Alfred's ability to give different time limits to text versus image data.  If it did, I would set Text to "forever".

 

Why not use snippets?  I do.  But the point of the forever history for me is to retrieve things I didn't know I needed to make a snippet.  "What was that long command I ran on server X last year? I can't access the server but I know I copied it at least once.. it started with.. <SEARCH> that's it!"   This doesn't happen every day, but it does happen periodically.  I've pulled long-ago copied data out of my Ditto clipboard history many times.  Sometimes I even need to browse through dozens of results, eg if I'm looking for a phone number I thought I'd only need once, and all I have to search for is the first two digits, 07.  But I can very often work out the right entry by looking at the date it was copied.   

 

The important thing is that I have the option, and in practice I've found it very useful.  I don't want to have to think, every time I copy something, "should I make this a snippet just in case I need it in 9 months?"  And more often or not I have no idea that I'll need it again, until I do.  I have my shell history (Bash and ZSH) set to never truncate for exactly the same reason, and that is regularly invaluable.

 

Beyond that, I honestly don't understand the reason not to add the option?  The default will always be low; 1 month or whatever it is now.  Only people who actually want longer need set it, and as they have the flexibility to only do so only for Text, there's really no space worries.  

 

So I have to say I find it surprising and a little disappointing that there appears to be so much resistance to implementing such a simple feature.  Lots of people are asking for it.  It's surely easy to do.  And it need not affect anyone who doesn't want it.  Why not put it in?

 

If there's some technical limitation - Alfred gets really slow when there's too much history, perhaps?  - then just put a warning up and let us take our own chances with it.  But if you're using SQLite or similar I really can't imagine there'd be a problem - it's certainly not for Ditto, which searches a huge DB just fine.

 

At the moment I am regularly checking every new clipboard manager that I discover to see if it can replace this part of Alfred, and give me the forever history that I want.  I just wish I didn't need to and could forget about it and just always use Alfred, which is perfect in most every other way.

 

Thanks very much.

Edited February 13, 2018 by TheBloke

[Andrew]

It's likely that in the future, the maximum persistence time for clipboard history will be increased.

 

Without going deeply into the architectural details, the reason this isn't a simple "turn up the volume to unlimited" type change is because there is are significant performance considerations.

 

It doesn't matter if setting the maximum clipboard history makes searching the clipboard history slower, this is, after all, the user's prerogative. The fundamental issue is more about maintaining an extremely large full text index on inserting new text data every single time you cmd+c, leading to higher idle CPU usage (and potentially lower battery life).

 

Alfred is intrinsically extremely lightweight, so before providing a longer clipboard history, a number of architectural changes would have to happen. For example, one solution could be to automatically archive data every week into a different "long term" store. This would allow Alfred's day to day usage to be extremely fast and low resource use, while providing access to those "every so often" needed items you copied 2 years ago.

 

To be clear on timescales, any architectural change won't happen on a maintenance release - but rest assured that this feature request is on my radar.

 

Cheers,

Andrew

[mawek]

I understand performance implications, but I personally would be more than happy to pay potential performance/battery price in order to achieve longer history.

So to circle back to the original question - why can't you add couple of more options - as optional choice (as it is now) before you implement full-scale solution? I believe users can decide themselves whether they are willing to pay performance price or not. In the worst case scenario I can always switch back to the current limit.

[TheBloke]

Thanks very much for the response and explanation, Andrew.

 

Personally I would be perfectly fine with taking my chances with having increased CPU, and battery is no concern on my desktop.  Hence my mention of a warning when the feature was enabled, so that we could potentially get the feature quicker for those who don't mind potential downsides, and the warning would keep away those who do worry about that.

 

I don't know your code of course, but I do know that other software is able to manage very large data sets in SQLite with no problem.

 

My Ditto database is now nearly 2GB, and I perceive no slowdown on Control-C, nor excess CPU usage.  It simply does an INSERT(..) to its SQLite tables for each new copy, and lets SQLite handle the indexing.  SQLite is fine with far larger databases than this.  I think Ditto does its purging of old data, beyond the expiration threshold, asynchronously (once an hour or once a day) so as not to add any extra load to each copy.

 

I'm on a fast SSD which does help of course.  The only problem I get with that huge Ditto DB is that I've put it on a Google Drive synced folder, so it's backed up, and that sync now takes a long time.  But of course that's primarily because Ditto doesn't allow different storage periods for Text vs Images.  If it did, I reckon my DB would be no more than a couple of hundred MB, which is trivial.

 

So while I do understand and respect you wanting to absolutely ensure performance and scalability, my personal experience is that other clipboard managers are able to handle it fine using the same DB backend as you use, and without needing to archive to multiple, date-based tables.  If anything I would have thought it would be easier in Alfred, given its (smart) distinction between Text and Image.

 

Anyway I don't mean to tell you your business.   I'm pleased to hear that it's on your radar, even if we do have to wait a bit longer.   I lost all my prior clipboard history when I came back to my Mac after >3 months away, so I'm starting from scratch.  But that does mean that if the new release happened within the next 3 months, it wouldn't make any difference to me anyway   

 

Thanks again for the fast response, and all your great work on Alfred.

 

 

Edited February 14, 2018 by TheBloke

[Andrew]

4 minutes ago, mawek said:

I believe users can decide themselves whether they are willing to pay performance price or not.

 

With 8 years of experience helping our Alfred users daily, I can say that this isn't always true. A very likely scenario is that a user will blindly select "unlimited", even though they don't need it, ignore the implications warning, then start complaining about Alfred appearing in Apple's "Significant Energy" menu.

 

As I said, I will very likely add in a longer expiry, but not as a trivial "turn up the volume".

[TheBloke]

Actually, thinking about my Google Drive issue - if you did implement multiple stores, and put them in different database files rather than just different tables, that would help a lot with keeping them synced to the cloud. 

 

One SQLite file for <= 3 months, one for >3 months.   Then the former would remain small and would be quick to sync each time it updated.  The latter would take longer, but it would only change once a day (presumably Alfred would handle this migration asynchronously and only periodically.)

 

I do agree with mawek that it would be nice to have the option right now, with whatever warnings/"Advanced User Only" flags might be necessary.  Or what about making it a command-line only option?  Requiring a PList edit.  I've seen other apps do that - don't add features to the UI, but put on a Wiki somewhere how to toggle from the CLI.   That solves the problem of users just turning it to "forever" then complaining; they won't even know it's there unless they see this thread, or go looking for an answer.

 

However if it has to be done with architectural changes, I can at least see one benefit - cloud syncing.  At least if it's done with multiple SQLite files rather than tables.

Edited February 14, 2018 by TheBloke

[raguay.customct]

Another way to help this issue is to have multiple stores each with a regular expression for putting clipboard history items into it and a special keyword to search in that store. That way, you can have one store for sql commands, one for websites, one for pictures, etc. Each having a regular expression check that automatically places that item into that store. Then, the user can use a keyword for searching in that store and a super keyword that searches all stores. But keep the default one to search just the standard clipboard history. I think that would give the flexibility that everyone would need, but keep the main clipboard history small enough to not bog down Alfred.

 

I believe Copied does something similar to this.

[deanishe]

21 hours ago, TheBloke said:

Actually, thinking about my Google Drive issue - if you did implement multiple stores, and put them in different database files rather than just different tables, that would help a lot with keeping them synced to the cloud. 

 

SQLite is a fundamentally awful fit for file-based syncing. Not to mention the security implications of putting something as sensitive as your clipboard history in a 3rd-party service unencrypted.

[Erik Hansen]

+1

I and several others at my company would love to have an option to have a longer (or unlimited!) clipboard history. 

[theSquashSH]

+1 I'd be willing to pay >another $30 on top of my existing Legendary license for 
unlimited clipboard history, and I'd fully accept any CPU/Memory hit necessary to get it.
 

I use Clipboard History as a general buffer for everything in my life, 
and losing everything beyond 3 months is a frequent source of headache.  
Here's a small sample of a few recent things I've lost due to history expiring:
 

• flight confirmation details
• commit summaries with commit ids (detached commits that are hard to find due to deleted branches)
• important UUIDs
• ssh public keys
• many many many file paths (lots of obscure config file paths that I never bother to remember)
• entire config files 
• blog post drafts
• comments on social media
• form fields on websites

 

It's always stuff that I don't realize at the time would be important later
so it would be pointless to try and use snippets for most of these.
 

Having a massive index of every meaningful string that's passed through my 
brain is incredibly useful.  If needed you could hide "6 months" "12 months" and "unlimited" behind an 
"Advanced settings" pane and display a big warning about potential performance 
downsides.
 

For now I just periodically back up `~/Library/Application\ Support/Alfred/Databases/clipboard.alfdb` 
to a separate folder, and merge the rows in it with a main database.  This at 
least allows me to query further back by querying the merged database directly.
 Maybe I'll build a workflow to do that if I have time, but no promises.
 

I've created a script that handles the backup of the db, merging it with an
infinite-history sqlite db in my home folder, and searching functionality.
 

 /* Delete any items that are the same in both databases */
    DELETE FROM merged_db.clipboard
        WHERE EXISTS(
            SELECT 1 FROM latest_db.clipboard latest
            WHERE $UNIQUE_FILTER
        );

    /* Insert all items from the latest_db backup  */
    INSERT INTO merged_db.clipboard
        SELECT * FROM latest_db.clipboard;

Full source: alfred-clipboard.sh (it's a fully functional infinite-history solution for Alfred, with backup, search, exporting, etc.)

One thing I want to try is periodically overwriting the Alfred internal db with my merged one,

and translating all the older timestamps so that they're artificially set to something within the last 3 months,

that way I can accumulate infinite history without Alfred attempting to expire anything.
In theory it should work fine, I'll probably update my script to do that sometime in the next week.

One thing I cant figure out though is the format of the timestamps (e.g. `584601195`), they're not valid UNIX timestamps,

they must be offsets in seconds from some custom hardcoded date.
When I tried prepending `1` to make it something like `1584601195`, it becomes a valid

unix timestamp, but the time is ~4 months in the future! If anyone has more info on the schema, let me know.
 

I also tried hacking around the limit by changing the Alfred binary directly
but unfortunately I was only able to find the limit in the .nib file (which
is useless as it's just the GUI definition).
I'd have to properly decompile Alfred it to find the actual limit logic...
 

$ ggrep --byte-offset --only-matching --text '3 Months' \
      '/Applications/Alfred 3.app/Contents/Frameworks/Alfred Framework.framework/Versions/A/Resources/AlfredFeatureClipboard.nib'
12590:3 Months

 

(Now I just have to convince the Google Chrome team to also allow storing 
browser history longer than 3 months... then the two biggest sources of 
data-loss pain in my life will be eliminated).

[mawek]

Any update on this feature? new release is out but no mention about clipboard size.. 😭

[deanishe]

@theSquashSH If you packaged your idea/script as a "Clipboard Archive" workflow, I think that would be an excellent solution to the problem.

 

Alfred gets to keep its snappy, relevant clipboard history search, and the people who want a permanent archive get one.

 

On 7/14/2019 at 6:11 PM, theSquashSH said:

One thing I cant figure out though is the format of the timestamps (e.g. `584601195`), they're not valid UNIX timestamps,

they must be offsets in seconds from some custom hardcoded date.

 

They're Mac timestamps. The Mac epoch started on 1/1/2001, not 1/1/1970, so to convert to a UNIX timestamp, add 978307200.

[theSquashSH]

I made a Workflow out of it, unfortunately the ergonomics are really bad because Alfred doesn't expose the clipboard-style preview pane UI to Workflows. With script filter output you can only display short snippets in the list.  Without the longer preview, it's really hard to tell which matches are correct and I found it next to useless.

 

Thanks for that tip about the timestamps! I completely forgot about the Mac epoch having a different start date.

 

I think the best long-term solution right now is to rewrite the Alfred sqlite db in place periodically to bump all the timestamps so that nothing ever expires.

It's a hack, but lets me keep using the nice preview pane UI and the native Alfred search instead of a script filter workflow.

The workflow can be found here: https://github.com/pirate/experiments/blob/master/Infinite Clipboard History.alfredworkflow (you'll need the script above to use it)


 

[deanishe]

14 minutes ago, theSquashSH said:

Thanks for that tip about the timestamps! I completely forgot about the Mac epoch having a different start date.

 

I think the best long-term solution right now is to rewrite the Alfred sqlite db in place periodically to bump all the timestamps so that nothing ever expires.

 

I think so, too. I'm pretty sure I knew about the Mac epoch from discussing this exact same thing before…

 

What do you think the best way to do that would be? Take every entry over a month old and make them a second apart?