Yeah, I see that too. It does leave a trace in the clipboard history.
The option to have the Alfred clipboard history unaffected when Alfred accesses selected text would be a great addition. That includes:
hotkey triggers using argument "Selection in macOS"
universal action triggers