Alfred performs many actions synchronously, but some are run asynchronously. File search is an example of a query that's run asynchronously, as the results are at the mercy of the speed of the index it's provided by, which could be a slow connected drive or a slightly corrupted OS X metadata index. This allows Alfred to return the basic results lightning fast, and add the further results as soon as they become available.
I would recommend reindexing your Mac, as this may help improve the speed at which results are returned; You can do this in Alfred's Advanced preferences by clicking the "Rebuild OS X metadata" button and following the necessary steps.
You may also want to consider creating a file filter workflow for your bookmarks instead of including them in your default results, unless you need to access a wide range of Safari bookmarks regularly. If you're only accessing one or two, you could create a keyword-based workflow for these, and exclude bookmarks from default results, giving you cleaner and faster results
