Try opening up Alfred's preferences, then go to Workflows -> Toggl, and click the "debug" icon in the upper right of the preferences window. Try getting the status of your active timers (`tgl timers`) and look at the messages that show up in the debug pane. Testing myself (which may not say much since I haven't use Toggl in a while so I have no active tasks), the GET request to toggl.com to load the timers takes 1-2 seconds. Using `tgl timers` again within a short time should use the cache and not make a new GET request.
Is the GET request taking a long time, or does it look like it might be something else?