Alfred Doesn't use Proxy?

[JonB]

Hi,

 

So i was using David Ferguson's google auto complete workflow as found here: http://jdfwarrior.tumblr.com/post/40354870777/a-few-alfred-2-workflows-to-get-you-started

 

This appears to work on a 'normal' connection, however when I have a proxy setup in network preferences, it doesn't seem to work.

 

There may be nothing that alfred can do, but thought I would raise it. I guess it would be easier to be able to handle this for 'all scripts', but it may ultimately have to be solved by supporting proxies in the underlying php/ruby/etc scripts by setting the script to make use of a proxy if available a-la : http://stackoverflow.com/questions/3487845/using-http-proxy-in-php

 

Jon

[jdfwarrior]

Alfred has no issues running through a proxy. I run a lot of things on a daily basis through a proxy. I just connected to a proxy and verified that my Google Auto Complete does indeed work.

 

I'm going to move this thread back to the Workflow Creators forum since it's definitely not a bug and seems to be more of a Workflow request.

[JonB]

Interesting, is there a way I can debug this then? Would be useful to find out whats going on.

[angie]

All the alfred workflows (like google autocomplete, rotten tomatoes, weather), dont work when I'm on office network which has auto proxy discovery set. Is there a way to get around it? Everything works fine when using regular internet at home.

[angie]

Revisited this problem after a year and right now I have a crude fix for it. Created a file .proxy with "proxy_address:port". Then added the following lines to google auto suggest workflow. 

$lines = file('/Some/path/.proxy',FILE_IGNORE_NEW_LINES);

$proxy_options=array(CURLOPT_PROXY => $lines[0]);

$orig = "{query}";
$xml = $wf->request( "http://google.com/complete/search?output=toolbar&q=".urlencode( $orig ) ,$proxy_options);

The next step is to create a workflow which will toggle proxy on/off by setting the file to proxy or null. 

[rice.shawn]

You could always check into the network setup command. Look through these commands, and see if they help you out at all.

networksetup -getproxyautodiscovery Wi-Fi
networksetup -getwebproxy Wi-Fi
networksetup -getsecurewebproxy Wi-Fi

Wi-Fi is, obviously, for Wi-Fi.

 

Use  networksetup -listallnetworkservices to find others if necessary.

[deanishe]

Hi,

 

So i was using David Ferguson's google auto complete workflow as found here: http://jdfwarrior.tumblr.com/post/40354870777/a-few-alfred-2-workflows-to-get-you-started

 

This appears to work on a 'normal' connection, however when I have a proxy setup in network preferences, it doesn't seem to work.

 

There may be nothing that alfred can do, but thought I would raise it. I guess it would be easier to be able to handle this for 'all scripts', but it may ultimately have to be solved by supporting proxies in the underlying php/ruby/etc scripts by setting the script to make use of a proxy if available a-la : http://stackoverflow.com/questions/3487845/using-http-proxy-in-php

 

Jon

 

Python/Ruby/cURL etc. all transparently support proxy servers, but it has to be set in a UNIX-y manner (via http_proxy environmental variable).

 

The problem is, neither OS X nor Alfred do this.

 

Alfred has no issues running through a proxy. I run a lot of things on a daily basis through a proxy. I just connected to a proxy and verified that my Google Auto Complete does indeed work.

 

I'm going to move this thread back to the Workflow Creators forum since it's definitely not a bug and seems to be more of a Workflow request.

 

I think you'll find the workflow isn't actually using the proxy. Your machine just also allows web access without using the proxy (unless it's a transparent proxy).

 

Workflows do not use a proxy because neither OS X nor Alfred propagates proxy settings to the standard http_proxy environmental variable or otherwise provide that information in a usable manner.

 

If your machine can only connect to the web via a proxy server, workflows will not work unless the author has taken steps to ensure it does work. Which is exceptionally hard, bordering on impossible. It's certainly way beyond the scope of your average workflow. 

 

You could always check into the network setup command. Look through these commands, and see if they help you out at all.

networksetup -getproxyautodiscovery Wi-Fi
networksetup -getwebproxy Wi-Fi
networksetup -getsecurewebproxy Wi-Fi

Wi-Fi is, obviously, for Wi-Fi.

 

Use  networksetup -listallnetworkservices to find others if necessary.

 

 

That's a massive, massive can of worms (I've looked into it because I use a proxy server). The only reasonable way for workflows to use proxy servers is for Alfred to pass the proxy settings to the workflow via the environment. It's super trivial for a Cocoa app to get the proxy server, but almost impossible for a workflow script to do so. Here's why:

 

I've apparently got 11 enabled services (NICs, iOS devices, other phones, FireWire, bluetooth, VPN etc.). Being selective about it, I'd only need to check "Wi-Fi", "Ethernet 1" and "Ethernet 2" (though, of course, I could have a proxy set for my VPN etc.) , and I also need to run up to 3 commands per connection (Ethernet 2 is my LAN connection):

$ networksetup -getwebproxy 'Ethernet 2'
Enabled: No
Server: 192.168.0.1
Port: 3128
Authenticated Proxy Enabled: 0

$ networksetup -getproxyautodiscovery 'Ethernet 2'
Auto Proxy Discovery: Off

$ networksetup -getautoproxyurl 'Ethernet 2'
URL: http://earl.local/cgi-bin/proxy.pac
Enabled: Yes

So, there you have it. For a workflow to use the proxy, it needs to call networksetup up to 33 times and then to download and parse the proxy.pac file (which is JavaScript).

 

Or Alfred could make a trivial Foundation call and set the http_proxy and https_proxy environmental variables, which are automatically used by most network libraries (Ruby's, Python's, cURL etc.).

 

Seeing as my feature request got zero response, most importantly from Andrew, a simple Foundation program that can get the system proxy and return it (or set the appropriate environmental variables) sounds like an ideal candidate for the bundler.

Edited June 24, 2014 by deanishe

[rice.shawn]

a simple Foundation program that can get the system proxy and return it (or set the appropriate environmental variables) sounds like an ideal candidate for the bundler.

 

Sounds brilliant. Any takers?

[deanishe]

Yeah, Andrew, with any luck…
 
Thinking about it a bit more, it's kinda nonsensical to add such a utility to the bundler: how would it be installed if the wrapper can't access the web because it doesn't know about the proxy server…
 
Are we supposed to base64 encode the binary in all the wrappers, and then install and call it from there? That's not only ridiculous, but also the only solution that would consistently work.
 
I know folks who can only access the web via a proxy are a small minority (and usually then only part of the time), but given how simple this would be for Alfred to do and how ridiculously difficult it is to do from a workflow, and that it would need to be done in every single workflow, I really think that this is something Andrew should be doing.
 
If Alfred set the http_proxy and https_proxy environmental variables before calling a workflow (which is relatively trivial for it to do vs the hoops a workflow author not using Objective-C has to jump through), it's something the majority of workflow authors will literally never even have to think about: it will be transparently taken care of it for them by the library/program (not 100% sure that PHP does so, but Ruby, Python, cURL etc. automatically use proxies set in this fashion).
 
There's literally no way on earth that every workflow author is going to consider—let alone implement—proxy support.
 
I mean, as a workflow author, what do you say when a user says—as in this thread—"I'm behind a proxy server. How do I get your workflow to work?"
 
Do you tell them, "sorry, it would take as much effort as I expended on the workflow to get proxy support working, and it still likely wouldn't work" (assuming that's within your capabilities), and/or, "sorry, the workflow would run 5x slower if I implemented proxy support"?
 
Or do you tell them, "sorry, Alfred doesn't support proxy servers with workflows. Try talking to Andrew, who's shown zero interest in the matter so far, or look for an alternative to Alfred"?
 
To my mind, proxy support (or the lack thereof) is 100% a matter for Andrew.
 
As a library or (especially) platform provider, it's kinda your duty to take care of this kind of arcane crap, so your users don't have to (if you want your library/platform to be seen positively). (I spent a lot of time trying to build Alfred-Workflow in such a way that users wouldn't have to overly worry about Unicode—a massive source of problems in Python 2 for those that aren't familiar with it.)

 

Certainly, I don't see the point in my spending several hours trying to get my workflows to support proxy servers (and seriously hurting their performance) when no other workflows do. Sure, it's kudos for me, but are such users really going to stick with Alfred?
 
The way I see it, either Andrew has Alfred export the necessary environmental variables, or he should post a clear "Powerpack largely useless with proxy servers" warning.
 
As noted above, the only reasonable way around Alfred not supplying workflows with the necessary environmental variables is to include a Mac-native helper app in every single workflow.

 

IMO, this is a crystal-clear case of "this is something Alfred needs to do itself".

Edited June 24, 2014 by deanishe

[rice.shawn]

Thinking about it a bit more, it's kinda nonsensical to add such a utility to the bundler: how would it be installed if the wrapper can't access the web because it doesn't know about the proxy server…

 

Well, I guess the assumption would be that they're not always on a proxy and that, at some point, they can get the utilities then.

 

As a precaution, it could be part of the regular bundler installation like TN is (well, that's because the bundler uses TN).

[deanishe]

I'm not sure it's worth it, tbh.

 

To make it useable, we'd have to integrate it into the bundler/wrapper such that the proxy is automatically and transparently set. Otherwise, it's just additional boilerplate ("btw, if you want your workflow to work with proxy servers—which you should—you have to add this extra code every single time.")

 

And even then, all the other workflows that don't use the bundler still wouldn't work with a proxy, which would be a confusing situation for most users, and possibly kick off a load of bug reports ("workflows A, B and C work with my proxy, so why doesn't yours?").

 

I suppose you can leverage that into "build your workflows with bundler, then it will work everywhere!", but I really do think this is something that belongs in Alfred itself.

 

It's an environmental settings (like locale), and the environment is Alfred's responsibility.

 

It runs workflows in a UNIX environment, so it should provide a sensibly-configured one (with http_proxy, LANG and the LC_ family of variables set to match the Cocoa environment), instead of passing on launchd's almost empty environment.

 

Trying to recreate the user's environment in a UNIX-y way in the almost empty one Alfred provides is tricky (LANG) to practically impossible (http_proxy) from within a workflow when you don't have access to the Objective-C runtime.

 

[Andrew]

I have been investigating this for a while, and have a fix lined up for 2.4 where http_proxy and https_proxy will be set into the task's environment variables

[deanishe]

Marvellous news! I've spent far too many hours trying to get this to work in my workflow library without success. It can only reasonably be done from the Objective-C runtime

 

Best of all, that means that Ruby/Python/cURL will Just Work™ without workflow authors having to even think about proxies.

 

Would there be any possibility of setting the LANG variable (and perhaps the LC_* vars where appropriate) as well? It's not particularly difficult to the language from defaults, but it's not entirely straightforward, either.

 

---

I've no idea when Apple will update the system Python to Python 3, but FWIW Python 3 workflows will not work (without a lot of arcane hackery) within the C (ASCII) locale Alfred currently calls workflows with. This is a bug with Py3, IMO, but the core developers consider it a feature

[Andrew]

 

Marvellous news! I've spent far too many hours trying to get this to work in my workflow library without success. It can only reasonably be done from the Objective-C runtime

 

Best of all, that means that Ruby/Python/cURL will Just Work™ without workflow authors having to even think about proxies.

 

Would there be any possibility of setting the LANG variable (and perhaps the LC_* vars where appropriate) as well? It's not particularly difficult to the language from defaults, but it's not entirely straightforward, either.

 

---

I've no idea when Apple will update the system Python to Python 3, but FWIW Python 3 workflows will not work (without a lot of arcane hackery) within the C (ASCII) locale Alfred currently calls workflows with. This is a bug with Py3, IMO, but the core developers consider it a feature

 

 

I'm not entirely sure about the implications of setting the LANG for an NSTask (or even the best place to source this in Obj C. NSLocale will tell me the preferred locale, but not the fully qualified version I see in Terminal). I'll add a ticket and look into this. Could simply setting the lang to e.g. en_GB.UTF-8 for me fix the Python 3 issue or is this a deeper rooted issue with NSTask?

[deanishe]

I can only speak for Python, but what would work in that case is setting LANG to the preferred language, not the locale. A locale of en_GB means English language, British locale, not British English. So for users who have their language set to English but Region set to Germany/Italy/etc., their locale would be en_DE or en_IT etc., which Python (maybe other languages, too?) chokes totally on.

 

I usually set the locale based on defaults read -g AppleLanguages, not defaults read -g AppleLocale for this reason.

 

With regard to Python 3: when Apple does replace Python 2 with 3, pretty much all workflows will break and need serious editing anyway, as there are significant differences between 2 and 3, but it is extremely difficult to get Python 3 code to run in the ASCII environment that launchd/Alfred use by default. Setting LANG to anything that includes UTF-8 should run Python 3 impeccably.

 

Setting PYTHONIOENCODING=UTF-8 will fix the problem, too.

Edited June 26, 2014 by deanishe

[Andrew]

Hmm interesting, I've added that to the ticket thanks

[deanishe]

Regarding LANG/LC_*. After a bit of testing (using date and strftime), PHP works fine with en_DE (i.e. produces a German date), Python throws an exception, and the shell date program ignores it and outputs an English date (de_DE gets you a German date). Ruby doesn't natively support locales.  For both Python and PHP, it's necessary to explicitly set the locale in code in any case.

 

It's probably not worth the bother, tbh. Workflows that are locale aware seem to be few and far between, and it's hard to second-guess how bilingual users want their software to behave.

 

I wouldn't worry about the Python 3 thing yet. I don't see Apple updating to version 3 any time soon, but seeing as Python 2 has been essentially discontinued, I think it's inevitable that it will happen at some point. (Hopefully, in addition to, not instead of, Python 2).

 

I just figured it's best to be forewarned that non-ASCII input/output in an ASCII environment will cause all manner of breakages and confusion with Python 3 (in most cases, the same code will run flawlessly in a shell where the encoding is UTF-8, but fail when run in Alfred/from launchd/cron where it's ASCII), so it's worth noting down somewhere for when the day comes.

 

It's not specific to NSTask or OS X (the same problem occurs on Linux). It's just the (imo broken) way Python 3 behaves.

Edited June 26, 2014 by deanishe

[rice.shawn]

Out of curiosity, but in relation to this, does anyone have access to Yosemite? If so, what are the native versions of:

 

Python,

PHP,

Bash,

Ruby,

zsh, and

Perl?

[deanishe]

Perl?

 

I'll put money on it not being version 6

[rice.shawn]

Well, even though I don't use it now, it'd be nice for Apple to give us something that isn't in an End of Life stage like Perl 5.16 on Mavericks now. 5.20 is at least the most recent stable, but it's probably too new for Apple to bother to include it.

 

I'd like to see PHP 5.5 at least, and it would be nice to have at least Bash 4 as well.

Edited June 26, 2014 by Shawn Rice

[deanishe]

Python's a bit of a special case in this regard. The default version on OS X, 2.7.5, is a little over a year old, but 2.7.x is basically frozen and only subject to bugfixes. There won't be a 2.8, and there won't be any new features for 2.7.

 

3 was released at the end of 2008, and that's where all new features have been added. The current 3.4 has heaps of cool, new features (a few of which have been backported to 2.7).

 

Thing is, Python 3 has a fundamentally different concept of strings/Unicode (or bytes/Unicode as Py3 sees it) to Py2, and all but the most trivial Py2 code will not run on Py3 unaltered.

 

Many Linux distros now include Python 3 by default, but they also include Python 2 and that's usually the default Python (i.e. what you get when you run /usr/bin/python). I expect that Apple will do the same, and include Python 3 in addition to Python 2.

 

When that does happen, an app like Alfred is faced with the choice of sticking with the existing model of Python 2, enforcing Python 3 (as Sublime Text 3 does), or offering both. And it'd be silly not to offer Python 3, as it's the only version of the language that's gaining new features, and not just bugfixes.

 

The major gotcha of Python 2 is that it allows you to mix encoded text and Unicode without complaint, but it will die in flames if the encoded text contains non-ASCII characters. It's up to the programmer to ensure that encoded text is correctly decoded before mixing it with Unicode. I put a lot of effort into Alfred-Workflow to ensure that it only returns Unicode and correctly encodes/decodes encoded text when required (stdin/stdout, argv, URLs). In most cases, users of Alfred-Workflow won't even need to know about the difference. And it provides helpers to handle the situations where it does become important.

 

Python 3, however, decrees that all text is Unicode, so it will (try to) decode all input from stdin/argv to Unicode and encode stdout/stderr back to whatever the default encoding is. Consequently, it has to get an encoding from somewhere, and that's the environment. Most shell environments specify UTF-8, but for right or wrong, on UNIX (Linux, OS X, etc.) the "default" is ASCII, so if you run a script via launchd/cron/init.d or Alfred, a Python 3 program that runs flawlessly in a shell will die in flames if it is passed or tries to output non-ASCII text in one of those environments.

 

You can kinda sorta get around this with some fairly arcane hacking, but that's way beyond your average workflow coder's knowledge/comfort zone.

 

Basically, few workflows witten in Python 2 would work unaltered in Python 3, and if Alfred wants to support Python 3, it has to set PYTHONIOENCODING=UTF-8, otherwise there'll be loads of people complaining that they can't solve a problem that Python's smartest users haven't been able to reasonably solve…

Edited July 7, 2014 by deanishe

[Andrew]

Just a quick update, if you hadn't spotted, the Alfred 2.4 test build 276 now supports http_proxy, https_proxy and no_proxy environment variables. You can grab it from the blog:

 

http://blog.alfredapp.com/2014/07/25/alfred-v2-4-yosemite-fixes-and-many-improvements/

 

Cheers,

Andrew

[deanishe]

Sweet.

 

It works fine if I set the proxy manually via "Web Proxy (HTTP)" and "Secure Web Proxy (HTTPS)": all my Python workflows automatically use the proxy.

 

Unfortunately, it doesn't work with  Automatic Proxy Configuration, which is what I use

 

 

 

 

Alfred itself has no trouble picking up and using the proxy. I've just watched it hit media.alfredapp.com/v2update/ via the proxy log.

 

I'm not sure how common it is to use a proxy.pac configuration file, but it's how I always used to do it when I was in charge of a network.

[Andrew]

Alfred itself has no trouble picking up and using the proxy. I've just watched it hit media.alfredapp.com/v2update/ via the proxy log.

 

I'm not sure how common it is to use a proxy.pac configuration file, but it's how I always used to do it when I was in charge of a network.

 

Hmm interesting - Does your auto proxy work in Terminal.app for your scripts? If so, what do the environment variables look like there (when http_proxy and https_proxy aren't manually set).

 

Cheers,

Andrew

[deanishe]

Terminal.app/iTerm.app don't set http(s)_proxy for you, regardless of the settings in System Preferences > Network.

 

Proxy env vars have to be set manually or in your dotfiles.