Can I see the URL that a browser receives?

[jceelen]

I have a question, I am working on a project for some time now and I cannot get this to work the way I want :-/. I am working on an Alfred workflow for the 10.000ft projectmanagement application which has a very slow web-interface. I boosted this through working with Alfred and I am using the python library Alfred-Workflow by @deanishe. It's all great! 

Except for one thing, I want it to open project-reports which I can access through an URL with parameters. It is a quite complex URL as you can see below but I managed to reproduce it the way I want it.

https://app.10000ft.com/reports?view=10&time=4&start=2016-12-05&end=2017-02-03&firstgroup=phase_name&secondgroup=user_name&filters=%5B%5B%22TESTPROJECT%20ZONDER%20KLANT%22%5D%2C%5B%5D%2C%5B%22%22%5D%2C%5B%5D%2C%5B%5D%2C%5B%5D%2C%5B%5D%2C%5B%22Confirmed%22%2C%22Future%22%5D%2C%5B%5D%2C%5B%5D%2C%5B%5D%2C%5B%5D%2C%5B%5D%2C%5B%5D%5D&version=2&title=Report%3A%20TESTPROJECT%20ZONDER%20KLANT%20-%205-12-2016

Now comes the funny part, the URL will not work through Alfred (I thought it was my programming skills)! If I paste this URL in the address bar of safari, it works. If I create an 'open url' action with this full URL instead of a query it does not work (the website breaks in different ways). I tried to disable utf-8 encoding, doesn't help. If I choose Chrome, it works!

Something happens between Alfred and Safari, but I want it to work in Safari, for my honour! :-)

 

But I am totally stuck. I am asking myself these questions:

1. Is there a way to see what alfred sends to safari and what safari receives and what happens in between? (the site redirect after receiving the URL)

2. Is there an alternative to the Open URL action to access the browser?

3. How can I fix this?

 

I do not expect it to be relevant for this problem but you can see my code over here: https://github.com/jceelen/alfred-10000ft-scripts/tree/develop

[deanishe]

The usual workaround for issue's with Alfred's Open URL Action is to use a Run Script (language = /bin/bash) and the open command, i.e.:

open "https://app.10000ft.com/reports?view=10&time=4&start=2016-12-05&end=2017-02-03&firstgroup=phase_name&secondgroup=user_name&filters=%5B%5B%22TESTPROJECT%20ZONDER%20KLANT%22%5D%2C%5B%5D%2C%5B%22%22%5D%2C%5B%5D%2C%5B%5D%2C%5B%5D%2C%5B%5D%2C%5B%22Confirmed%22%2C%22Future%22%5D%2C%5B%5D%2C%5B%5D%2C%5B%5D%2C%5B%5D%2C%5B%5D%2C%5B%5D%5D&version=2&title=Report%3A%20TESTPROJECT%20ZONDER%20KLANT%20-%205-12-2016"

 

[jceelen]

Thanks for the fast response! Unfortunately it does not work :-/

[deanishe]

Have you tried adding some Debug utilities to make sure the URL is really what you expect?

 

You can't simply replace Alfred's Open URL with a call to open. You need to ensure the URL is properly encoded first.

 

Also, "it does not work" isn't giving us enough information to tell what's going wrong. Posting the workflow usually helps, but in this case, you need an account to use it.

 

Can you post one of the URLs that isn't working? And any messages from Alfred's debugger.

 

If I had to guess, I'd say it was line 144 in 10000ft.py. Why are you unquoting the query string? That is probably making the URL invalid.

 

Also, you shouldn't try to combine a partial URL with a query string in Alfred's Open URL action. It's designed for inserting a single parameter into a URL, not an entire query string, so it's probably URL-encoding your query string as a single parameter (Alfred has no way of knowing whether you're passing an entire query string or a single parameter. It assumes the latter). You should generate the entire URL yourself and just use {query} as the value for the Open URL action. Alfred handles this case differently and doesn't URL-encode {query}.

Edited December 6, 2016 by deanishe

[jceelen]

tl;dr: I reported a bug at 10.000ft but still want to understand what happens between Alfred and Safari, and what the best approach is to send complex URLs to the browser. 

 

4 hours ago, deanishe said:

Have you tried adding some Debug utilities to make sure the URL is really what you expect?

 

You can't simply replace Alfred's Open URL with a call to open. You need to ensure the URL is properly encoded first.

 

Also, "it does not work" isn't giving us enough information to tell what's going wrong. Posting the workflow usually helps, but in this case, you need an account to use it.

Can you post one of the URLs that isn't working? And any messages from Alfred's debugger.

 

I understand that the URL should be correct, that's where I was looking for solutions in the first place. I can send you loads of URLs that are not working :-) but that does not seem relevant right now since I know that the problem is not occurring in my .py script. Something goes wrong between Alfred and the browser. Let me explain: I have an URL that works if i paste it in Safari. If I ask Alfred to open exactly the same URL it in Safari (my default browser), either with open through a script as suggested or with open url as an action it does not work as expected.

 

Q: Do you agree on this?

 

The result/behaviour that I don't expect is that the report is missing markup / CSS. I do not see this behaviour in Chrome btw. But maybe other things are going wrong under the hood, an uncertainty I don't like when i comes to project reports :-). I must admit that the 10.000ft application is a bit wonky, I just discovered that if I paste the URL a second time, I get the same unwanted behaviour, so I also filed a bug report about errors in this reports section. But it is still strange that if I paste the same URL everything is OK the first time, right?

 

Q: Is there a way to see what Alfred sends to the browser or what the browser receives from Alfred?

 

4 hours ago, deanishe said:

If I had to guess, I'd say it was line 144 in 10000ft.py. Why are you unquoting the query string? That is probably making the URL invalid.

 

You are looking at the master, check out the development branch for the most recent code. It is still not perfect but I think I need to solve the Alfred > Safari (or the bug in the website) problem first before fixing this. 

 

Trivial n00b Q: I thought encoding was the problem so I added .encoding(utf-8) on line 308 & 309. Is this necessary/correct? 

 

4 hours ago, deanishe said:

Also, you shouldn't try to combine a partial URL with a query string in Alfred's Open URL action. It's designed for inserting a single parameter into a URL, not an entire query string, so it's probably URL-encoding your query string as a single parameter (Alfred has no way of knowing whether you're passing an entire query string or a single parameter. It assumes the latter). You should generate the entire URL yourself and just use {query} as the value for the Open URL action. Alfred handles this case differently and doesn't URL-encode {query}.

 

Thanks for explaining, good to have a better understanding of how this works in Alfred. So if I understand correctly there are two approaches: 

- build the full URL in the python script only put {query} in as a value for the Open URL action

- put the full URL in the Open URL value and add variables in it? What is this the best way to insert multiple variables? 

 

Q: What is the best way to build/send complex URLs like these to the default browser through Alfred?

Edited December 6, 2016 by jceelen

[deanishe]

1 hour ago, jceelen said:

tl;dr: I reported a bug at 10.000ft but still want to understand what happens between Alfred and Safari, and what the best approach is to send complex URLs to the browser. 

 

 

I understand that the URL should be correct, that's where I was looking for solutions in the first place. I can send you loads of URLs that are not working :-) but that does not seem relevant right now since I know that the problem is not occurring in my .py script. Something goes wrong between Alfred and the browser. Let me explain: I have an URL that works if i paste it in Safari. If I ask Alfred to open exactly the same URL it in Safari (my default browser), either with open through a script as suggested or with open url as an action it does not work as expected.

 

Q: Do you agree on this?

 

That Safari accepts a URL isn't a strong indication that the URL is valid. Safari is pretty forgiving about what it lets you paste in its URL bar, and will open a lot of invalid URLs. OTOH, if the open command won't accept your URL, it is very likely invalid (providing you're passing the URL correctly, so bash isn't mangling it).

 

1 hour ago, jceelen said:

Trivial n00b Q: I thought encoding was the problem so I added .encoding(utf-8) on line 308 & 309. Is this necessary/correct? 

 

It's unnecessary and incorrect. Workflow.add_item() wants Unicode strings.

 

What you should be encoding are the values passed to urlencode(). If you pass in Unicode that can't be encoded to ASCII, urlencode() will throw an exception.

 

1 hour ago, jceelen said:

Q: What is the best way to build/send complex URLs like these to the default browser through Alfred?

 

Build the URL yourself and use {query} as the entire URL. Or use the open shell command, which is guaranteed not to alter your URL. open will send the URL to your default browser unless you specify a different application with the -a flag.

[jceelen]

Thanks for your patience and quick feedback! 

 

3 hours ago, deanishe said:

That Safari accepts a URL isn't a strong indication that the URL is valid. Safari is pretty forgiving about what it lets you paste in its URL bar, and will open a lot of invalid URLs. OTOH, if the open command won't accept your URL, it is very likely invalid (providing you're passing the URL correctly, so bash isn't mangling it).

 

Just to be sure if we are on the same page: in all three cases the URL opens :-) but the way the site interprets the parameters is wrong.

 

Last Q: Is there a way to see what Alfred sends to the browser or what the browser receives from Alfred?

[deanishe]

19 minutes ago, jceelen said:

Just to be sure if we are on the same page: in all three cases the URL opens :-) but the way the site interprets the parameters is wrong.

 

Which is yet another indication that the URL is not actually correct.

 

Like I said, Safari happily accepts (some kinds of) invalid URLs. And by all appearances, you're not passing the parameters to the web server in the expected fashion.

 

Why are you so sure that the problem lies with Alfred/OS X/the website and not your code? 95% of the time, it's your own code that's the problem, and not the software used by thousands of other people who aren't having the same problem…

 

19 minutes ago, jceelen said:

Last Q: Is there a way to see what Alfred sends to the browser or what the browser receives from Alfred?

 

Open Alfred's debugger and choose Log: All information

[jceelen]

10.000ft responded and they can reproduce the problem and are working on a fix. I just want to understand it. The logger with 'all information' was helpful too als shows the same URLs so I'll await the fix at the site. Thanks for helping me out with this.

 

PS: I encountered an issue with encoding ASCII input this week and fixed it with:

params = urlencode(params, 'utf-8')

 

[deanishe]

On 14/12/2016 at 10:34 PM, jceelen said:

PS: I encountered an issue with encoding ASCII input this week and fixed it with:

params = urlencode(params, 'utf-8')

 

 

What do you mean by that? urlencode in Py2 doesn't have an encoding parameter, and Py3 would require you to specify encoding='utf-8'.

 

But Alfred-Workflow doesn't support Py3.

[jceelen]

That is very strange, someone on Stackoverflow suggested this and I tried (see the code) it solved my problem.