httplib没有获得所有重定向代码
|
我正在尝试获取似乎多次重定向的页面的最终URL。在浏览器中尝试以下示例URL，并将其与我的代码段底部的最终URL进行比较：
链接多次重定向
这是我正在运行的测试代码，请注意，获得代码200的最终URL与浏览器中的URL不相同。我有什么选择？
Python 2.7.1+ (r271:86832, Apr 11 2011, 18:13:53)
[GCC 4.5.2] on linux2
Type \"help\", \"copyright\", \"credits\" or \"license\" for more information.
>>> import httplib
>>> from urlparse import urlparse
>>> url = \'http://www.usmc.mil/units/hqmc/\'
>>> host = urlparse(url)[1]
>>> req = \'\'.join(urlparse(url)[2:5])
>>> conn = httplib.HTTPConnection(host)
>>> conn.request(\'HEAD\', req)
>>> resp = conn.getresponse()
>>> print resp.status
301
>>> print resp.msg.dict[\'location\']
http://www.marines.mil/units/hqmc/
>>> url = \'http://www.marines.mil/units/hqmc/\'
>>> host = urlparse(url)[1]
>>> req = \'\'.join(urlparse(url)[2:5])
>>> conn = httplib.HTTPConnection(host)
>>> conn.request(\'HEAD\', req)
>>> resp = conn.getresponse()
>>> print resp.status
302
>>> print resp.msg.dict[\'location\']
http://www.marines.mil/units/hqmc/default.aspx
>>> url = \'http://www.marines.mil/units/hqmc/default.aspx\'
>>> host = urlparse(url)[1]
>>> req = \'\'.join(urlparse(url)[2:5])
>>> conn = httplib.HTTPConnection(host)
>>> conn.request(\'HEAD\', req)
>>> resp = conn.getresponse()
>>> print resp.status
200
>>> print resp.msg.dict[\'location\']
Traceback (most recent call last):
File \"<stdin>\", line 1, in <module>
KeyError: \'location\'
>>> print url
http://www.marines.mil/units/hqmc/default.aspx //THIS URL DOES NOT RETURN A 200 IN ANY BROWSER I HAVE TRIED
没有找到相关结果
已邀请:
2 个回复
孝箱捆讨
蓄荣糖些
import httplib2 def getContentLocation(link): h = httplib2.Http(\".cache_httplib\") h.follow_all_redirects = True resp = h.request(link, \"GET\")[0] contentLocation = resp[\'content-location\'] return contentLocation if __name__ == \'__main__\': link = \'http://podcast.at/podcast_url344476.html\' print getContentLocation(link)请注意，此示例还使用了缓存（urllib和httplib均不支持该缓存）。因此，这将重复运行明显更快。这对于爬网/抓取可能很有趣。如果不想缓存，请将替换为。