Ytukyg

i want crawl data from amazon,requests respone error 503,404,200 then i will check link die or live. But current link die or link live or link not found will respone 503.i dont know how check link ?.if respone 200 then will clear for software of me. THANKS YOUR HELP ME!!!!

link = "https://www.amazon.com/dp/B07K896272"enter code here

browser = webdriver.Firefox(executable_path=r'D:PythonToolAmzToolgeckodriver.exe')

browser.get(link)

res = requests.get(str(link).strip())

print(str(res))

If you hardcoded the string into variable link, you dont need to cast it to type str.

requests.get(link) // is good enough.

Also, if you want to print the response content,

print(res.text)

Don't really understand the question, but
you can check the response status by:

res = requests.get(link)

if res.status_code:

    #Bad Code - 400s/500s



else:

    #All good

Also, somes sites doesn't allow requests. You can try to be more "human" by adding in headers with user-agent and also use session. Session will keep the cookies. (kinda making it stateful)

session = requests.session()

session.headers['User-Agent'] = "YOUR USER AGENT HERE"

session.get("https://www.amazon.com/")



res = session.get(link)

print(res.text)

Some sites requires javascript to load the page. If that's the case, you will want to use selenium. using requests will not load the javascript page.

Or if u want to load the page with javascript first, and want to use requests:

session = requests.session()

session.headers['User-Agent'] = "YOUR USER AGENT HERE"

browser = webdriver.Firefox(executable_path=r'D:PythonToolAmzToolgeckodriver.exe')

browser.get(link)

for cookie in driver.get_cookies():

    c = {cookie['name']: cookie['value']}

    session.cookies.update(c)

browser.close()

res = session.get(link)

print(res.text)