I've been using random user agent library to assign a random user-agent to each request,

from random_user_agent.user_agent import UserAgent from random_user_agent.params import SoftwareName, OperatingSystem # you can also import SoftwareEngine, HardwareType, SoftwareType, Popularity from random_user_agent.params # you can also set number of user agents required by providing `limit` as parameter software_names = [SoftwareName.CHROME.value] operating_systems = [OperatingSystem.WINDOWS.value, OperatingSystem.LINUX.value] user_agent_rotator = UserAgent(software_names=software_names, operating_systems=operating_systems, limit=100) # Get list of user agents. user_agents = user_agent_rotator.get_user_agents() # Get Random User Agent String. user_agent = user_agent_rotator.get_random_user_agent() result = requests.get(url, headers={'User-Agent':f'{user_agent}'}) 

At some point when sending requests, my user agent would look like this:

\tMozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36 

the \t would cause my request to fail with the above error message, is this a library bug?

one way would be to use strip() on user_agent

but the more efficient way I believe would be to specify at the below step choose only from those starting with 'Mozilla' and then strip()? As the library may contain wrong user agents, how could I implement that?

user_agent = user_agent_rotator.get_random_user_agent() 

So here specify user_agent_rotator.get_random_user_agent() only starting with 'Mozilla' then strip()?

1

Reset to default

Know someone who can answer? Share a link to this question via email, Twitter, or Facebook.

Your Answer

Sign up or log in

Sign up using Google Sign up using Facebook Sign up using Email and Password

Post as a guest

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy