I've been using random user agent library to assign a random user-agent to each request,
from random_user_agent.user_agent import UserAgent from random_user_agent.params import SoftwareName, OperatingSystem # you can also import SoftwareEngine, HardwareType, SoftwareType, Popularity from random_user_agent.params # you can also set number of user agents required by providing `limit` as parameter software_names = [SoftwareName.CHROME.value] operating_systems = [OperatingSystem.WINDOWS.value, OperatingSystem.LINUX.value] user_agent_rotator = UserAgent(software_names=software_names, operating_systems=operating_systems, limit=100) # Get list of user agents. user_agents = user_agent_rotator.get_user_agents() # Get Random User Agent String. user_agent = user_agent_rotator.get_random_user_agent() result = requests.get(url, headers={'User-Agent':f'{user_agent}'}) At some point when sending requests, my user agent would look like this:
\tMozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36 the \t would cause my request to fail with the above error message, is this a library bug?
one way would be to use strip() on user_agent
but the more efficient way I believe would be to specify at the below step choose only from those starting with 'Mozilla' and then strip()? As the library may contain wrong user agents, how could I implement that?
user_agent = user_agent_rotator.get_random_user_agent() So here specify user_agent_rotator.get_random_user_agent() only starting with 'Mozilla' then strip()?
1 Reset to default