I'm trying to parse the site. I don't want to use selenium. Requests is coping. BUT! something strange is happening. I can't cut out the text I need with a regular expression (and it's there - you can see it if you do print(data.text)) But re doesn't see him. If this text is copied to notepad++, it outputs this - it sees these characters as a single line.
import requests import re data = requests.get(') print(data.text) What is it and how to work with it?pay attention to the line numbers
72 Answers
You can try to use their Ajax API to load all usernames + thumb images:
import pandas as pd import requests url = ' headers = {'X-Requested-With': 'XMLHttpRequest'} all_data = [] for p in range(1, 4): # <-- increase number of pages here data = requests.get(url.format(p * 144), headers=headers).json() for m in data['models']: all_data.append((m['username'], m['display_name'], m['thumb_image'].replace('{ext}', 'jpg'))) df = pd.DataFrame(all_data, columns=['username', 'display_name', 'thumb']) print(df.head()) Prints:
username display_name thumb 0 wetlilu Little_Lilu // 1 mellannie8 mellannieSEX // 2 mokkoann mokkoann // 3 ogurezzi CynEp-nuCbka // 4 Pepetka22 _-Katya-_ // Avoid using . in a regex unless you really want to get any character; here, the usernames (as far as I can see) only contain - and alphanumeric characters, so you can retrieve them with:
re.findall(r'"username":"([\w|-]+)"',data.text) An even simpler way, which will remove the need to deal with special characters by getting all characters except " is:
re.findall(r'"username":"([^"]+)"',data.text) So here's a way of getting the info you seek (I joined them into a dictionary, but you can change that to whatever you prefer):
import requests import re data = requests.get(') with open ("return.txt",'w', encoding = 'utf-8') as f: f.write(data.text) names = re.findall(r'"username":"([^"]+)"',data.text) disp_names = re.findall(r'"display_name":"([^"]+)"',data.text) thumbs = re.findall(r'"thumb_image":"([^"]+)"',data.text) names_dict = {name:[disp, thumb.replace('{ext}', 'jpg')] for name, disp, thumb in zip(names, disp_names, thumbs)} Example
names_dict['JuliaCute'] # ['_Cute', # '\\/\\/i.bimbolive.com\\/live\\/055\\/2b0\\/15d\\/xbig_lq\\/d89ef4.jpg'] 
