I apply find_all on a beautifulsoup object, and find something, which is an bs4.element.ResultSet object or a list.
I want to further do find_all in there, but it's not allowed on a bs4.element.ResultSet object. I can loop through each element of the bs4.element.ResultSet object to do find_all. But can I avoid looping and just convert it back to a beautifulsoup object?
Here is my code:
html_1 = """ <table> <thead> <tr> <th>A</th> <th>B</th> <th>C</th> <th>D</th> </tr> </thead> </table> """ soup = BeautifulSoup(html_1, 'html.parser') type(soup) #bs4.BeautifulSoup # do find_all on beautifulsoup object th_all = soup.find_all('th') # the result is of type bs4.element.ResultSet or similarly list type(th_all) #bs4.element.ResultSet type(th_all[0:1]) #list # now I want to further do find_all th_all.find_all(text='A') #not work # can I avoid this need of loop? for th in th_all: th.find_all(text='A') #works 2 Answers
ResultSet class is a subclass of a list and not a Tag class which has the find* methods defined. Looping through the results of find_all() is the most common approach:
th_all = soup.find_all('th') result = [] for th in th_all: result.extend(th.find_all(text='A')) Usually, CSS selectors may help you solve it in one go except that not everything you can do with find_all() is possible with the select() method. For instance, there is no "text" search available in bs4 CSS selectors. But, if, for example, you had to find all, say, b elements inside th elements, you could do:
soup.select("th td") I know this is many years too late but I went down the rabbit hole the other day and found that ResultSet is subclassed from a Python list (source code); it's really just a list that additionally has .source attribute which is often a null object.
Now, back to OP's main question, a filtering can be done during the find_all() call by passing the tag type and the string it should match. This returns a ResultSet of tags (denoted th_all below). To extract the actual texts inside these tags, we must loop again to through it.
html_1 = """ <table> <thead> <tr> <th>A</th> <th>B</th> <th>C</th> <th>D</th> </tr> </thead> </table> """ soup = BeautifulSoup(html_1, 'html.parser') th_all = soup.find_all('th', string='A') # [<th>A</th>] texts = [th.string for th in th_all] # ['A'] To answer the second part of the question:
How do we convert a ResultSet into a BeautifulSoup object?
We can explicitly cast it as one. Then we can call find_all() on it.
th_all = soup.find_all('th') soup2 = BeautifulSoup('\n'.join(map(str, th_all))) soup2.find_all(string='A') # ['A'] However, since we can already do a search on the ResultSet, it's probably not desirable in this context.
