Python爬蟲：BeautifulSoup的使用

request能取到網頁上面的數據，但是這些是屬於結構化的數據，我們不能直接使用，需要將這些數據進行轉化，從而方便使用

BeautifulSoup能將標籤移除掉，從而獲得網頁上的數據以及內容

1、將特定標籤的內容取出來

單個標籤

from bs4 import BeautifulSoup html_sample = <html> <body><h1 id ="title"> HelloWorld</h1><a href="#" class="link">This is link1</a><a href="# link2" class = "link"> This is link2</a></body></html>

soup= BeautifulSoup(html_sample,html.parser)
header=soup.select(h1)
print(header[0].text)

多個相同的標籤

from bs4 import BeautifulSoup html_sample = <html> <body><h1 id ="title"> HelloWorld</h1><a href="#" class="link">This is link1</a><a href="# link2" class = "link"> This is link2</a></body></html>

soup= BeautifulSoup(html_sample,html.parser)
header=soup.select(a)
for alink in header:
print(alink.text)

2、取出含有特定css屬性的元素

id前面需要加#

from bs4 import BeautifulSoup html_sample = <html> <body><h1 id ="title"> HelloWorld</h1><a href="#" class="link">This is link1</a><a href="# link2" class = "link"> This is link2</a></body></html>

soup= BeautifulSoup(html_sample,html.parser)
header=soup.select(#title)
print(header)

class前面加.

from bs4 import BeautifulSoup html_sample = <html> <body><h1 id ="title"> HelloWorld</h1><a href="#" class="link">This is link1</a><a href="# link2" class = "link"> This is link2</a></body></html>

soup= BeautifulSoup(html_sample,html.parser)
header=soup.select(.link)
for alink in header:
print(alink.text)

3、取得a標籤裡面鏈接的內容

from bs4 import BeautifulSoup html_sample = <html> <body><h1 id ="title"> HelloWorld</h1><a href="#" class="link">This is link1</a><a href="# link2" class = "link"> This is link2</a></body></html>

soup= BeautifulSoup(html_sample,html.parser)
header=soup.select(a)
for alink in header:
print(alink[href])