得到ID之後，你才可以拼接出來下面的鏈接

https://juejin.im/user/用戶ID/following

爬蟲編寫

分析好了之後，就可以創建一個scrapy項目了

items.py 文件，用來限定我們需要的所有數據，注意到下面有個_id = scrapy.Field() 這個先預留好，是為了mongdb準備的，其他的欄位解釋請參照注釋即可。

class JuejinItem(scrapy.Item):

_id = scrapy.Field()
username = scrapy.Field()
job = scrapy.Field()
company =scrapy.Field()
intro = scrapy.Field()
# 專欄
columns = scrapy.Field()
# 沸點
boiling = scrapy.Field()
# 分享
shares = scrapy.Field()
# 贊
praises = scrapy.Field()
#
books = scrapy.Field()
# 關注了
follow = scrapy.Field()
# 關注者
followers = scrapy.Field()
goods = scrapy.Field()
editer = scrapy.Field()
reads = scrapy.Field()
collections = scrapy.Field()
tags = scrapy.Field()

編寫爬蟲主入口文件 JuejinspiderSpider.py

import scrapy
from scrapy.selector import Selector
from Juejin.items import JuejinItem

class JuejinspiderSpider(scrapy.Spider):
name = JuejinSpider
allowed_domains = [juejin.im]
# 起始URL 5c0f372b5188255301746103
start_urls = [https://juejin.im/user/55fa7cd460b2e36621f07dde/following]

def parse 函數，邏輯不複雜，處理兩個業務即可 1. 返回item 2. 返回關注列表的Request

item的獲取，我們需要使用xpath匹配即可，為了簡化代碼量，我編寫了一個提取方法，叫做get_default函數。

def get_default(self,exts):
if len(exts)>0:
ret = exts[0]
else:
ret = 0
return ret

def parse(self, response):
#base_data = response.body_as_unicode()
select = Selector(response)
item = JuejinItem()
# 這個地方獲取一下數據
item["username"] = select.xpath("//h1[@class=username]/text()").extract()[0]
position = select.xpath("//div[@class=position]/span/span/text()").extract()
if position:
job = position[0]
if len(position)>1:
company = position[1]
else:
company = ""
else:
job = company = ""
item["job"] = job
item["company"] = company
item["intro"] = self.get_default(select.xpath("//div[@class=intro]/span/text()").extract())
# 專欄
item["columns"] = self.get_default(select.xpath("//div[@class=header-content]/a[2]/div[2]/text()").extract())
# 沸點
item["boiling"] = self.get_default(select.xpath("//div[@class=header-content]/a[3]/div[2]/text()").extract())
# 分享
item["shares"] = self.get_default(select.xpath("//div[@class=header-content]/a[4]/div[2]/text()").extract())
# 贊
item["praises"] = self.get_default(select.xpath("//div[@class=header-content]/a[5]/div[2]/text()").extract())
#
item["books"] = self.get_default(select.xpath("//div[@class=header-content]/a[6]/div[2]/text()").extract())

# 關注了
item["follow"] = self.get_default(select.xpath("//div[@class=follow-block block shadow]/a[1]/div[2]/text()").extract())
# 關注者
item["followers"] = self.get_default(select.xpath("//div[@class=follow-block block shadow]/a[2]/div[2]/text()").extract())

right = select.xpath("//div[@class=stat-block block shadow]/div[2]/div").extract()
if len(right) == 3:
item["editer"] = self.get_default(select.xpath("//div[@class=stat-block block shadow]/div[2]/div[1]/span/text()").extract())
item["goods"] = self.get_default(select.xpath("//div[@class=stat-block block shadow]/div[2]/div[2]/span/span/text()").extract())
item["reads"] = self.get_default(select.xpath("//div[@class=stat-block block shadow]/div[2]/div[3]/span/span/text()").extract())

else:
item["editer"] = ""
item["goods"] = self.get_default(select.xpath("//div[@class=stat-block block shadow]/div[2]/div[1]/span/span/text()").extract())
item["reads"] = self.get_default(select.xpath("//div[@class=stat-block block shadow]/div[2]/div[2]/span/span/text()").extract())

item["collections"] = self.get_default(select.xpath("//div[@class=more-block block]/a[1]/div[2]/text()").extract())
item["tags"] = self.get_default(select.xpath("//div[@class=more-block block]/a[2]/div[2]/text()").extract())
yield item # 返回item

上述代碼，已經成功返回了item，打開setting.py文件中的pipelines設置，測試一下是否可以存儲數據,順便在 DEFAULT_REQUEST_HEADERS 配置一下request的請求參數。

setting.py

DEFAULT_REQUEST_HEADERS = {
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8,
Accept-Language: en,
"Host": "juejin.im",
"Referer": "https://juejin.im/timeline?sort=weeklyHottest",
"Upgrade-Insecure-Requests": "1",
"User-Agent": "Mozilla/5.0 瀏覽器UA"
}

ITEM_PIPELINES = {
Juejin.pipelines.JuejinPipeline: 20,
}

本爬蟲數據存儲到mongodb裡面，所以需要你在pipelines.py文件編寫存儲代碼。

import time
import pymongo

DATABASE_IP = 127.0.0.1
DATABASE_PORT = 27017
DATABASE_NAME = sun
client = pymongo.MongoClient(DATABASE_IP,DATABASE_PORT)
db = client.sun
db.authenticate("dba", "dba")
collection = db.jujin # 準備插入數據

class JuejinPipeline(object):

def process_item(self, item, spider):
try:
collection.insert(item)
except Exception as e:
print(e.args)

運行代碼之後，如果沒有報錯，完善最後一步即可，在Spider裡面將爬蟲的循環操作完成

list_li = select.xpath("//ul[@class=tag-list]/li") # 獲取所有的關注
for li in list_li:
a_link = li.xpath(".//meta[@itemprop=url]/@content").extract()[0] # 獲取URL
# 返回拼接好的數據請求
yield scrapy.Request(a_link+"/following",callback=self.parse)

所有的代碼都已經寫完啦

全站用戶爬蟲編寫完畢，厲害吧。

擴展方向

1. 爬蟲每次只爬取關注列表的第一頁，也可以循環下去，這個不麻煩
2. 在setting.py中開啟多線程操作
3. 添加redis速度更快，後面會陸續的寫幾篇分散式爬蟲，提高爬取速度
4. 思路可以擴展，N多網站的用戶爬蟲，咱後面也寫幾個

歡迎關注她的公眾號，非本科程序員

推薦閱讀：