猫史档案馆

爬虫的方式搜搜狗网

用户:TimeLinkForever查看:0 回复:9 评论:0 创建时间:2020-02-13T17:57:04

# encoding:gbk

import urllib.request, urllib.error, urllib.parse,sys
from bs4 import BeautifulSoup


def getsearch(a):
    output = ""

    if " " in a:
        a = a.split()
        for add in a:
            output += add
            output += '%20'
    else:
        output = a
    return output

TimeLinkForever

剩下的代码写不上来了，字数超过了，还不能频繁发帖，要的告诉我邮箱

TimeLinkForever

def Search(Text, num):  
    header = {"User-Agent" : "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0;"}

    request = urllib.request.Request('https://www.sogou.com/web?

TimeLinkForever

query=%s&page=%s'% (Text,num), headers = header)
    Myblabla = urllib.request.urlopen(request).read()
    print("ok")

    MySoup = BeautifulSoup(Myblabla,'html.parser',from_encoding = 'gbk')
    Links = MySoup.find_all('a')

TimeLinkForever

    file = open("Final.cpp",'a')
    for Link in Links:
        try:
            if str(Link['href']).strip().startswith('/'):


                print('link:%s,txt:%s'% ('http://www.sogou.com' + Link['href'].strip()+"\n",str(Link.get_text()).strip()+"\n"))
            else:

TimeLinkForever

print('link:%stxt:%s'% (Link['href'].strip()+"\n",str(Link.get_text()).strip()+"\n"))#.replace(u'\xa0', u'').encode('utf-8')
        except:

            print("Failed")
    file.close()
    return

TimeLinkForever

print('link:%stxt:%s'% (Link['href'].strip()+"\n",str(Link.get_text()).strip()+"\n"))#.replace(u'\xa0', u'').encode('utf-8')
        except:

            print("Failed")
    file.close()
    return

print("Get Ready To Search!")

TimeLinkForever

b = int(input("PageAmount>>>"))
a = input("PythonSeForSogou>>>")
hap = 1
for i in range(0,b):
    Search(getsearch(a),hap)
    hap+=1

TimeLinkForever

自己整理一下顺序就好，字数太多网速慢只能这样

TimeLinkForever

先看帖子内容，然后从下往上看