【python×selenium】ユーザーエージェントを偽装してサイトにアクセスしてみよう。

退屈なことはPythonにやらせよう ?ノンプログラマーにもできる自動化処理プログラミング

新品価格
￥4,070から
(2020/4/12 17:54時点)

今回はユーザーエージェントをランダムに変更してアクセスするプログラムを備忘録として残しておきます。

■ユーザーエージェントとは？

WEBブラウザがサーバーにアクセスするときにブラウザの種類（Chrome等）OSの種類（Windows10等）それぞれのバージョンなどの情報が送られています。

サーバーでは、ブラウザが申告してきたユーザーエージェント名を利用して、それぞれのブラウザ・OSに合わせたHTMLを返します。このユーザーエージェント名はブラウザが申告するものなので書き換えることが可能なのです。

■ユーザーエージェント(UA)偽装するプログラム

こちらの確認君に自動でアクセスしてユーザーエージェントを確認します。

https://www.ugtop.com/spill.shtml

使用するUAの情報はこちらのサイトのものを使用しました。様々な情報が載っているのでお好みのものをお使い下さい。

https://developers.whatismybrowser.com/useragents/explore/

実際のコードがこちらになります。

<!-- Crayon Syntax Highlighter v_2.7.2_beta -->
 
        <div id="crayon-68175b374d2f7732885686" class="crayon-syntax crayon-theme-classic crayon-font-monaco crayon-os-pc print-yes notranslate" data-settings=" minimize scroll-mouseover" style="margin-top: 12px; margin-bottom: 12px; font-size: 12px !important; line-height: 15px !important; height: auto;">
         
            <div class="crayon-toolbar" data-settings=" mouseover overlay hide delay" style="font-size: 12px !important; height: 18px !important; line-height: 18px !important; margin-top: -18px; display: none; position: absolute; z-index: 2;"><span class="crayon-title"></span>
            <div class="crayon-tools" style="font-size: 12px !important;height: 18px !important; line-height: 18px !important;"><div class="crayon-button crayon-nums-button crayon-pressed" title="Toggle Line Numbers"><div class="crayon-button-icon"></div></div><div class="crayon-button crayon-plain-button" title="Toggle Plain Code"><div class="crayon-button-icon"></div></div><div class="crayon-button crayon-wrap-button" title="Toggle Line Wrap"><div class="crayon-button-icon"></div></div><div class="crayon-button crayon-expand-button" title="Expand Code"><div class="crayon-button-icon"></div></div><div class="crayon-button crayon-copy-button" title="Copy"><div class="crayon-button-icon"></div></div><div class="crayon-button crayon-popup-button" title="Open Code In New Window"><div class="crayon-button-icon"></div></div></div></div>
            <div class="crayon-info" style="min-height: 16.8px !important; line-height: 16.8px !important;"></div>
            <div class="crayon-plain-wrap"><textarea wrap="soft" class="crayon-plain print-no" data-settings="dblclick" readonly="" style="tab-size: 4; font-size: 12px !important; line-height: 15px !important; z-index: 0; opacity: 0; overflow: hidden;">import random
import os
from selenium import webdriver
 
#UAをいくつか格納しておく
user_agent = ['Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36',
              'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.121 Safari/537.36',
              'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.157 Safari/537.36',
              'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.113 Safari/537.36'
              ]
 
 
options = webdriver.ChromeOptions()
 
#ここにuser_agentからランダムで読み込み
options.add_argument('--user-agent=' + user_agent[random.randrange(0, len(user_agent), 1)])
print(user_agent[random.randrange(0, len(user_agent), 1)])
# chromedriverの場所を指定します
chromedriver = "C:\webdriver\chromedriver.exe"
 
#optionsを設定します
driver = webdriver.Chrome(chrome_options=options,executable_path=chromedriver)
 
#確認君のサイトを開きます。
driver.get('https://www.ugtop.com/spill.shtml')</textarea></div>
            <div class="crayon-main" style="position: relative; z-index: 1; overflow: hidden;">
                <table class="crayon-table" style="">
                    <tbody><tr class="crayon-row">
                <td class="crayon-nums " data-settings="show">
                    <div class="crayon-nums-content" style="font-size: 12px !important; line-height: 15px !important;"><div class="crayon-num" data-line="crayon-68175b374d2f7732885686-1">1</div><div class="crayon-num crayon-striped-num" data-line="crayon-68175b374d2f7732885686-2">2</div><div class="crayon-num" data-line="crayon-68175b374d2f7732885686-3">3</div><div class="crayon-num crayon-striped-num" data-line="crayon-68175b374d2f7732885686-4">4</div><div class="crayon-num" data-line="crayon-68175b374d2f7732885686-5">5</div><div class="crayon-num crayon-striped-num" data-line="crayon-68175b374d2f7732885686-6">6</div><div class="crayon-num" data-line="crayon-68175b374d2f7732885686-7">7</div><div class="crayon-num crayon-striped-num" data-line="crayon-68175b374d2f7732885686-8">8</div><div class="crayon-num" data-line="crayon-68175b374d2f7732885686-9">9</div><div class="crayon-num crayon-striped-num" data-line="crayon-68175b374d2f7732885686-10">10</div><div class="crayon-num" data-line="crayon-68175b374d2f7732885686-11">11</div><div class="crayon-num crayon-striped-num" data-line="crayon-68175b374d2f7732885686-12">12</div><div class="crayon-num" data-line="crayon-68175b374d2f7732885686-13">13</div><div class="crayon-num crayon-striped-num" data-line="crayon-68175b374d2f7732885686-14">14</div><div class="crayon-num" data-line="crayon-68175b374d2f7732885686-15">15</div><div class="crayon-num crayon-striped-num" data-line="crayon-68175b374d2f7732885686-16">16</div><div class="crayon-num" data-line="crayon-68175b374d2f7732885686-17">17</div><div class="crayon-num crayon-striped-num" data-line="crayon-68175b374d2f7732885686-18">18</div><div class="crayon-num" data-line="crayon-68175b374d2f7732885686-19">19</div><div class="crayon-num crayon-striped-num" data-line="crayon-68175b374d2f7732885686-20">20</div><div class="crayon-num" data-line="crayon-68175b374d2f7732885686-21">21</div><div class="crayon-num crayon-striped-num" data-line="crayon-68175b374d2f7732885686-22">22</div><div class="crayon-num" data-line="crayon-68175b374d2f7732885686-23">23</div><div class="crayon-num crayon-striped-num" data-line="crayon-68175b374d2f7732885686-24">24</div><div class="crayon-num" data-line="crayon-68175b374d2f7732885686-25">25</div></div>
                </td>
                        <td class="crayon-code"><div class="crayon-pre" style="font-size: 12px !important; line-height: 15px !important; -moz-tab-size:4; -o-tab-size:4; -webkit-tab-size:4; tab-size:4;"><div class="crayon-line" id="crayon-68175b374d2f7732885686-1"><span class="crayon-e">import </span><span class="crayon-e">random</span></div><div class="crayon-line crayon-striped-line" id="crayon-68175b374d2f7732885686-2"><span class="crayon-e">import </span><span class="crayon-e">os</span></div><div class="crayon-line" id="crayon-68175b374d2f7732885686-3"><span class="crayon-e">from </span><span class="crayon-e">selenium </span><span class="crayon-e">import </span><span class="crayon-v">webdriver</span></div><div class="crayon-line crayon-striped-line" id="crayon-68175b374d2f7732885686-4"> </div><div class="crayon-line" id="crayon-68175b374d2f7732885686-5"><span class="crayon-p">#UAをいくつか格納しておく</span></div><div class="crayon-line crayon-striped-line" id="crayon-68175b374d2f7732885686-6"><span class="crayon-v">user_agent</span><span class="crayon-h"> </span><span class="crayon-o">=</span><span class="crayon-h"> </span><span class="crayon-sy">[</span><span class="crayon-s">'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36'</span><span class="crayon-sy">,</span></div><div class="crayon-line" id="crayon-68175b374d2f7732885686-7"><span class="crayon-h">              </span><span class="crayon-s">'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.121 Safari/537.36'</span><span class="crayon-sy">,</span></div><div class="crayon-line crayon-striped-line" id="crayon-68175b374d2f7732885686-8"><span class="crayon-h">              </span><span class="crayon-s">'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.157 Safari/537.36'</span><span class="crayon-sy">,</span></div><div class="crayon-line" id="crayon-68175b374d2f7732885686-9"><span class="crayon-h">              </span><span class="crayon-s">'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.113 Safari/537.36'</span></div><div class="crayon-line crayon-striped-line" id="crayon-68175b374d2f7732885686-10"><span class="crayon-h">              </span><span class="crayon-sy">]</span></div><div class="crayon-line" id="crayon-68175b374d2f7732885686-11"> </div><div class="crayon-line crayon-striped-line" id="crayon-68175b374d2f7732885686-12"> </div><div class="crayon-line" id="crayon-68175b374d2f7732885686-13"><span class="crayon-v">options</span><span class="crayon-h"> </span><span class="crayon-o">=</span><span class="crayon-h"> </span><span class="crayon-v">webdriver</span><span class="crayon-sy">.</span><span class="crayon-e">ChromeOptions</span><span class="crayon-sy">(</span><span class="crayon-sy">)</span></div><div class="crayon-line crayon-striped-line" id="crayon-68175b374d2f7732885686-14"> </div><div class="crayon-line" id="crayon-68175b374d2f7732885686-15"><span class="crayon-p">#ここにuser_agentからランダムで読み込み</span></div><div class="crayon-line crayon-striped-line" id="crayon-68175b374d2f7732885686-16"><span class="crayon-v">options</span><span class="crayon-sy">.</span><span class="crayon-e">add_argument</span><span class="crayon-sy">(</span><span class="crayon-s">'--user-agent='</span><span class="crayon-h"> </span><span class="crayon-o">+</span><span class="crayon-h"> </span><span class="crayon-v">user_agent</span><span class="crayon-sy">[</span><span class="crayon-v">random</span><span class="crayon-sy">.</span><span class="crayon-e">randrange</span><span class="crayon-sy">(</span><span class="crayon-cn">0</span><span class="crayon-sy">,</span><span class="crayon-h"> </span><span class="crayon-e">len</span><span class="crayon-sy">(</span><span class="crayon-v">user_agent</span><span class="crayon-sy">)</span><span class="crayon-sy">,</span><span class="crayon-h"> </span><span class="crayon-cn">1</span><span class="crayon-sy">)</span><span class="crayon-sy">]</span><span class="crayon-sy">)</span></div><div class="crayon-line" id="crayon-68175b374d2f7732885686-17"><span class="crayon-e">print</span><span class="crayon-sy">(</span><span class="crayon-v">user_agent</span><span class="crayon-sy">[</span><span class="crayon-v">random</span><span class="crayon-sy">.</span><span class="crayon-e">randrange</span><span class="crayon-sy">(</span><span class="crayon-cn">0</span><span class="crayon-sy">,</span><span class="crayon-h"> </span><span class="crayon-e">len</span><span class="crayon-sy">(</span><span class="crayon-v">user_agent</span><span class="crayon-sy">)</span><span class="crayon-sy">,</span><span class="crayon-h"> </span><span class="crayon-cn">1</span><span class="crayon-sy">)</span><span class="crayon-sy">]</span><span class="crayon-sy">)</span></div><div class="crayon-line crayon-striped-line" id="crayon-68175b374d2f7732885686-18"><span class="crayon-p"># chromedriverの場所を指定します</span></div><div class="crayon-line" id="crayon-68175b374d2f7732885686-19"><span class="crayon-v">chromedriver</span><span class="crayon-h"> </span><span class="crayon-o">=</span><span class="crayon-h"> </span><span class="crayon-s">"C:\webdriver\chromedriver.exe"</span></div><div class="crayon-line crayon-striped-line" id="crayon-68175b374d2f7732885686-20"> </div><div class="crayon-line" id="crayon-68175b374d2f7732885686-21"><span class="crayon-p">#optionsを設定します</span></div><div class="crayon-line crayon-striped-line" id="crayon-68175b374d2f7732885686-22"><span class="crayon-v">driver</span><span class="crayon-h"> </span><span class="crayon-o">=</span><span class="crayon-h"> </span><span class="crayon-v">webdriver</span><span class="crayon-sy">.</span><span class="crayon-e">Chrome</span><span class="crayon-sy">(</span><span class="crayon-v">chrome_options</span><span class="crayon-o">=</span><span class="crayon-v">options</span><span class="crayon-sy">,</span><span class="crayon-v">executable_path</span><span class="crayon-o">=</span><span class="crayon-v">chromedriver</span><span class="crayon-sy">)</span></div><div class="crayon-line" id="crayon-68175b374d2f7732885686-23"> </div><div class="crayon-line crayon-striped-line" id="crayon-68175b374d2f7732885686-24"><span class="crayon-p">#確認君のサイトを開きます。</span></div><div class="crayon-line" id="crayon-68175b374d2f7732885686-25"><span class="crayon-v">driver</span><span class="crayon-sy">.</span><span class="crayon-e">get</span><span class="crayon-sy">(</span><span class="crayon-s">'https://www.ugtop.com/spill.shtml'</span><span class="crayon-sy">)</span></div></div></td>
                    </tr>
                </tbody></table>
            </div>
        </div>
<!-- [Format Time: 0.0017 seconds] -->