Ever wish you could just ask Google’s official SEO documentation a question, and get a direct answer? I built a chatbot that lets you do exactly that.

Try it here

Whether you're trying to understand how Google handles a specific SEO scenario or just need a reliable source to support your recommendations, this tool uses vector search to surface the most relevant content directly from Google’s guidelines.

How did I build it? (in short):

• Collected all SEO-related URLs from Google’s sitemaps

• Scraped the content from those pages

• Created text embeddings using Sentence Transformers

• Stored them in Pinecone (a vector database)

• Used LangChain to fetch relevant chunks based on a user’s query

• Passed the retrieved context to GPT to generate a summarized answer

• Built the front end using Streamlit

It’s lightweight, fast, and grounded in official documentation. Try it out and see what insights you can uncover.

This article was rewritten using ChatGPT

Try it here

However, we are not aware of what type of queries they have tested or how many queries. Queries in some niches can be very specific and different.

So I decided to write a Python script to test the set of keywords for which we get traffic. This helped us to gain insights into our specific industry. For example, I got to know that when I enable SGE Labs AIO snippets show up much more often than when SGE Labs are disabled -

So how do we do it?

For this, we cannot use scrapers like Scrapy because it will probably get blocked and will not be able to perform any actions on the page like - searching or clicking on show more button.

For scraping AIO Snippet, we have to -

1. Make sure we are on a US IP Address (Use VPN if required)

2. Log in to a Gmail account that can see AIO Snippets (With or without SGE Labs enabled).

3. Search for the query.

4. Check if the AIO Snippet is present.

5. If it is present then click on the “Show More” button

6. Scrape All the Links and Take a Screenshot

All these steps are possible using - Selenium.

What is Selenium? Selenium is an open-source tool that allows you to automate web browsers. It provides a way to interact with web pages through a variety of web drivers (like ChromeDriver for Chrome, GeckoDriver for Firefox, etc.). With Selenium, you can simulate a real user's interaction with a web page, such as clicking buttons, filling out forms, and extracting information via a script.

Selenium is mostly used for testing purposes, but it has good applications in web scraping as well.

So Initially I used Selenium but Google was able to detect it and did not allow me to log in. So I had to use an Undetected Chrome Driver - which blocks Google from detecting that we are sending automated requests to Google.

However I was facing some issues with running Undetected Chrome Driver in headless mode, so I used Selenium Base in Headless and Undetected mode.

What is Headless Mode? In Headless mode, the web browser runs without a graphical user interface (GUI). So your Chrome browser will be running but you will not be able to see it, which is useful because you want this whole process to run in the background. But if you want to see what’s going on Chrome browser, you can disable headless mode.

So to summarize - We will scrape the AIO snippet from Google Search using Selenium Base in Undetected and Headless mode. So without further ado, let’s jump into the script.

Note: I recommend to read the whole article because there are multiple values you will have to replace in the script to work.

Before we start we need to install some packages - Selenium, SeleniumBase, and Pandas. So run the following commands in your terminal -

pip install selenium
pip install selenium-base
pip install pandas

1. Importing Packages

So we are importing Driver from Selenium Base, and some packages from Selenium to scrape via Class Names and to use Keyboard.

Time for pausing between each query or function. Random for randomizing our waiting time - so that Google cannot understand our patterns.

Pandas for dataframes.

from seleniumbase import Driver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
import time
import pandas as pd
import random

Reading a CSV file of queries using Pandas, and then converting it to a list.

I have also mentioned some sample queries which most of the times shows AIO Snippet. So you can use them to test first.

df = pd.read_csv(r'query_list.csv',encoding='latin-1')

df = df.replace("/"," ",regex=True)

print(df)

query_list = df['query'].values.tolist()

#test_queries
#query_list = ['what is digital marketing','features of macbook pro m3','write few lines about christmas','what are the properties of metal']

print(query_list[:5])

number_of_queries = len(query_list)

2. Initializing Chrome Driver and setting Google’s login URL to a variable.

Note that here if you do not want to run the Chrome driver in headless mode, you can simply set headless=False.

#path to chromedriver
driver = Driver(uc=True, headless=True)

#login to google manually
google_login_url = 'https://accounts.google.com/v3/signin/identifier?hl=en-gb&ifkv=AaSxoQyH1Qet0VlLGN7HL7CD1zRZkw0dVFB4bXApnkylfeDCW4kx3vmK9HBjvDjfi4a6yi3WFJJfug&flowName=GlifWebSignIn&flowEntry=ServiceLogin&dsh=S1544068279%3A1715766753444785&ddm=0'

3. Login to Google

Following is a function to log in to Google. There are some important points to remember here -

1. Make sure to create a new Google account or use an alternative one (we all have one), and avoid risking your main account.

2. Turn off 2FA on the account you are using to avoid verifying every time you run the script.

3. When you run this script for the first time, do not use Headless mode and verify yourself on your device with US IP.

4. Once you have done that, next time Google will hopefully not ask for any verification.

Explanation of the below function -

First, we visit the login URL, then we wait for 5 seconds. And then we find the email field and enter the email. Then we again wait for 5 seconds and then enter a password.

It will also print the URL it is on while entering your email and password. If the URL changes after entering the password, it means we are logged in.

If something goes wrong, it will take the screenshot and save it in your folder.

#function to login to google
def login(email,password):
    driver.get(google_login_url)

    try:
      current_url = driver.current_url

      time.sleep(5)

      # Find and fill in the email input field
      email_input = driver.find_element("name", "identifier")
      email_input.send_keys(email)
      email_input.send_keys(Keys.RETURN)

      print('Email Entered')

      current_url = driver.current_url
      print(f"Current URL: {current_url}")

      time.sleep(5)

      password_input = driver.find_element(By.NAME, 'Passwd')
      password_input.send_keys(password)
      password_input.send_keys(Keys.RETURN)

      print('Password Entered')

      current_url = driver.current_url
      print(f"Current URL: {current_url}")

      time.sleep(5)
    except:
      print('Not able to login')
      driver.save_screenshot('<PATH TO YOUR FOLDER>/error.png')

4. Search on Google, Click on Show More Button Scrapre Links

The following are functions to

1. Search on Google by finding the search bar using the class name

2. Clicking on a show more button which shows up when the AI Overview snippet is available

3. Scraping the links that are inside the AI Overview Snippet. It also checks if your domain name is available in the AI Overview Snippet. So replace “my_domain_name” with your “domain_name”

#function to search
def search(url, class_name, search_term):
    driver.get(url)
    input_element = driver.find_element(By.CLASS_NAME, class_name)
    input_element.send_keys(search_term, Keys.ENTER)

#function to click on any button
def click_button(class_name):
    button = driver.find_element(By.CLASS_NAME, class_name)
    button.click()

#function to scrape links
def scrape_links(class_name, query):
    links = driver.find_elements(By.CLASS_NAME, class_name)
    for j in links:
        link = j.get_attribute('href')
        query_link = query + '@' + link
        all_links.append(query_link)
        if 'my_domain_name' in link:
            query_link = query + '@' + link
            my_domain_link.append(query_link)

Note that these Class names can change in the future, so to get the class names in the future, simply right-click on the element and copy the class name.

5. Declare Variables to store Values, Run Login Function, Selecting terms to Search

Next, we will declare variables to store the values. In the result we will store the status of whether AI Overview is visible or not, then we will check if our domain link is available or not and store it inside my_domain_link, then all the links get stored inside all_links, and queries we are searching for will be stored inside queries variable.

Then we run the function of login. Here you have to replace the placeholders with your actual account credentials.

Then we define the Google URL, we are adding parameters of Country and Language just to be sure we are seeing US Search results.

And then we will define the search terms to find on the page. So, initially, when AIO was in beta, multiple terms and phrases were showing up above AI Overview. But now there is just one term - “AI Overview”. In the future, if these terms change, you can alter them to the latest ones.

#declare variables to store the values
result = []
my_domain_link = []
all_links = []
queries = []

login('email_id','password') #replace with your email and password

url_to_visit = 'https://www.google.com/?gl=us&hl=en'

#search_terms
search_term_0 = 'AI Overview'
# search_term_1 = 'Generative AI is experimental'
# search_term_2  = 'AI overviews are experimental'
# search_term_3 = 'Get an AI Overview for this search?'
# search_term_4 = 'overview is not available for this search'
# search_term_5 = 'Try again later.'
search_term_6 = 'Our systems have detected unusual traffic'

6. Starting the Loop for Search and Scrape

Now we are starting the loop to search and scrape.

1. If the driver is not able to search, then it will print “Cannot Search”. Else it will print the query number it is searching for.

2. After searching it will scrape the source.

3. If AI Overview is found,

   1. Then it will click on the “Show More” button

   2. Takes the screenshot and saves it to the given folder path

   3. Append the Result to the result variable

   4. Scrape links and append them to the all_links variable

   5. Check if your domain name is available and then store the link to my_domain_link if available.

4. Now after 40 to 50 queries, Google is definetly going to block you. If it blocks you then the page will have the phrase “Our systems have detected unusual traffic”.

   1. If Google blocks us, then we wait randomly for 5 to 7 minutes and try again.

5. If nothing is found then it will append “AI Overview Not Found” in the result variable.

#starting the loop

for i, j in zip(query_list, range(0, number_of_queries)):
    try:
        print('current_query_number: ' + str(j))

        try:
            search_bar_class = 'gLFyf'
            search(url_to_visit, search_bar_class, i)
            time.sleep(3)
        except:
            print('Cannot Search')
            result.append('cannot search')
            queries.append(i)

        
        page_source = driver.page_source #scraping page source

        #screenshot path and file name
        screenshot_file_path = '/Users/username/Desktop/screenshots/' #replace with your path
        screenshot_file_name = screenshot_file_path + i[:200] + '.png'
        
        if search_term_0 in page_source:
            try: 
                try:
                    click_button('ni04u') #show more button
                except:
                    click_button('in7vHe') #show more button

                scroll = 'window.scrollBy(0, 30)'
                driver.execute_script(scroll)
                time.sleep(1)

                result.append('AI Overview Found')
                print('AI Overview Found')
                queries.append(i)
                scrape_links('ibUR7b', i)
                driver.save_screenshot(screenshot_file_name)
                print('Screenshot saved')
            
            except:
                print('Some error even if AI Overview is found')
                result.append('AI Overview Found')
                queries.append(i)
                all_links.append('Link Not Found')
                my_domain_link.append('Link Not Found')
                driver.save_screenshot(screenshot_file_name)
                print('Screenshot Saved')
            
        elif search_term_6 in page_source:
            print('Google is Blocking')
            result.append('Google Blocked')
            queries.append(i)
            driver.save_screenshot(screenshot_file_name)
            time.sleep(random.randint(300,400))
        
        else:
            result.append('AI Overview Not Found')
            queries.append(i)
            driver.save_screenshot(screenshot_file_name)
            print('Screenshot Saved')
        
        time.sleep(random.randint(7,10))

    except:

        final_df = pd.DataFrame({'Query': queries, 'AI overview Status': result})

        my_domain_links = pd.DataFrame(my_domain_link, columns=['My Domain Links'])
        my_domain_links = my_domain_links.drop_duplicates(subset=['My Domain Links']).reset_index(drop=True)

        all_links = pd.DataFrame(all_links, columns=['all_links'])
        all_links = all_links.drop_duplicates(subset=['all_links']).reset_index(drop=True)

7. Saving the Data to CSV File

Once the loop is completed, we will quit the browser and create Dataframes from the data we stored in our variables. And save the Dataframes to a CSV file.

You can also extend this script to upload these dataframes to Google Sheets using gpread and gspread_dataframe.

driver.quit()

final_df = pd.DataFrame({'Query': queries, 'AI overview Status': result})

my_domain_links = pd.DataFrame(my_domain_link, columns=['My Domain Links'])
my_domain_links= my_domain_links.drop_duplicates(subset=['My Domain Links']).reset_index(drop=True)

all_links = pd.DataFrame(all_links, columns=['all_links'])
all_links = all_links.drop_duplicates(subset=['all_links']).reset_index(drop=True)

print('SGE Output')
print(final_df)

print('My Domain Links')
print(my_domain_links)

print('All Links')
print(all_links)

final_df.to_csv(r'sge_output.csv',encoding='utf-8')
my_domain_links.to_csv(r'my_domain_links.csv',encoding='utf-8')
all_links.to_csv(r'all_links.csv', encoding='utf-8')

Important Note: This script takes a lot of time to run because we cannot send too many concurrent requests to Google. So it is best to run it for like 200 to 300 queries which should take around 2 to 3 hours.

Somre Requirements before running the script -

1. You need to be on US IP Address. VPNs will work but not the ones which are browser extensios.

2. You should have Chrome Browser installed

3. Script will take few hours to run, so make sure your computer is active for longer time.

Following is the whole script -

from seleniumbase import Driver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
import time
import pandas as pd
import random


df = pd.read_csv(r'query_list.csv',encoding='latin-1')

df = df.replace("/"," ",regex=True)

print(df)

query_list = df['query'].values.tolist()

#test_queries
#query_list = ['what is digital marketing','features of macbook pro m3','write few lines about christmas','what are the properties of metal']

print(query_list[:5])

number_of_queries = len(query_list)

#path to chromedriver
driver = Driver(uc=True, headless=True)

#login to google manually
google_login_url = 'https://accounts.google.com/v3/signin/identifier?hl=en-gb&ifkv=AaSxoQyH1Qet0VlLGN7HL7CD1zRZkw0dVFB4bXApnkylfeDCW4kx3vmK9HBjvDjfi4a6yi3WFJJfug&flowName=GlifWebSignIn&flowEntry=ServiceLogin&dsh=S1544068279%3A1715766753444785&ddm=0'

#function to login to google
def login(email,password):
    driver.get(google_login_url)

    try:
      current_url = driver.current_url

      time.sleep(5)

      # Find and fill in the email input field
      email_input = driver.find_element("name", "identifier")
      email_input.send_keys(email)
      email_input.send_keys(Keys.RETURN)

      print('Email Entered')

      current_url = driver.current_url
      print(f"Current URL: {current_url}")

      time.sleep(5)

      password_input = driver.find_element(By.NAME, 'Passwd')
      password_input.send_keys(password)
      password_input.send_keys(Keys.RETURN)

      print('Password Entered')

      current_url = driver.current_url
      print(f"Current URL: {current_url}")

      time.sleep(5)
    except:
      print('Not able to login')
      driver.save_screenshot('<PATH TO YOUR FOLDER>/error.png')

#function to search
def search(url, class_name, search_term):
    driver.get(url)
    input_element = driver.find_element(By.CLASS_NAME, class_name)
    input_element.send_keys(search_term, Keys.ENTER)

#function to click on any button
def click_button(class_name):
    button = driver.find_element(By.CLASS_NAME, class_name)
    button.click()

#function to scrape links
def scrape_links(class_name, query):
    links = driver.find_elements(By.CLASS_NAME, class_name)
    for j in links:
        link = j.get_attribute('href')
        query_link = query + '@' + link
        all_links.append(query_link)
        if 'my_domain_name' in link:
            query_link = query + '@' + link
            my_domain_link.append(query_link)


#declare variables to store the values
result = []
my_domain_link = []
all_links = []
queries = []

login('email_id','password') #replace with your email and password

url_to_visit = 'https://www.google.com/?gl=us&hl=en'

#search_terms
search_term_0 = 'AI Overview'
# search_term_1 = 'Generative AI is experimental'
# search_term_2  = 'AI overviews are experimental'
# search_term_3 = 'Get an AI Overview for this search?'
# search_term_4 = 'overview is not available for this search'
# search_term_5 = 'Try again later.'
search_term_6 = 'Our systems have detected unusual traffic'

#starting the loop

for i, j in zip(query_list, range(0, number_of_queries)):
    try:
        print('current_query_number: ' + str(j))

        try:
            search_bar_class = 'gLFyf'
            search(url_to_visit, search_bar_class, i)
            time.sleep(3)
        except:
            print('Cannot Search')
            result.append('cannot search')
            queries.append(i)

        
        page_source = driver.page_source #scraping page source

        #screenshot path and file name
        screenshot_file_path = '/Users/username/Desktop/screenshots/' #replace with your path
        screenshot_file_name = screenshot_file_path + i[:200] + '.png'
        
        if search_term_0 in page_source:
            try: 
                try:
                    click_button('ni04u') #show more button
                except:
                    click_button('in7vHe') #show more button

                scroll = 'window.scrollBy(0, 30)'
                driver.execute_script(scroll)
                time.sleep(1)

                result.append('AI Overview Found')
                print('AI Overview Found')
                queries.append(i)
                scrape_links('ibUR7b', i)
                driver.save_screenshot(screenshot_file_name)
                print('Screenshot saved')
            
            except:
                print('Some error even if AI Overview is found')
                result.append('AI Overview Found')
                queries.append(i)
                all_links.append('Link Not Found')
                my_domain_link.append('Link Not Found')
                driver.save_screenshot(screenshot_file_name)
                print('Screenshot Saved')
            
        elif search_term_6 in page_source:
            print('Google is Blocking')
            result.append('Google Blocked')
            queries.append(i)
            driver.save_screenshot(screenshot_file_name)
            time.sleep(random.randint(300,400))
        
        else:
            result.append('AI Overview Not Found')
            queries.append(i)
            driver.save_screenshot(screenshot_file_name)
            print('Screenshot Saved')
        
        time.sleep(random.randint(7,10))

    except:

        final_df = pd.DataFrame({'Query': queries, 'AI overview Status': result})

        my_domain_links = pd.DataFrame(my_domain_link, columns=['My Domain Links'])
        my_domain_links = my_domain_links.drop_duplicates(subset=['My Domain Links']).reset_index(drop=True)

        all_links = pd.DataFrame(all_links, columns=['all_links'])
        all_links = all_links.drop_duplicates(subset=['all_links']).reset_index(drop=True)

driver.quit()

final_df = pd.DataFrame({'Query': queries, 'AI overview Status': result})

my_domain_links = pd.DataFrame(my_domain_link, columns=['My Domain Links'])
my_domain_links= my_domain_links.drop_duplicates(subset=['My Domain Links']).reset_index(drop=True)

all_links = pd.DataFrame(all_links, columns=['all_links'])
all_links = all_links.drop_duplicates(subset=['all_links']).reset_index(drop=True)

print('SGE Output')
print(final_df)

print('My Domain Links')
print(my_domain_links)

print('All Links')
print(all_links)

final_df.to_csv(r'sge_output.csv',encoding='utf-8')
my_domain_links.to_csv(r'my_domain_links.csv',encoding='utf-8')
all_links.to_csv(r'all_links.csv', encoding='utf-8')

If you are facing any issues, feel free to contact me on Linkedin.

Connect on Linkedin

Subscribe now

Share

Usually, anyone quickly checks using the “site:example.com” operator on Google.

But here the challenge was the client had a list of 100+ websites.

The goal was to understand which website is the smallest with a good amount of traffic. That may give us an idea of where are the low-hanging fruits.

Now that’s where SERP API helps!

What is SERP API? SERP API is an API that helps you scrape anything from Google results. Literally, anything! Here are things you can fetch from SERP API -

How to do it?

First, you need to sign up on SERP API and get your API Key. The first 100 searches are free.

And then you only need the below script to scrape the number of google search results.

Let’s understand the code.

Importing Libraries.

from serpapi import GoogleSearch
import pandas as pd

Make a list of all the queries you would search for on Google using the site operator.

And then declare 2 lists, the number of pages, and site names so that we can keep on appending our data to these lists.

queries = ['site:example1.com', 'site:example2.com'] #add all the queries here.

nb_pages = []
site_name = []

Write a for loop to search for these queries and get a JSON output from SERP API.

Since SERP API gives output as a nested dictionary you need to fetch the exact values you want.

Keep on appending the data to the lists we made above. Once you have everything in the list just create a Panda data frame out of it and save it as CSV.

for i in queries:
    params ={
        "engine": "google",
        "q": i,
        "api_key": "Enter Your SERP API Key here"
    }

    search = GoogleSearch(params)
    results = search.get_dict()
    # organic_results = results["organic_results"]

    search_results = results.get('search_information', {}).get('total_results')
    
    query = results.get('search_information', {}).get('query_displayed')

    nb_pages.append(str(search_results))
    site_name.append(str(query))

    print(str(search_results) + ' ' + str(query))


df_pages = pd.DataFrame(nb_pages)
df_sites = pd.DataFrame(site_name)

That’s it. You will get a list of a number of search results for each query.

There are a lot of other cool things you can do from SERP API. You can sign up and also play in their playground - https://serpapi.com/playground.

Hope this helped. Thanks for reading. Sharing is caring!

Subscribe now

Few reasons for slow indexing rate are -

1. New website

2. Content is not very unique

3. Google already have a lot of URLs indexed in that category

First 2 reasons matched with my clients website.

They are new, with not so unique content. But they have very less competitors.

Bigger Problem - They have thousands of pages published.

So I decided to do two things -

1. Improve the internal linking

2. Use Google Indexing API

Since we were on low engineering bandwidth, first was not possible at the time.

I went ahead and started sending Bulk Indexing Request to Google.

Here are the results -

We went from 0 to 33K in 7 Days!

Following are the steps on how to do it -

1. Enable Indexing API in Google Cloud Console

2. Creating Service Account

3. Creating Credentials

4. Adding service account email to Search Console

Before these steps, you will have to make sure that you have permission to send Indexing requests using API.

Click here to visit the Indexing API tester

You should land on this page -

If the above screen did not appear, follow the below steps -

1. Search for “Indexing API” in the search bar, and click on “Indexing API” under “Products”. Here is direct URL.

2. Click on “Reference” tab

3. Click on “publish” under “urlNotifications”

Then add the following details in the Request body -

Scroll below and click on “Execute” button.

If it responds with “200” status, then you have the permission else it will respond with Error 403 - Permission denied.

In that case, make sure you have Owner-level access in Search console.

Now that you have verified it, let’s start with the steps.

Here is the loom video on how to do it.

Enabling Indexing API

1. Visit Google Cloud Console

2. Create a project if you haven’t

3. Search for Indexing API

Click on “Enable”

Creating a Service Account Email

After Enable Indexing API, you will land on its Dashboard.

1. Click on “Credentials” on the left menu bar.

1. Click on “Manage Service Accounts”

2. Click on “Create Service Account” at the top

3. Add any service account name you want to and click “Create and Continue”

4. Under select, a role, search for “Owner” and click on “Continue”

5. Click “Done” and then copy the Email into some notepad shown on the display

6. Make sure to add the e-mail as “Owner” in your Google search console property.

Creating and Download Credentials File

Click on the 3 dots of this Service Account Email and then select “Manage Keys”

Click on “Add Key” > “Create”

This will download the Credentials JSON file. Make sure to save it in some important folder.

Adding service account email to Search Console

Now add the email you copied in the second step to your Search console and give it owner-level access.

The next step is to prepare a CSV file with maximum of 200 URLs in it. Make sure to name the column name “url”. Download the sample CSV file here.

Visit this tool.

Upload the CSV file in the first option.

Upload the credentials file in the second option. (It will be temporarily stored and deleted later).Click on Request Indexing.

Done!

It will return with the response received from the Google Indexing API.

Following is the complete Python script if you want to do it yourself locally -

from oauth2client.service_account import ServiceAccountCredentials
from googleapiclient.discovery import build 
from googleapiclient.http import BatchHttpRequest
import httplib2
import json
import pandas as pd


#creating url list
all_urls_df = pd.read_csv('PATH TO CSV FILE')
top_200_urls = all_urls_df['url'].values.tolist()

#creating request body
values_list = []

for i in range(0,200):
    values_list.append("URL_UPDATED")

zip_iterator = zip(top_200_urls, values_list)

a_dict = dict(zip_iterator)

requests = a_dict
# print(requests)


JSON_KEY_FILE = ('PATH TO JSON KEY FILE')

SCOPES = [ "https://www.googleapis.com/auth/indexing" ]

ENDPOINT = 'https://indexing.googleapis.com/batch'

#Credentials
credentials = ServiceAccountCredentials.from_json_keyfile_name(JSON_KEY_FILE, scopes = SCOPES)
http = credentials.authorize(httplib2.Http())

#building the service
service = build('indexing', 'v3', credentials=credentials)

#creating function to display the response by api
def insert_event(request_id, response, exception):
    if exception is not None:
        print(exception)
    else:
        print(response)

batch = service.new_batch_http_request(callback=insert_event)

# creating for loop to add urls and api type in batch function
for url, api_type in requests.items():
    batch.add(service.urlNotifications().publish(body={"url": url, "type": api_type}))

#running batch function
batch.execute()

Thanks for reading!

Sharing is caring!

Facing issue? Tweet @stanabk

Share

Subscribe now

Read SEO Snippets in the Substack app

Available for iOS and Android

Get the app

Thanks to the python package people-also-ask.

In this snippet, I am going to show you how you can extract the related questions of any topic from Google’s PAA section -

The first step is to obviously install the required packages. Type the following in CMD -

pip install people-also-ask

And then write these simple four lines of code -

import people_also_ask

topic = 'python seo'
questions = people_also_ask.get_related_questions(topic)

for i in questions:
print(i)

Here is what output looks like -

It becomes a bit complex when you want to do this in bulk and want to clean up the data.

So I have made this tool where you can upload the CSV file and get all the related questions - Also Ask

I will soon upload the whole code on how to perform this task in bulk.

Hope this helped!

Thanks for reading.

Tweet @stanabk if facing any issues.

Sharing is caring!

Share

He is really helping the Digital Marketing community a lot by making this awesome tool free to use.

So, in this snippet, I am going to use Advertools for checking the status code of 1000+ URLs in my sitemap.

I first thought of using Screaming Frog but then it will crawl and extract everything, yeah I can untick those options but I found Advertools faster and easier

So let's first grab the sitemap. I am gonna use the sitemap of great Bill Slawski’s website - SEO by the Sea

Now that we have the sitemap let’s just jump right into the code.

The first step is to install and import the required packages.

Type the following in CMD to install packages -

pip install panadas

pip install advertools

Import the packages -

import pandas as pd
import advertools as adv

The next step is to convert the sitemap to a data frame and then to a list

sitemap_url = 'https://www.seobythesea.com/post-sitemap1.xml'

sitemap_df = adv.sitemap_to_df(sitemap_url)

url_list = sitemap_df['loc'].tolist()

Then we will feed this list of URLs to advertools and check the response headers. We will then store the URLs and their status code in a CSV file.

adv.crawl_headers(url_list, 'url_status.jl')

headers_df = pd.read_json('url_status.jl', lines = True)

print(headers_df[['url','status']])

headers_df[['url','status']].to_csv(r'C:\Users\ABShukla\Documents\url_status_checker.csv')

Here is what output looks like -

Full code snippet -

import pandas as pd

import advertools as adv

sitemap_url = 'https://www.seobythesea.com/post-sitemap1.xml'

sitemap_df = adv.sitemap_to_df(sitemap_url) url_list = sitemap_df['loc'].tolist()

adv.crawl_headers(url_list, 'url_status.jl')

headers_df = pd.read_json('url_status.jl', lines = True)

print(headers_df[['url','status']])

headers_df[['url','status']].to_csv(r'C:\Users\ABShukla\Documents\url_status_checker.csv')

Hope this helped!

Thanks for reading.

Tweet @stanabk if facing any issues.

Sharing is caring!

Share

Analyzing 1000+ pages one by one will be obviously a boring and tedious task.

But thanks to Google, they have made PageSpeed API available.

And thanks to Screaming Frog, they let us connect the PageSpeed API key with their tool.

This solves a very big problem. We can use PageSpeed API Key and Screaming frog to analyze PageSpeed data of 1000+ URLs.

Let’s get started.

The first step is to get the PageSpeed API Key.

Click on this URL.

Then click on this button which says “Get a key”

Then it will ask you to select a project, if you do not have any project created then you will have to create one.

Choose the project and click Next.

Once you click next, it will share the API Key with you.

It looks something like this - AIzaSyAj3NwrzcQuM8YZWAxPXTASjzXv-sMbzro

(That’s a fake key btw)

This is your secret API key, so do not share it with anyone.

Now that you have an API key, open up Screaming Frog.

Go to Configuration > API Access > PageSpeed Insights

Enter your PageSpeed API Key and click on “Connect”

If your API Key is correct it will get connected.

The next step is to select the metrics you want to fetch.

You can open up these metric groups and select the specific metrics you want to analyze.

I usually go for Lighthouse and CrUX metrics.

Next, we need to make sure that Screaming Frog does not waste time in crawling the URLs. Because we just need PageSpeed metrics data.

There is no ideal way to do that, but there is a workaround.

Navigate to Configuration > Spider > Extraction

Uncheck everything. This way crawler will not waste time extracting the data from the pages.

Now you are all set to start the crawl. So hit that “Start” button.

And go to the “PageSpeed” tab once the crawl is finished.

You will have all your data there.

It will look something like this -

That’s all!

Thanks for reading.

Sharing is caring!

Tweet @stanabk if any queries.

Share

Subscribe now

I am sure a lot of people would want to do this for multiple reasons.

Maybe for building internal linking opportunities, or brand mentions.

Screaming Frog has made it easy to perform such tasks.

So, let’s find an internal linking opportunity for forbes.com.

The first step is to install the Screaming Frog if you haven’t yet. You can download it from here.

The next step is to fetch the top 1000 pages. I am going to use Ahrefs for that, you can also use Google search console for your own website.

Then, we will check which pages are not getting much traffic. I found one such page from Ahrefs -

https://www.forbes.com/sites/shaharziv/2020/05/12/blue-state-bailout-red-state-residents-received-largest-stimulus-checks-and-millions-in-federal-aid/

It looks like it is related to “stimulus”. So, we will perform a custom search on the top 1000 pages for stimulus.

So let’s open up the Screaming frog. And under Configuration > Custom you will see “Search”

Click on Search, and then add “stimulus” (your keyword) in the Text and make sure you select HTML in the drop-down menu.

Once you have added the keyword, hit Ok.

And start the crawl!

Once the crawl is finished, you can go “Custom Search” tab, and click on “Search 1 (Contains)”.

On the right side, you can see how many occurrences are there for “stimulus” in that specific URL. We found 23 URLs that mention “Stimulus”

Now you can download the report, sort it in descending order. And go through the URLs to find the internal linking opportunity.

For example, this URL has a lot of mentions of “stimulus” -

You can perform this analysis for multiple keywords and find thousands of internal linking opportunities!

Thanks for reading.

Sharing is caring!

Tweet @stanabk for any queries.

Share

Subscribe now

Now that is not a very good User experience.

Not only for you but also for Search Engine bots like Google.

It creates confusion for users and also for bots.

Hence, it is always a best practice to keep your Meta Titles and first Heading either similar or exactly the same.

But let’s say you have thousands or even millions of URLs on your website and you want to analyze the similarity percentage between Titles and Headings.

Because even if it is not exactly the same, it may be similar.

For example, “Tips for writing SEO-friendly content” and “SEO-Friendly content writing tips” are very similar.

Hence, it is not possible to find a Title-Heading mismatch by just using excel IF Formula (I wish it was though!).

But it is not as complicated as it looks. A simple python script can help us achieve this.

To keep it simple, I am going to take random 100 URLs from this website -

www.additudemag.com

You can grab the list of URLs here with Title and Heading.

We are going to use a python package called: fuzzywuzzy.

Fuzzywuzzy uses fuzzy logic to compare two texts and it gives a similarity score as an output.

Let’s install the required packages first -

pip install fuzzywuzzy

pip install pandas

pip install python-Levenshtein

Python-Levenshtein is an optional package to avoid warnings. In case you are facing issues installing this package on Windows, install it using its wheel file.

Now let’s jump right into the code -

Import required packages

from fuzzywuzzy import fuzz
from fuzzywuzzy import process
import pandas as pd

The next step is to download the above Google sheet as a CSV and store it somewhere on your local storage.

Once we have the file ready, we will use them to create Panda Dataframes.

df = pd.read_csv(r'PATH TO CSV FILE')

Now that we have our data frame ready, let’s convert them to a list so that we can run them through a loop one by one.

headings = df['Heading'].values.tolist()
titles = df['Title'].values.tolist()

sim_score = [] #declaring a blank list to store similarity score values

The next step is to run a for loop on both lists and apply fuzzy logic to them.

for i,j in zip(headings, title):
sim_score.append(fuzz.ratio(i,j))

The above loop will feed the heading and title to the fuzzy logic and fill the sim_score list with the similarity score values.

Now we will merge the two data frames and store them in a CSV file.

similarity_score_df = pd.DataFrame(sim_score)
similarity_score_df.columns = ['similarity_score']
final_df = pd.concat([df, similarity_score_df], axis=1)
final_df.to_csv(r'Path to your output csv file')

Here is what the final output CSV file looks like -

Full code below -

from fuzzywuzzy import fuzz
from fuzzywuzzy import process
import pandas as pd

df = pd.read_csv(r'PATH TO CSV FILE')

headings = df['Heading'].values.tolist()
titles = df['Title'].values.tolist()

sim_score = []

for i,j in zip(headings, titles):
    sim_score.append(fuzz.ratio(i,j))

similarity_score_df = pd.DataFrame(sim_score)
similarity_score_df.columns = ['similarity_score']

final_df = pd.concat([df, similarity_score_df], axis=1)

final_df.to_csv(r'PATH TO OUTPUT FILE')

Thanks for reading!

Sharing is caring

Tweet if facing any issues: @stanabk

Share

Subscribe now