Send requests to the Google Indexing API using googleAuthR

Two years ago, Google introduced the Indexing API with the intent of solving an issue that affected jobs/streaming websites – having outdated content in the index. The Google Developers documentation says:

“You can use the Indexing API to tell Google to update or remove pages from the Google index. The requests must specify the location of a web page. You can also get the status of notifications that you have sent to Google. Currently, the Indexing API can only be used to crawl pages with either job posting or livestream structured data.”

Many SEOs are using Indexing API also for non-job-related websites and that’s why I decided to build an R script to try out the API (and it worked).

I’m going to show you how the script works, but don’t forget that there is a free quota of 200 URLs sent per day!

Create the Indexing API Credentials

First of all, you have to generate the client id and client secret keys for the APIs.

Open the Google API Console and go to the API Library.

Open the Indexing API page and enable the API. Then go to the Credentials page and there you’ll find your credentials.

googleAuthR package & Indexing API options

I created the script using the googleAuthR package, which allows you to send requests to Google APIs.

The code takes in input, a character vector of maximum 200 URLs and returns in a data frame the response of the API (see the screenshot below).

You can use the script to update or delete pages. You just have to change the line 38:

Use type = “URL_UPDATED” if you have to update the page
Use type = “URL_DELETED” if you have to remove the page

# LAST UPDATE: 20-03-2021
# Install & Load the packages

install.packages("googleAuthR")
install.packages("tidyverse")
install.packages("readr")

library("googleAuthR")
library("tidyverse")
library("readr")

# Set credentials and scope

clientId <- "PASTE HERE YOUR CLIENT ID"
clientSecret <- "PASTE HERE YOUR CLIENT SECRET" 
scope <- "https://www.googleapis.com/auth/indexing"

options("googleAuthR.client_id" = clientId, 
        "googleAuthR.client_secret" = clientSecret, 
        "googleAuthR.scopes.selected" = scope,
        "googleAuthR.verbose" = 0 # Not mandatory - I just use it to debug the script
        )

# Google API OAuth

gar_auth()

# List of URLs - Daily Limit of 200 URLs

urls <- read_csv("~/Desktop/Your-file-name.csv")
urls <- urls[ ,1]

# indexingApi function - you can use the function to send requestes to the indexing API using urls vector as an input
# It also GET the response from the API and stores it in a data frame

indexingApi <- function(page) {
    
    body <-  list(
                  url = page,
                  type = "URL_UPDATED")
    
    f <- gar_api_generator("https://indexing.googleapis.com/v3/urlNotifications:publish",
                           "POST")
    
    result <- f(the_body = body)
    result <- as.data.frame(result[[6]][[1]][[2]])
    
    return(result)
    
}

# The API responses are stored in a data frame

APIResponse <- map_dfr(
                      .x = urls,
                      .f = indexingApi)

# You can download the API responses as .csv file

write.csv(APIResponse, "Your-file-name.csv")

# LAST UPDATE: 20-03-2021

# Install & Load the packages

install.packages("googleAuthR")

install.packages("tidyverse")

install.packages("readr")

library("googleAuthR")

library("tidyverse")

library("readr")

# Set credentials and scope

clientId <- "PASTE HERE YOUR CLIENT ID"

clientSecret <- "PASTE HERE YOUR CLIENT SECRET"

scope <- "https://www.googleapis.com/auth/indexing"

options("googleAuthR.client_id" = clientId,

"googleAuthR.client_secret" = clientSecret,

"googleAuthR.scopes.selected" = scope,

"googleAuthR.verbose" = 0 # Not mandatory - I just use it to debug the script

)

# Google API OAuth

gar_auth()

# List of URLs - Daily Limit of 200 URLs

urls <- read_csv("~/Desktop/Your-file-name.csv")

urls <- urls[ ,1]

# indexingApi function - you can use the function to send requestes to the indexing API using urls vector as an input

# It also GET the response from the API and stores it in a data frame

indexingApi <- function(page) {

body <- list(

url = page,

type = "URL_UPDATED")

f <- gar_api_generator("https://indexing.googleapis.com/v3/urlNotifications:publish",

"POST")

result <- f(the_body = body)

result <- as.data.frame(result[[6]][[1]][[2]])

return(result)

}

# The API responses are stored in a data frame

APIResponse <- map_dfr(

.x = urls,

.f = indexingApi)