In this guide we will explain how to set up BigQuery.
Please be aware that BigQuery does have free usage level BUT will charge for storage and query usage if you exceed the free quota.
What is BigQuery?
BigQuery is serverless data warehouse and analytics platform from Google.
BigQuery is used by many professional analytics teams to store, query and analyse large data sets at scale.
There are really two parts to BigQuery:
Datasets - The datasets are where you store data.
Tables - The tables are where you query the datasets and analyse the data.
Why set up BigQuery and Indexing Insight?
BigQuery allows Indexing Insight customers to unlock advanced SEO & AI analysis.
When you setup the BigQuery feature you will push your Google indexing data daily straight into a BigQuery dataset.
By pushing index data into BigQuery you will allow you to do EVEN MORE with Indexing Insight, including:
Index Reporting – Pull automated Google index data into your dashboards (Looker Studio, Power BI, etc.) to report on your indexing health.
Combine index data – Merge datasets in BigQuery for advanced analysis, so you can get to the root of a problem fast.
Add index data to AI workflows – Feed Google index data into your AI workflows to layer indexing signals into your SEO analysis for actionable insights.
Check out our Data Studio guide here.
How to set up BigQuery and Indexing Insight
To add BigQuery you just need to follow these steps.
You must have a Google Cloud project with verified billing to enable BigQuery.
Step 1: Enable BigQuery in your selected project
Go to your Google Cloud Console.
When you open the console make sure to switch to the project you'd like to export your Indexing Insight data.
Then enable BigQuery in your selected project. To do this go to the sidebar to APIs & Services > Enabled APIs & services. Enable BigQuery API (if enabled then no action needed).
Step 2: Create the dataset within the project
In your selected project, you will need to create a new dataset.
To do this hover over the project in the BigQuery console and click on the three little dots. Then select create dataset.
You will need to fill in a data set ID and select a location.
You can push different website's indexing data to the same dataset. It's just we will push it to a different table.
Step 3: Grant permission to Indexing Insight data dumps
Finally, you will need to grant our tool permission to dump your website's indexing data into the dataset.
To do this go to the AIM and Admin Permission settings for the selected project. And click on Grant Access.
Then you need to add the following New principle to the account:
You will need to add the following roles for this new user:
BigQuery Data Editor
BigQuery Job User
Then hit save.
That's it for Google Cloud and BigQuery.
Step 4: Setup BigQuery export feature in Indexing Insight
Navigate to Account Settings > Integrations > Export data to BigQuery.
You will be taken to the Export data to BigQuery page. And click on the Export data to BigQuery.
Here you will need to input a few details before we can start exporting the data.
Step 5: Input details into the BigQuery export feature
First, you will need to pick an active website you are monitoring.
Then, pick your export type. There are 3 types of exports based on your needs:
Index Reporting - The aggregated data you see in our graphs that can be used to create charts that can be used in dashboards.
Latest Indexing Data - The latest indexing data for all your monitored page URLs that you see in our data tables.
Historic Indexing Data (Coming Soon) - Store current and historic indexing data for each page URL on a daily basis.
You have to pick 1 export type BUT you can create a new table within the dataset to store the other type of data. So, don't worry if you want to store the other data type.
Finally, you need to get the IDs from BigQuery for the selected project ID and dataset. Input these names into the Export settings. The table name is set by default.
The table will be automatically be created within the selected project and dataset.
A separate table will be created for each data export.
You should have BigQuery export settings that look like this:
Hit save. And that's it!
When is data exported to your BigQuery?
The BigQuery data exports will be dumped the next time our scheduler runs (so usually the next day).