How to get Service Provider back in Google Analytics

Difficulty: 5/5

Requirements: Google Tag Manager & Google Analytics with 1 spare custom dimension

If you’re seeing Service Provider (not set) in your Analytics reports – we share your pain. Unfortunately, over the past few months, we’ve heard chatter that Google were planning to drop two relatively unknown but important dimensions: Service Provider and Network Domain. It seems that as of Feb 4th 2020, these dimensions are no longer collecting data.

A couple of use-cases for having these dimensions:

• Bot Filtering
• Direct Traffic Analysis

There are more but these are the two we use them for on a regular basis.

Filtering Bot Traffic using an IP Lookup Service

This is the focus of this post – get Service Provider back online ASAP. In the same way Google did it, we can just run a lookup against a user’s IP address and determine their service provider.

To do this we’re going to use an IP Lookup Service which has API access. We’ve built a script which can run in GTM to do all the legwork and provide GA with the ISP info as a new custom dimension.

There are a lot of lookup services available which can help us out here. We’ve chosen one which is free, allows HTTPS requests (not a lot of free ones offer this) and isn’t hugely expensive when you decide you need to pay for more lookups. It also doesn’t require an API key or any authentication, you simply call the JSON API url in JS to get the lookup info. The free service has a request limit of 20 lookups per minute. The paid version is unlimited and costs EUR 20 a month which isn’t a bad compromise if you are in dire need of ISP data. We have built this guide to throttle requests per session to help with this limit. If you have a lot of active users on your site at any given time, some of these requests will be ignored and no ISP info will be given.

If you run this JSON request url in your browser, it’ll return your own lookup info:

https://extreme-ip-lookup.com/json/

Example Output

{
   "businessName" : "",
   "businessWebsite" : "",
   "city" : "Sheffield",
   "continent" : "Europe",
   "country" : "United Kingdom",
   "countryCode" : "GB",
   "ipName" : "host12-345-67-890.in-addr.btopenworld.com",
   "ipType" : "",
   "isp" : "British Telecommunications PLC",
   "lat" : "53.38297",
   "lon" : "-1.4659",
   "org" : "British Telecommunications PLC",
   "query" : "12.345.67.890",
   "region" : "England",
   "status" : "success"
}

This info is perfect for our needs and also gives us some extra data on the house. For now, let’s just focus on grabbing the key pair labelled “isp”

We’ve done the hard work of fetching, parsing and providing this data as a dataLayer push below:

IP Lookup Code

<script>
  (function() {
    
    // Session cookie to block future attempts to lookup the ISP - only attempt once per session
    try {
        {{JS - Set Cookie}}("gtm_isp_lookup", true, "", "/", {{URL - Hostname no WWW}});
    } catch (e) {

    }

    var endpoint = 'https://extreme-ip-lookup.com/json/';
    var xhr = new XMLHttpRequest();
    
    xhr.onreadystatechange = function () {
      if (this.readyState == 4 && this.status == 200) {
        var response = JSON.parse(this.responseText);
        if (response.status !== 'success') {
          return
        }
    
        try {
          window.dataLayer = window.dataLayer || [];
          window.dataLayer.push({
            'event': 'isp-lookup',
            'isp-lookup': {
              'lat': response.lat,
              'lon': response.lon,
              'isp': response.isp,
              'org': response.org,
              'continent': response.continent,
              'country': response.country,
              'region': response.region,
              'city': response.city,
              'businessName': response.businessName,
              'businessWebsite': response.businessWebsite
            }
          });
        } catch (e) {}
      }
    };
    xhr.open('GET', endpoint, true);
    xhr.send();

  })();
</script>

To walk through this code:

1. Save a first-party cookie for the duration of the session to prevent spamming the lookup service. This is a simple flag to tell GTM to not keep re running the code
2. Open the connection to the service
3. Check it’s returning a good result
4. Parse the JSON object into a variable
5. Send this variable information to the dataLayer as an event with all details of the lookup where available
6. Close the lookup connection

With this information sent out as a dataLayer event, we can capture it using a Custom Event trigger and pass this information to Google Analytics as a non-interactive Event with a custom dimension set for the ISP.

This code relies on other variables and triggers we will run through below.

Note, it won’t run outside of GTM without modification.

New GTM Variable: 1P Cookie – ISP

Type: First Party Cookie

Value: gtm_isp_lookup

New GTM Variable: URL – Hostname no WWW

Type: URL

Value: Host Name

Tick: Strip “www.”

New GTM Variable: JS – Set Cookie

Type: Custom JavaScript Code

Value: (code snippet below)

function() {
  return function(name, value, ms, path, domain) {
    if (!name || !value) {
      return;
    }
    var d;
    var cpath = path ? '; path=' + path : '';
    var cdomain = domain ? '; domain=' + domain : '';
    var expires = '';
    if (ms) {
      d = new Date();
      d.setTime(d.getTime() + ms);
      expires = '; expires=' + d.toUTCString();
    }
    document.cookie = name + "=" + value + expires + cpath + cdomain;
  }
}

Kudos to Simo for this one. It’s a nice little function saved as a JS variable which we can use in other parts of GTM as a function to process cookie saving.

New GTM Tag: SCRIPT – ISP Lookup

Type: Custom HTML

Simply paste in the script we shared above

New GTM Trigger: GTM Load – ISP Not Set

Type: Window Loaded

Condition: 1P Cookie – ISP does not equal true

Save the trigger to the tag and save the tag. This trigger prevents us from hitting the lookup service per hit and instead will only fire the code when the First Party Cookie isn’t set. As the cookie is set to Session expiry, we are throttling the lookup attempts per user. Also, doing an ISP lookup per hit would be a bit overkill considering the likelihood of it changing is slim.

New Custom Dimension in Google Analytics

Create a new Custom Dimension called ISP. Head over to GA > Admin > Custom Definitions > Custom Dimensions

Click “+ New Custom Dimension”

Name it ISP and set Scope to Session

Once saved, you’ll be able to get the Index number and apply it to a new Event in Google Tag Manager later on. Make a note of it for now.

New GTM Variable: EDLV – ISP – ISP

We need to grab the Service Provider data and store it as a Data Layer Variable in GTM. This will be used later to send it to Google Analytics. We will set a Default Value on this variable to allow us to count the number of sessions recording no ISP information.

Type: Data Layer Variable

Value: isp-lookup.isp

Set Default Value: (not set)

Format Value: Change Case to Lowercase

New Google Analytics Event: GA – Event – ISP Lookup

This Event is acting as a transport bus so we can deliver the new ISP info to GA for the Session. You’ll need to adjust the custom dimension index to match a newly created custom dimension in Google Analytics. Ours is 99 for demonstration purposes.

New Trigger: CE – ISP Lookup Completed

The code we’ve written fires a dataLayer event called “isp-lookup” – when this happens, it means the ISP information was searched for. This will be how we fire our GA Event to pass any ISP information found over to Google Analytics.

Add the trigger to the new GA Event and save it.

With all of these pieces in place, and deployed, you will start to see new ISP data populating under your Custom Dimension. Side by side, the data is similar and will aid in filtering out any unwanted bot traffic from your Analytics production views.

For quick analysis, we have compared the data side by side in Data Studio:

If you know the traffic you’re looking to filter, you can build a new view filter in GA and start blocking bot traffic right away.

As a quick example, we could do something like the below screenshot the same way we would using the now deprecated dimension Internet Service Provider.

If you’re feeling particularly lazy, here’s the GTM Container Export I used for this demonstration. Be sure to change the UA ID for the GA Event.

GTM Container Download

Note: this is not 100% plug and play, you should always test imported containers you haven’t built for configuration updates, errors and malicious code usage.

Data Privacy Considerations

• When you utilise a 3rd party vendor such as Extreme IP, they become a Data Processor as they take the user’s IP Address to identify Internet Service Provider information.
• You may want to consider this update as part of your privacy policy if you are looking to conform with regulations such as GDPR or CCPA which require explicit consent to process personally identifiable information (PII)
• Our method does not store user IP addresses in any format, raw or hashed. We only store the Service Provider information as a Custom Dimension.

Author William Julian-Vicary

Posted February 11, 2020