Adobe analytics testing automation: Easy data validation

Adobe analytics testing is an integral part of most data analysts in web analytics arena.

Adobe analytics logo

Yes it is redundant, repetitive but at the same time, unavoidable and therefore, most important part of our lives! ๐Ÿ™„

So, to all my fellow WAPs – (Web Analytics Professionals) – I present to you an automated way to test and record adobe analytics beacons coming out of any web-page.   ๐Ÿ˜‚

Background – Thinking about adobe analytics testing

There are times when we have to QA a bunch of URLs for our clients to test and validate all the adobe analytics parameters are recorded correctly.

Then, there are times when we have to record and store all the values of every prop, evar and event that fires on page load. ๐Ÿ™„

Problem with adobe analytics testing

Needless to say, manually loading all these web-pages, using a browser plugin or making use of network call to inspect the adobe analytics calls is hectic and tedious. Especially while handling clients with large website or web analysts dealing with multiple adobe analytics clients at a time. ๐Ÿฅต

Oh I wish if there was some way of adobe analytics automation solution for these type of QA tasks – except from buying license of expensive tag audit tools like observepoint.  But wait, Why only wish? Why not build one for myself? ๐Ÿ˜›  ๐Ÿค”

Yes, I should feed all the URLs to a tool, run a program and go grab a cup of coffee. By the time I return, all the URLs should be QAed and results should be put to a document or excel sheet.

How convenient isn’t it? !! ๐Ÿ˜

Solution Approach

Firstly, there is a buffet of robust programming languages. There are so many open-source tools to pick from. I first wanted to build a comprehensive, stand alone app on JAVA platform and use Selenium web services for network calls. However, having a demanding desk job, frequent long travels and an adorable 2-year old daughter at home, I neither have bandwidth nor the heart for such hard work. So, dropped this idea. ๐Ÿ‘Ž ๐Ÿ™ƒ

Likewise, there is ‘python‘ – one of the most discussed coding language with it’s ‘snake’ ๐Ÿ like flexibility and agility. However, I’ll have to learn a completely new language and syntax for that. ๐Ÿ‘Ž  

Finally, I finally chose the awesome nodeJS to make my wish come true. It is simple, free, has great community and based on my area of expertise – JavaScript. So, if you want this nice little tool to work for you, you better know the basics of JavaScript and concept of nodeJS.  ๐Ÿ‘

Our Ads are non-intrusive and help us to run this blog.
Please add this website as an exception in your ad blocker.

The Solution – Automated adobe analytics testing!

Adobe automation thought process

Let me show you how I imagined the program to work: ๐Ÿค”

Before you proceed, note that this solution is not limited to adobe analytics testing automation but same logic code can be used to automate testing of any other analytics tools too.

Solution : Thought process to build adobe analytics audit tool

Adobe analytics testing automation process

What all do you need for this adobe analytics testing tool? 

  • NodeJS installed in your computer. That’s it.

You can pretty much run this program just after that. However, if you wish to make any changes, you would need a good code editor. – I prefer Visual Studio Code – which is free and my favorite. ๐Ÿงก

Tool Design

It consist of only 5 small files. ๐Ÿ‘

โ€ข One package.json file
โ€ข One Server.js file
โ€ข Two HTML (PUG) files.
โ—‹ One to get URL path
โ—‹ One to display adobe analytics test result
โ€ข One CSS file

Program Structure

Adobe automation project

NodeJS – npm- packages & libraries required: [Refer ‘dependencies in package.json code.

  • http – since it is web related program
  • tableify – to parse end result to a table
  • fs, util, bodyparser,  – file system editor – to read HAR file
  • express – to run local server
  • puppeteer – to load the URL in a headless browser.
  • PUG -which is based on Jade framework – for dynamic web pages.

I’ve also tried to use few libraries to increase usability like to check for a valid URL, if URL exists etc. but they aren’t mandatory if you know what your’e doing. 

Actual code

Program name:- getAdobeCall

File:- package.json

Note that I’ve used nodemon – for easy server operations. (Start and Stop)

{
  "name": "getadobecall",
  "version": "1.0.0",
  "description": "Gets adobe analytics call details on any website",
  "main": "server.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1",
    "start": "nodemon server.js"
  },
  "keywords": [
    "adobe",
    "adobe analytics",
    "analytics",
    "network call",
    "shivanandana",
    "hegde",
    "eVar",
    "prop"
  ],
  "author": "Shivanandana Hegde",
  "license": "ISC",
  "dependencies": {
    "body-parser": "^1.19.0",
    "chrome-har": "^0.11.0",
    "express": "^4.17.1",
    "express-validator": "^6.1.1",
    "fs": "0.0.1-security",
    "http": "0.0.0",
    "nodemon": "^1.19.1",
    "path": "^0.12.7",
    "puppeteer": "^1.19.0",
    "tableify": "^1.1.0",
    "url-exists": "^1.0.3",
    "util": "^0.12.1"
  }
}

getURLForm.pug

doctype html
html
  head
  link(rel='stylesheet', href='/css/styles.css')
  body
  div(align='center')
      h1 Get Adobe Analytics Variables 
      p ...
      p <b>#{caution}</b>
  form(action='/thank' method='post' name='form1' align='center')
      label(name='web-page') Enter URL Below:
      <br/><br/>
      input(type= 'text' name='url')
      <br/><br/>
       button.btn.btn-primary(type='submit') Fetch
  div(align='right')
      p Work of:- Shivanandana Hegde

postResult.pug

doctype html
html
  head
  link(rel='stylesheet', href='/css/styles.css') 
  body
  div(align='center')
        h1 URL requested 
        h3 #{urlVar}
        h1 Adobe analytics variables names and values
  div(align='right')
        p Work of:- Shivanandana Hegde      
  div(align='center')      
        p !{result}

(Main file) server.js

var http = require("http");
var htm;
//from webpage
var tableify = require('  ');
var fs = require('fs');// file system
var obj = '';
var x;
var finalOutput = [];
//validators
const events = [];
//From getNetCal
const observe = [
  'Page.loadEventFired',
  'Page.domContentEventFired',
  'Page.frameStartedLoading',
  'Page.frameAttached',
  'Network.requestWillBeSent',
  'Network.requestServedFromCache',
  'Network.dataReceived',
  'Network.responseReceived',
  'Network.resourceChangedPriority',
  'Network.loadingFinished',
  'Network.loadingFailed',
];
const { promisify } = require('util');
var pageURL1 = '';
const puppeteer = require('puppeteer');
const { harFromMessages } = require('chrome-har');
// list of events for converting to HAR
//For Running Server, Details.
var express = require('express');
var app = express();
var bodyParser = require('body-parser');
var urlencodedParser = bodyParser.urlencoded({ extended: true });
var path = require('path');
//for pug
app.set('views', path.join(__dirname, 'views'));
app.set('view engine', 'pug');
//Give permission to CSS files
app.use(express.static(path.join(__dirname, 'public')));
//Validator library
const { check } = require('express-validator')
//Check if URL exists
const urlExists = require("url-exists");
var validPage;
var server = app.listen(8080, function () {
  var host = server.address().address
  var port = server.address().port
  console.log("Server running & listening at %s:%s Port", host, port)
});


//First Form - GET
app.get('/getAdobeCall', function (req, res) {
  //  res.sendFile(path.join(__dirname +'/getURLForm.html'));
  res.render('getURLForm', { caution: 'URL should start with (http/https)' });
});


app.post('/thank', urlencodedParser, async function (req, res) {
  //res.render('/thank');
  pageURL1 = req.body.url;
  console.log(pageURL1);
  if (!pageURL1.includes('http://') && !pageURL1.includes('https://')) {
    pageURL1 = 'http://' + pageURL1;
  }

  urlExists(pageURL1, function (err, exists) {
    if (exists) {
      //res.end('Good URL');
      validPage = 1;
    }
    else {
      var badurl = '';
      badurl += "<body>";
      badurl += "<br/><br/><br/><br/>"
      badurl += "<div align='center'>"
      badurl += "<h2> BAD URL </h2>"
      badurl += "</div>"
      badurl += "</body>";
      //res.send(badurl);
      console.log("Bad URL is:-" + pageURL1);
      res.end(badurl);
    }

  });
  // event types to observe
  const browser = await puppeteer.launch();
  const page = await browser.newPage();
  // register events listeners
  const client = await page.target().createCDPSession();
  await client.send('Page.enable');
  await client.send('Network.enable');
  observe.forEach(method => {
    client.on(method, params => {
      events.push({ method, params });
    });
  });

  // perform tests
  await page.goto(pageURL1);
  await browser.close();
  // convert events to HAR file
  const har = harFromMessages(events);
  //fs.write('cdw.har', JSON.stringify(har, undefined, 4), 'w');
  await promisify(fs.writeFile)('networkCalls.har', JSON.stringify(har, undefined, 4));
  console.log('File created successfully')
  obj = await JSON.parse(fs.readFileSync('networkCalls.har', 'utf8'));
  //Loop to get the adobe network call.
  for (i in obj.log.entries) {
    x += obj.log.entries[i];
    if (obj.log.entries[i].request.url.includes('b/ss')) {
      finalOutput = obj.log.entries[i].request.queryString; // Caught the adobe output here
    }
  }
  htm = await tableify(finalOutput);

  //POST THE PROCESSD RESULT
  res.render('postResult', {
    urlVar: req.body.url,
    result: htm
  })
  //res.send(reply);
  console.log(pageURL1);
  //console.log(req.body.url);
});

โœ”๏ธ That is All... See the below video illustration to know how this adobe analytics testing tool works.

Scope of improvements

Hey, this is just the basic foundation of my solution. Of-course it has limited functionality as of now. Like:

  • The program can read only 1X1 adobe analytics network calls but may not work for 2X2 network calls.
  • It works only for first Adobe call. (If there are multiple adobe calls on page load)
  • Speaking of page load, this is only for ‘page load’ calls. So, for onClick events, this needs further optimization.
  • Mostly stable (like 85% of the time) but sometime, I’ll have to restart the node/npm to make it work.
  • Add URL validation – to check if URL exists or not.
Our Ads are non-intrusive and help us to run this blog.
Please add this website as an exception in your ad blocker.

There are many other enhancement options to consider like :

Read all URLs from a CSV/Excel file.

for (every row in the excel sheet) 
{
/* Do this operation and 
 store the adobe output in a file. */
}

Check for data in the file
Alert me if there is any error.
  • Create alert mechanism for wrong or missing values in eVar or props or events
  • Create similar tool for other analytics tools like google analytics, coremetrics etc.?

If you made use of this, let me know in comments/email.

Feel free to copy this code and improve it based on your requirement. ๐Ÿ‘

Verdict: Life is good, open source is awesome, this world is wonderful !! 
Thanks to Ryan Dahl and other people who created nodeJS.

Thanks to all the people who’ve created these wonderful nodeJS packages and special thanks to those self-less contributors of stackoverflow!! You guys always reinforce my faith in humanity! ๐Ÿ™‚ ๐Ÿงก

Cheers๐Ÿบ ๐Ÿ™‚

My recent posts on Digital marketing and Analytics:

2 comments

    • harsh raj on May 21, 2020 at 4:27 pm
    • Reply

    Getting error at the time of starting server

    1. Your Node installation has problems. You need to fix that.

Leave a Reply

Your email address will not be published.