Adobe analytics testing automation: Easy data validation

UPDATE: It’s been over 2 years since I published this post on adobe analytics testing automation. Several people have contacted me for a working version of this tool ever since. I’ve shared all the code required for this project here and recently I’ve published a live, working demo version of the tool.

Check the demo here – https://adobe-analytics.herokuapp.com/
Up-to 3 links can be validated at a time. The idea is to follow the same logic for 100s of URLs in a file!

History:

Adobe analytics testing is an integral part of most data analysts in web analytics arena.

Yes it is redundant, repetitive but at the same time, unavoidable and therefore, most important part of our lives! ?

So, to all my fellow WAPs – (Web Analytics Professionals) – I present to you an automated way to test and record adobe analytics beacons coming out of any web-page.   ?

Jump to Section

Toggle

Background – Thinking about adobe analytics testing

There are times when we have to QA a bunch of URLs for our clients to test and validate all the adobe analytics parameters are recorded correctly.

Then, there are times when we have to record and store all the values of every prop, evar and event that fires on page load. ?

Problem with adobe analytics testing

Needless to say, manually loading all these web-pages, using a browser plugin or making use of network call to inspect the adobe analytics calls is hectic and tedious. Especially while handling clients with large website or web analysts dealing with multiple adobe analytics clients at a time. ?

Oh I wish if there was some way of adobe analytics automation solution for these type of QA tasks – except from buying license of expensive tag audit tools like observepoint.  But wait, Why only wish? Why not build one for myself? 😛  ?

Yes, I should feed all the URLs to a tool, run a program and go grab a cup of coffee. By the time I return, all the URLs should be QAed and results should be put to a document or excel sheet.

How convenient isn’t it? !! ?

Solution Approach

Firstly, there is a buffet of robust programming languages. There are so many open-source tools to pick from. I first wanted to build a comprehensive, stand alone app on JAVA platform and use Selenium web services for network calls. However, having a demanding desk job, frequent long travels and an adorable 2-year old daughter at home, I neither have bandwidth nor the heart for such hard work. So, dropped this idea. ? ?

Likewise, there is ‘python‘ – one of the most discussed coding language with it’s ‘snake’ ? like flexibility and agility. However, I’ll have to learn a completely new language and syntax for that. ?  

Finally, I finally chose the awesome nodeJS to make my wish come true. It is simple, free, has great community and based on my area of expertise – JavaScript. So, if you want this nice little tool to work for you, you better know the basics of JavaScript and concept of nodeJS.  ?

The Solution – Automated adobe analytics testing!

Let me show you how I imagined the program to work: ?

Before you proceed, note that this solution is not limited to adobe analytics testing automation but same logic code can be used to automate testing of any other analytics tools too.

Solution : Thought process to build adobe analytics audit tool

What all do you need for this adobe analytics testing tool? 

• NodeJS installed in your computer. That’s it.

You can pretty much run this program just after that. However, if you wish to make any changes, you would need a good code editor. – I prefer Visual Studio Code – which is free and my favorite. ?

Tool Design

It consist of only 5 small files. ?

• One package.json file • One Server.js file • Two HTML (PUG) files. ○ One to get URL path ○ One to display adobe analytics test result • One CSS file

Program Structure

NodeJS – npm- packages & libraries required: [Refer ‘dependencies in package.json code.

• http – since it is web related program
• tableify – to parse end result to a table
• fs, util, body–parser,  – file system editor – to read HAR file
• express – to run local server
• puppeteer – to load the URL in a headless browser.
• PUG -which is based on Jade framework – for dynamic web pages.

I’ve also tried to use few libraries to increase usability like to check for a valid URL, if URL exists etc. but they aren’t mandatory if you know what your’e doing. 

Actual code

Program name:- getAdobeCall

File:- package.json

Note that I’ve used nodemon – for easy server operations. (Start and Stop)

{
  "name": "getadobecall",
  "version": "1.0.0",
  "description": "Gets adobe analytics call details on any website",
  "main": "server.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1",
    "start": "nodemon server.js"
  },
  "keywords": [
    "adobe",
    "adobe analytics",
    "analytics",
    "network call",
    "shivanandana",
    "hegde",
    "eVar",
    "prop"
  ],
  "author": "Shivanandana Hegde",
  "license": "ISC",
  "dependencies": {
    "body-parser": "^1.19.0",
    "chrome-har": "^0.11.0",
    "express": "^4.17.1",
    "express-validator": "^6.1.1",
    "fs": "0.0.1-security",
    "http": "0.0.0",
    "nodemon": "^1.19.1",
    "path": "^0.12.7",
    "puppeteer": "^1.19.0",
    "tableify": "^1.1.0",
    "url-exists": "^1.0.3",
    "util": "^0.12.1"
  }
}

getURLForm.pug

doctype html
html
  head
  link(rel='stylesheet', href='/css/styles.css')
  body
  div(align='center')
      h1 Get Adobe Analytics Variables 
      p ...
      p <b>#{caution}</b>
  form(action='/thank' method='post' name='form1' align='center')
      label(name='web-page') Enter URL Below:
      <br/><br/>
      input(type= 'text' name='url')
      <br/><br/>
       button.btn.btn-primary(type='submit') Fetch
  div(align='right')
      p Work of:- Shivanandana Hegde

postResult.pug

doctype html
html
  head
  link(rel='stylesheet', href='/css/styles.css') 
  body
  div(align='center')
        h1 URL requested 
        h3 #{urlVar}
        h1 Adobe analytics variables names and values
  div(align='right')
        p Work of:- Shivanandana Hegde      
  div(align='center')      
        p !{result}

(Main file) server.js

var http = require("http");
var htm;
//from webpage
var tableify = require('  ');
var fs = require('fs');// file system
var obj = '';
var x;
var finalOutput = [];
//validators
const events = [];
//From getNetCal
const observe = [
  'Page.loadEventFired',
  'Page.domContentEventFired',
  'Page.frameStartedLoading',
  'Page.frameAttached',
  'Network.requestWillBeSent',
  'Network.requestServedFromCache',
  'Network.dataReceived',
  'Network.responseReceived',
  'Network.resourceChangedPriority',
  'Network.loadingFinished',
  'Network.loadingFailed',
];
const { promisify } = require('util');
var pageURL1 = '';
const puppeteer = require('puppeteer');
const { harFromMessages } = require('chrome-har');
// list of events for converting to HAR
//For Running Server, Details.
var express = require('express');
var app = express();
var bodyParser = require('body-parser');
var urlencodedParser = bodyParser.urlencoded({ extended: true });
var path = require('path');
//for pug
app.set('views', path.join(__dirname, 'views'));
app.set('view engine', 'pug');
//Give permission to CSS files
app.use(express.static(path.join(__dirname, 'public')));
//Validator library
const { check } = require('express-validator')
//Check if URL exists
const urlExists = require("url-exists");
var validPage;
var server = app.listen(8080, function () {
  var host = server.address().address
  var port = server.address().port
  console.log("Server running & listening at %s:%s Port", host, port)
});


//First Form - GET
app.get('/getAdobeCall', function (req, res) {
  //  res.sendFile(path.join(__dirname +'/getURLForm.html'));
  res.render('getURLForm', { caution: 'URL should start with (http/https)' });
});


app.post('/thank', urlencodedParser, async function (req, res) {
  //res.render('/thank');
  pageURL1 = req.body.url;
  console.log(pageURL1);
  if (!pageURL1.includes('http://') && !pageURL1.includes('https://')) {
    pageURL1 = 'http://' + pageURL1;
  }

  urlExists(pageURL1, function (err, exists) {
    if (exists) {
      //res.end('Good URL');
      validPage = 1;
    }
    else {
      var badurl = '';
      badurl += "<body>";
      badurl += "<br/><br/><br/><br/>"
      badurl += "<div align='center'>"
      badurl += "<h2> BAD URL </h2>"
      badurl += "</div>"
      badurl += "</body>";
      //res.send(badurl);
      console.log("Bad URL is:-" + pageURL1);
      res.end(badurl);
    }

  });
  // event types to observe
  const browser = await puppeteer.launch();
  const page = await browser.newPage();
  // register events listeners
  const client = await page.target().createCDPSession();
  await client.send('Page.enable');
  await client.send('Network.enable');
  observe.forEach(method => {
    client.on(method, params => {
      events.push({ method, params });
    });
  });

  // perform tests
  await page.goto(pageURL1);
  await browser.close();
  // convert events to HAR file
  const har = harFromMessages(events);
  //fs.write('cdw.har', JSON.stringify(har, undefined, 4), 'w');
  await promisify(fs.writeFile)('networkCalls.har', JSON.stringify(har, undefined, 4));
  console.log('File created successfully')
  obj = await JSON.parse(fs.readFileSync('networkCalls.har', 'utf8'));
  //Loop to get the adobe network call.
  for (i in obj.log.entries) {
    x += obj.log.entries[i];
    if (obj.log.entries[i].request.url.includes('b/ss')) {
      finalOutput = obj.log.entries[i].request.queryString; // Caught the adobe output here
    }
  }
  htm = await tableify(finalOutput);

  //POST THE PROCESSD RESULT
  res.render('postResult', {
    urlVar: req.body.url,
    result: htm
  })
  //res.send(reply);
  console.log(pageURL1);
  //console.log(req.body.url);
});

✔️ That is All... See the below video illustration to know how this adobe analytics testing tool works.

Scope of improvements

Hey, this is just the basic foundation of my solution. Of-course it has limited functionality as of now. Like:

• The program can read only 1X1 adobe analytics network calls but may not work for 2X2 network calls.
• It works only for first Adobe call. (If there are multiple adobe calls on page load)
• Speaking of page load, this is only for ‘page load’ calls. So, for onClick events, this needs further optimization.
• Mostly stable (like 85% of the time) but sometime, I’ll have to restart the node/npm to make it work.
• Add URL validation – to check if URL exists or not.

There are many other enhancement options to consider like :

Read all URLs from a CSV/Excel file.

for (every row in the excel sheet) 
{
/* Do this operation and 
 store the adobe output in a file. */
}

Check for data in the file
Alert me if there is any error.

If you made use of this, let me know in comments/email.

Feel free to copy this code and improve it based on your requirement. ?

Verdict: Life is good, open source is awesome, this world is wonderful !! 
Thanks to Ryan Dahl and other people who created nodeJS.

Thanks to all the people who’ve created these wonderful nodeJS packages and special thanks to those self-less contributors of stackoverflow!! You guys always reinforce my faith in humanity! 🙂 ?

Cheers… ? 🙂

My recent posts on Digital marketing and Analytics:

• Web Analytics – Insights & Challenges
• How to install Analytics on your Website: DIY (Google Analytics & GTM) Web Analytics – Insights & Challenges
• What is Analytics? Understand Digital Analytics in simplest way Web Analytics – Insights & Challenges