Last Updated on December 10, 2021 by Shivanandana Hegde
UPDATE: It’s been over 2 years since I published this post on adobe analytics testing automation. Several people have contacted me for a working version of this tool ever since. I’ve shared all the code required for this project here and recently I’ve published a live, working demo version of the tool.
Check the demo here – https://adobe-analytics.herokuapp.com/
Up-to 3 links can be validated at a time. The idea is to follow the same logic for 100s of URLs in a file!
History:
Adobe analytics testing is an integral part of most data analysts in web analytics arena.
Yes it is redundant, repetitive but at the same time, unavoidable and therefore, most important part of our lives! ?
So, to all my fellow WAPs – (Web Analytics Professionals) – I present to you an automated way to test and record adobe analytics beacons coming out of any web-page. ?
Background – Thinking about adobe analytics testing
There are times when we have to QA a bunch of URLs for our clients to test and validate all the adobe analytics parameters are recorded correctly.
Then, there are times when we have to record and store all the values of every prop, evar and event that fires on page load. ?
Problem with adobe analytics testing
Needless to say, manually loading all these web-pages, using a browser plugin or making use of network call to inspect the adobe analytics calls is hectic and tedious. Especially while handling clients with large website or web analysts dealing with multiple adobe analytics clients at a time. ?
Oh I wish if there was some way of adobe analytics automation solution for these type of QA tasks – except from buying license of expensive tag audit tools like observepoint. But wait, Why only wish? Why not build one for myself? 😛 ?
Yes, I should feed all the URLs to a tool, run a program and go grab a cup of coffee. By the time I return, all the URLs should be QAed and results should be put to a document or excel sheet.
How convenient isn’t it? !! ?
Solution Approach
Firstly, there is a buffet of robust programming languages. There are so many open-source tools to pick from. I first wanted to build a comprehensive, stand alone app on JAVA platform and use Selenium web services for network calls. However, having a demanding desk job, frequent long travels and an adorable 2-year old daughter at home, I neither have bandwidth nor the heart for such hard work. So, dropped this idea. ? ?
Likewise, there is ‘python‘ – one of the most discussed coding language with it’s ‘snake’ ? like flexibility and agility. However, I’ll have to learn a completely new language and syntax for that. ?
Finally, I finally chose the awesome nodeJS to make my wish come true. It is simple, free, has great community and based on my area of expertise – JavaScript. So, if you want this nice little tool to work for you, you better know the basics of JavaScript and concept of nodeJS. ?
The Solution – Automated adobe analytics testing!
Let me show you how I imagined the program to work: ?
Before you proceed, note that this solution is not limited to adobe analytics testing automation but same logic code can be used to automate testing of any other analytics tools too.
Solution : Thought process to build adobe analytics audit tool
What all do you need for this adobe analytics testing tool?
- NodeJS installed in your computer. That’s it.
You can pretty much run this program just after that. However, if you wish to make any changes, you would need a good code editor. – I prefer Visual Studio Code – which is free and my favorite. ?
Tool Design
It consist of only 5 small files. ?
• One package.json file • One Server.js file • Two HTML (PUG) files. ○ One to get URL path ○ One to display adobe analytics test result • One CSS file |
Program Structure
NodeJS – npm- packages & libraries required: [Refer ‘dependencies in package.json code.
- http – since it is web related program
- tableify – to parse end result to a table
- fs, util, body–parser, – file system editor – to read HAR file
- express – to run local server
- puppeteer – to load the URL in a headless browser.
- PUG -which is based on Jade framework – for dynamic web pages.
I’ve also tried to use few libraries to increase usability like to check for a valid URL, if URL exists etc. but they aren’t mandatory if you know what your’e doing.
Actual code
Program name:- getAdobeCall
File:- package.json
Note that I’ve used nodemon – for easy server operations. (Start and Stop)
{
"name": "getadobecall",
"version": "1.0.0",
"description": "Gets adobe analytics call details on any website",
"main": "server.js",
"scripts": {
"test": "echo \"Error: no test specified\" && exit 1",
"start": "nodemon server.js"
},
"keywords": [
"adobe",
"adobe analytics",
"analytics",
"network call",
"shivanandana",
"hegde",
"eVar",
"prop"
],
"author": "Shivanandana Hegde",
"license": "ISC",
"dependencies": {
"body-parser": "^1.19.0",
"chrome-har": "^0.11.0",
"express": "^4.17.1",
"express-validator": "^6.1.1",
"fs": "0.0.1-security",
"http": "0.0.0",
"nodemon": "^1.19.1",
"path": "^0.12.7",
"puppeteer": "^1.19.0",
"tableify": "^1.1.0",
"url-exists": "^1.0.3",
"util": "^0.12.1"
}
}
getURLForm.pug
doctype html
html
head
link(rel='stylesheet', href='/css/styles.css')
body
div(align='center')
h1 Get Adobe Analytics Variables
p ...
p <b>#{caution}</b>
form(action='/thank' method='post' name='form1' align='center')
label(name='web-page') Enter URL Below:
<br/><br/>
input(type= 'text' name='url')
<br/><br/>
button.btn.btn-primary(type='submit') Fetch
div(align='right')
p Work of:- Shivanandana Hegde
postResult.pug
doctype html
html
head
link(rel='stylesheet', href='/css/styles.css')
body
div(align='center')
h1 URL requested
h3 #{urlVar}
h1 Adobe analytics variables names and values
div(align='right')
p Work of:- Shivanandana Hegde
div(align='center')
p !{result}
(Main file) server.js
var http = require("http");
var htm;
//from webpage
var tableify = require(' ');
var fs = require('fs');// file system
var obj = '';
var x;
var finalOutput = [];
//validators
const events = [];
//From getNetCal
const observe = [
'Page.loadEventFired',
'Page.domContentEventFired',
'Page.frameStartedLoading',
'Page.frameAttached',
'Network.requestWillBeSent',
'Network.requestServedFromCache',
'Network.dataReceived',
'Network.responseReceived',
'Network.resourceChangedPriority',
'Network.loadingFinished',
'Network.loadingFailed',
];
const { promisify } = require('util');
var pageURL1 = '';
const puppeteer = require('puppeteer');
const { harFromMessages } = require('chrome-har');
// list of events for converting to HAR
//For Running Server, Details.
var express = require('express');
var app = express();
var bodyParser = require('body-parser');
var urlencodedParser = bodyParser.urlencoded({ extended: true });
var path = require('path');
//for pug
app.set('views', path.join(__dirname, 'views'));
app.set('view engine', 'pug');
//Give permission to CSS files
app.use(express.static(path.join(__dirname, 'public')));
//Validator library
const { check } = require('express-validator')
//Check if URL exists
const urlExists = require("url-exists");
var validPage;
var server = app.listen(8080, function () {
var host = server.address().address
var port = server.address().port
console.log("Server running & listening at %s:%s Port", host, port)
});
//First Form - GET
app.get('/getAdobeCall', function (req, res) {
// res.sendFile(path.join(__dirname +'/getURLForm.html'));
res.render('getURLForm', { caution: 'URL should start with (http/https)' });
});
app.post('/thank', urlencodedParser, async function (req, res) {
//res.render('/thank');
pageURL1 = req.body.url;
console.log(pageURL1);
if (!pageURL1.includes('http://') && !pageURL1.includes('https://')) {
pageURL1 = 'http://' + pageURL1;
}
urlExists(pageURL1, function (err, exists) {
if (exists) {
//res.end('Good URL');
validPage = 1;
}
else {
var badurl = '';
badurl += "<body>";
badurl += "<br/><br/><br/><br/>"
badurl += "<div align='center'>"
badurl += "<h2> BAD URL </h2>"
badurl += "</div>"
badurl += "</body>";
//res.send(badurl);
console.log("Bad URL is:-" + pageURL1);
res.end(badurl);
}
});
// event types to observe
const browser = await puppeteer.launch();
const page = await browser.newPage();
// register events listeners
const client = await page.target().createCDPSession();
await client.send('Page.enable');
await client.send('Network.enable');
observe.forEach(method => {
client.on(method, params => {
events.push({ method, params });
});
});
// perform tests
await page.goto(pageURL1);
await browser.close();
// convert events to HAR file
const har = harFromMessages(events);
//fs.write('cdw.har', JSON.stringify(har, undefined, 4), 'w');
await promisify(fs.writeFile)('networkCalls.har', JSON.stringify(har, undefined, 4));
console.log('File created successfully')
obj = await JSON.parse(fs.readFileSync('networkCalls.har', 'utf8'));
//Loop to get the adobe network call.
for (i in obj.log.entries) {
x += obj.log.entries[i];
if (obj.log.entries[i].request.url.includes('b/ss')) {
finalOutput = obj.log.entries[i].request.queryString; // Caught the adobe output here
}
}
htm = await tableify(finalOutput);
//POST THE PROCESSD RESULT
res.render('postResult', {
urlVar: req.body.url,
result: htm
})
//res.send(reply);
console.log(pageURL1);
//console.log(req.body.url);
});
✔️ That is All... See the below video illustration to know how this adobe analytics testing tool works.
Scope of improvements
Hey, this is just the basic foundation of my solution. Of-course it has limited functionality as of now. Like:
- The program can read only 1X1 adobe analytics network calls but may not work for 2X2 network calls.
- It works only for first Adobe call. (If there are multiple adobe calls on page load)
- Speaking of page load, this is only for ‘page load’ calls. So, for onClick events, this needs further optimization.
- Mostly stable (like 85% of the time) but sometime, I’ll have to restart the node/npm to make it work.
- Add URL validation – to check if URL exists or not.
There are many other enhancement options to consider like :
Read all URLs from a CSV/Excel file.
for (every row in the excel sheet)
{
/* Do this operation and
store the adobe output in a file. */
}
Check for data in the file
Alert me if there is any error.
- Create alert mechanism for wrong or missing values in eVar or props or events
- Create similar tool for other analytics tools like google analytics, coremetrics etc.?
If you made use of this, let me know in comments/email.
Feel free to copy this code and improve it based on your requirement. ?
Verdict: Life is good, open source is awesome, this world is wonderful !!
Thanks to Ryan Dahl and other people who created nodeJS.
Thanks to all the people who’ve created these wonderful nodeJS packages and special thanks to those self-less contributors of stackoverflow!! You guys always reinforce my faith in humanity! 🙂 ?
Cheers… ? 🙂
My recent posts on Digital marketing and Analytics:
4 replies on “Adobe analytics testing automation: Easy data validation”
Getting error at the time of starting server
Your Node installation has problems. You need to fix that.
Hi,
I am getting error while starting the server – npm start. Can you please help.
I have added all your files to my github https://github.com/arjunbm13/AdobeAnalyticsAutomation
Can you please help me? can we connect on whatsapp? – 9008567857
Hi Arjun,
If there is error starting the server, it’s mostly because of issues in modules/libraries installed in node. Refer this: https://stackoverflow.com/questions/61074827/i-cant-solve-error-while-running-npm-start
From the time I’ve put this, libraries would have changed and you may need to tweak the code..