A new type of agency adserver I

I've been thinking about the agency side adserver. Alot.

It seems to me that there are a plethora of publisher side adservers out there and it's easy to see why. There are more publishers than there are agencies. It's a bigger market and one which is more lucrative. This has led to some great adservers in the publisher space and even an open source one - openX which is used by small publishers the world over. But how about the agency side?

By default people use DFA. It's a decent system and they've got enough in there for everyone's need. They also allow you to do the other things like templates and rich media without too much extra cost. Some people find the workflow good as well (these people no doubt have just been using it too long - start flaming now). There are other solutions out there but they are too expensive, agencies don't operate with the kind of profit margins that publishers do. Most other agency adservers are more what I would consider evolutionary than revolutionary, they put in a couple of extra features or smooth some things out but they don't really do anything extremely different.

This is where I start thinking. If you could build your own adserver from scratch, what would you do different. There are obviously some steps that need to be in there such as uploading a schedule, uploading creatives and assigning creatives to schedule and creating tags based on the schedule, but how should these be done? Also what features do the current adservers have? How can we make serving cheaper? How can we make sure that the solution is future-proof?

I've been having a look and I think the technology is finally here to solve a lot of these problems. Node.js is blazingly fast (asynchronous IO), and allows a common language between front end and back end - javascript. CDN networks display static files at a fraction of the cost. Cloud computing and platform as a service (PaaS) allows instantly scaling architecture. NoSQL databases allow fast information retrieval, sideways scalability and the flexibility to add new functionality easily. New javascript engines and HTML5 will allow more complexity on the client side by being faster and allowing local storage and non breaking I/O with the server (ever wanted to add in fields and take them out in real time to see what a report would look like?)

So what is the plan? I will attempt to build an adserver. It's quite an ambitious plan I admit. To pull this off I'll need to become an expert in client side javascript (I'm going to go with JQuery & JQuery UI), HTML5, Node.js, expressJS (application MVC framework for Node.js), MongoDB (NoSQL DB), AWS (amazon web services to be used as a platform, may change to Joyent or Cloud Foundry), CDN options and whole range of other concepts in between while balancing a job and social life. The main thing to get first is enough knowledge to start coding and a plan as to how things will work. I have been studying for the past couple of weeks and believe I have enough knowledge to start some simple things and luckily the framework will be flexible so I can change things as I see better ways but in the next installment I will unfold the full extent of my plans

run function after DOM loaded

Here is a bit of code that will call the functions runAds() after the DOM has loaded. This is getting a bit advanced, but fun if you want to fiddle. Just be aware that IE will let you document.write stuff to the page at this point, but Firefox will start a new page if you want to document.write anything in. So if you are using this, you'll need to do it by adding iframes in to the source and load the ads in through there:





var alreadyrunflag=0 //flag to indicate whether target function has already been run

if (document.addEventListener)
document.addEventListener("DOMContentLoaded", function(){alreadyrunflag=1; runAds()}, false)
else if (document.all && !window.opera){
document.write('\<script type="text/javascript" id="contentloadtag" defer="defer" src="javascript:void(0)"\>\<\/script\>')
var contentloadtag=document.getElementById("contentloadtag")
contentloadtag.onreadystatechange=function(){
if (this.readyState=="complete"){
alreadyrunflag=1
runAds()
}
}
}

try{
var backupOnload = window.onload;
window.onload=function(){
if((typeof(backupOnload) != "undefined") &&(backupOnload != null)){backupOnload();}
setTimeout("if (!alreadyrunflag) runAds()", 0)
}
}catch(e){}


another one you might like to play with is the defer attribute for calling an external script:

document.write('\<SCR' + 'IPT SRC="' + allAds + '?" type="text/JavaScript" onLoad="runAds();" defer="defer" language="JavaScript"\>');
document.write('\</SCR' + 'IPT\>');

have fun - but remember that we're starting to push the boundaries of what we can do with our adcode here, so test in all browsers and different versions and then make sure you test with all sorts of ads including rich media

get query string value

Most people will be trying to get values from a query string by using the .split() javascript function. Basically the way to do this is to grab your string after the first question mark

string = string.substring(string.indexOf("?")+1);

then splitting on ampersand

values = string.split("&");

then going through each value checking for the left hand side and returning the right if it matches

for(i=0;i\<values.length;i++)
if((values[i].split("="))[0] == name)
try{return (values[i].split("="))[1]}catch(e){return ""};
return "";

I have also done it in the past just using string indexOfs:

//zMSmyStr is the query string, zMStagline is the variable name being searched for
function zMSgetValue(zMSmyStr,zMStagline)
{
zMSmyStr = zMSmyStr+"=";
var zMSind = zMStagline.indexOf(zMSmyStr)+zMSmyStr.length;
if(zMSind==(zMSmyStr.length-1)) //test for myStr not found
return "";
zMStagline += "&";
return zMStagline.substring(zMSind,zMStagline.indexOf("&",zMSind));
}

but I decided to save space to redo these as a regex and save time and space. So here it is:

function fd_fn_Query(myQS)
{
try{
return myURL.match(new RegExp("\[?&]"+myQS+"=([^&]*)"))[1];
}catch(e){return "";}
}

much smaller and easier to use, feel free to take this and modify to suit your needs.

Prototype.js and for(..in..) loops

I had a problem a while ago where I was using a for(..in..) loop to go through a hash map (like an array but has strings for the index instead of numbers). Now you NEED a for(..in..) loop for these, because you don't always know what the indexes are going to be called, and you can't call them sequentially like a normal array. Problem was that I was getting back what looked like javascript functions instead of the values I expected, so I did what any programmer would do after finding no easy explanation - I just checked for that function keyword, the length of the string etc, and skipped over the values that were not in a format that would work. Since then I have encountered the problem again, but luckily it was on a site where the developer knew his stuff:

http://www.prototypejs.org/api/array

it seems that the prototype.js javascript library absolutely hates the for(..in..) loop. So for these site you have to return the array as an enumerable type and then use the each function. If you are writing code to go across multiple sites, you may want to use a try catch block to check the existence of each going forward.

You have now been warned

**Edit**

I stumbled upon this easy fix:

var object = new Object();
for(var key in object){
if(!(key in Object.prototype)){
// do some stuff SAFELY!
}
}

How To: Build a click macro insertion tool (part 2)

Please refer back to part 1 to go through setting up the bare bones.

In this part we will setup the script that can chang.values entered in and touch upon Regular Expressions or Regex. Regex is a fairly advanced topic and many experienced programmers struggle with it, while at first it may look like line noise it is in fac.very logical an.very powerful.

first off, let's make the button call a function. In the file from the previous tutorial I put in an onClick attribute on the button? after that put in a call to a function, let's say changeMe();

now let's make the function between the <script> blocks:

function changeMe(){alert( document.getElementById("myTag").value);}

This will make an alert box come up with the string inside the box. We can also set th.value:

function changeMe(){ document.getElementById("myTag").value = "HELLO WORLD";}

This will change th.value of the box to HELLO WORLD.

now for some regex fun. Let's do a useful example, Say we want this tool to secure any insecure creatives. What we need to do is change any http: to https: for any items brought in. To do this we first have to look for http:, the forward slashes you will see are what defines this as a regex

/http:/ - this is good, but what happens if we don't want to change the http in the links? we need only change the src="http:

/src="http:/ - This may be a little too strict, as there are many ways of putting this in. For instance the user may use double quotes or single quotes

/src=["']http:/ - the square brackets represent a character class - it means match any chacter inside. We will go deeper in to these in other how tos, but how about spaces around the equals sign?

/src\S*=\S*["']http:/ - the \S represents a whitespace and * represent 0 or more of the previous character. now I will make a couple of changes we will need later

/(src\S*=\S*['"]http):/gi - The brackets store the result of the match that we will need when replacing th.value, as we need to know whether to replace with a double or single quote, to make this pull everything up to where the s is though I have placed the brackets around most of the string. also I put g and i afterwards which are flags. The g means global, so for all instances - not just the first and i means ignore case.

now lets put this as the function:

function changeMe(){ document.getElementById("myTag").value = document.getElementById("myTag").value.replace(/(src\S*=\S*['"]http):/gi,"$1s:");}

the replace function takes the regex as the first argument and the string to replace it with as the second argument. The $1 part in the string is replaced byt what is matched in the first set of brackets.

now we are replacing the src=, but flash files also have data= o.value=. To fix this we could chain our regex replaces together like:

function changeMe(){ document.getElementById("myTag").value = document.getElementById("myTag").value.replace(/(src\S*=\S*['"]http):/gi,"$1s:").replace(/(data\S*=\S*['"]http):/gi,"$1s:").replace(/(value\S*=\S*['"]http):/gi,"$1s:");}

or we could use the regex OR, which is a pipe | and group them together. So it becomes (src|value|data). The function then becomes:

function changeMe(){ document.getElementById("myTag").value = document.getElementById("myTag").value.replace(/(src|value|data\S*=\S*['"]http):/gi,"$1s:");}

and that's it! what I would recommend is creating a new button and assigning this - and then we can play in different functions on different buttons

end result code:

<html>
<head>
<script>
function changeMe(){ document.getElementById("myTag").value = document.getElementById("myTag").value.replace(/((src|value|data)\S*=\S*['"]http):/gi,"$1s:");}
</script>
</head>
<body>
<form name="theForm">
<textarea rows=20 cols=20 name="myTag" id="myTag"></textarea><br>
<input type="button.value="go" onClick="changeMe();">
</form>
</body>
</html>

How To: improve adcode

now most major publishers have a similar approach to adcode as what I will outline. First I will go over the basics of a piece of adcode and what is needed. Then I will address the different types and finally list what can be improved in traditional adcode and how to do it.

Adcode Elements:

what we should look at first is the URL for the impression call of any piece of adcode. The elements that will make this up are:
1. Domain - this will call the server where the adserver is
2. Call - there will usually be a part that defines what kind of adcall it is, impression, click, and also what it should return
3. Site and location - variables that match to the targeting within your adserver
4. targeting - these are varibales that may or may not already be stored in your adserver. These fine tune the targeting
5. size/adtype - to define how big the ad is and what type of ad
6. other variables - different adserver will pas through different variables dependant on what is needed to use it's capabilities
7. cache buster - usually ord= or random=

Splitting the elements:

if you look at these elements you can see that some are used for all tags on a site (site, domain etc), some are used for the page (location, some targeting) and some are used for only that particular adunit (size, ord). So we code that is printed out several times for a site, or a page that only needs to be put in once. Why don't we group these variables and use javascript to write them out when needed

Advantages:

what we are going to do is make functions that will put in the code as needed when needed. What this allows is a change in one place to change all code for the page/site as opposed to changing all parts of the code. This makes code cleaner and easier to adjust. It also makes code easier to implement. If there is code already for the page, all we need to do is put in the code for an island to have an island appear - in fact one piece of island code could be put on any page

and it should pick up the site and page variables automatically. Also the code is non-adserver specific, so if you ever change adservers, you only need to change the file that hold the main methods - no changes to the site necessary

Method:

what we should do is create a separate .js file that holds methods to create the random cache buster and to automatically put in site variables. We then need a function that will set page variable dependant on what is passed in from the page. The final function we need will use the variables already set and variables passed in from the actual ad to write out the code.

an example of page code:

script src=" adcode.js ">/script>
script>
site = "site";
createSiteVars(site);
location = "location";
createPageVars(location);
/script>

then to call an ad:

script>
callAd(300,250,"javascript");
/script>

for extra bonus points, host the external script file with all functions somewhere you can access it so that you can hack away and make changes to site coding in a second if need be. Some examples of different implementations are whitepages.com.au and thevine.com.au

How To: Build a click macro insertion tool (part 1)

Most publishers will already have one of these, it's a tool where you can place agency tags, click a button and magically have your click macro inserted. What I will do here is show you how you can build one using just HTML and javascript so they can be hosted on an intranet site, or users can keep them on their desktop and use one. This first part will go through how to build the page and touch lightly upon the next steps.

1) tools you will need:

for this little tool all we need is:
- a text editor (my favourite is textpad)
- a browser to test (I like to test mine in IE and Firefox, though usually for a tool I will also test in safari, chrome and opera as well)
- examples of agency tags (you should be able to get these from past campaigns)

2) The foundation

a few places will have a tool for each tag, what we'll try and do is create a single page with a simple interface. All the user should see is a box to place the tag and a big friendly button that will run our scripts. Later I will go through how to extend the tool and the addition of other buttons that will do things like strip iframes, and even wrap the tag in popunder code.

3) first step

open up your text editor and put in the framework for an html page: <html><head><script></script></head><body></body></html>. Best to put them on new lines.

4) the interface

All we need is a textarea and a button in a form. So between your body tags put in:

<form name="theForm">
<textarea rows=20 cols=20 name="myTag" id="myTag"></textarea><br>
<input type="button" value="go" onClick="">
</form>

you now have a basis you can work from. You can use the onClick for the button to call a javascript function and you can view and alter the textarea value using document.getElementById( "myTag").value - or document.theForm.myTag.value .

And that's it - now you have a nice clean interface which we can use for the next How To!

How To: improve a flash template file

For those of you using Atlas, or other systems that allow you change the template file that a flash ad uses - consider the below points and whether they are in your current template, and whether you should change to make a more robust functioning template.

1st check: Are you using flashvars?

long ago flash templates would pass variables to the flash file in the query string. The problem with this method is that the URL may become too long and the browser may drop some information. Try using flashvars to pass things like your clickTag < http://kb.adobe.com/selfservice/viewContent.do?externalId=tn_16417 >

2nd check: Are you escaping ampersands in the click URL?

make sure you are escaping the ampersand before putting it in to the flashvar string. New ampersands tell the script that it is a new variable being passed. You may like to experiment with only escaping the query string, or just replacing all ampersands with %26 using a .replace() function.

3rd check: Is wmode set?

wmode should always be set - and usually to opaque. This will prevent problems with dhtml menus and other flash files when they are over the top of one another

4th check: how does the backup work?

Have a look where the script and no script blocks are. I have seen a couple of sniffers where if flash wasn't installed and javascript was, it would show the backup twice.

5th check: Secure calls?

If you have secure sites - consider checking the document.location for https, and if present, update all other calls on the page to use https in the code.

6th check: Eolas issue

This should no longer be a consideration for updated browsers, do you need to keep that external javascript call to allow for it?

***UPDATE***

There is another enhancement that I neglected to mention as it is for the preview screen rather than the live template. If you use the same template for previewing a flash ad as the live one then you can put a check in for the document.location before running the below.

Make your background red - this is great for checking that a flash ad has put in it's background correctly and that wmode is setting it to opaque. you can do this with the simple document.bgColor="red";

Tools of the Trade

There are many handy bits of software and plugins around. Here is a list of things that I use to maximise efficiency and help debug. What are some of your favourite?

Firefox
- Firebug (used for debugging DOM and javascript)
- Greasemonkey (can alter how javascript acts)
- IMacros (if you're using a web frontend you can make macros to enter vast sets of creative for you from a csv file!!!)
- HTTP watch (http headers)
- JSView (quick view of javascript files)
- Web Developer (handy toolbar)
- firecookie (manage cookies)

IE
- IE7Pro (allow you to mod it's behaviour)
- script debugger (check out javascript files and what they call)
- Fiddler (HTTP headers)
- multiple IE (run ie6 alongside ie7 - doesn't work so well for rich media)
- IE Developer toolbar (DOM inspector)
- Bookmarklets ( https://www.squarefree.com/bookmarklets/webdevel.html - generated source especially)

Flash
- Action Script Viewer (for clicktag check)
- URL Action Editor (for small fixes)
- Sothink SWF Decompiler (get to the nuts and bolts)

Wireshark (network analyser)
Chrome (browser)
Safari (browser)
Opera (browser)
Textpad (regex and replace all is your friend)
Excel (spreadsheet - VLookup is your friend)
GIMP (image manipulation)
LogMeIn (log in to your computer remotely)

watch that variable scope with AS3!

if you're used to actionscript2 in flash and making banners, you're used to:

on (release){
getURL(clickTag,"_blank");
}

But AS3 ushers in a new dawn where the language is more object orientated and comes away from those handy little tricks. First thing you have to do is bring in the click tag variable from root, something like:

clickTAG = root.loaderInfo.parameters.clickTag;

Then you need to catch the action:

public function HandleClick(param1:MouseEvent) : void
{
trace("click");
ExternalInterface.call("window.open", clickTAG, "arplayer", "height=372,width=820");
return;
}

now the above is actually an example of how to open a window of a custom size.

Now I was dealing with a case recently where the clickTAG = was brought in on the root frame - so in otherwords it was within a function called frame1(). Now this means that clickTAG is only defined for this function and not for the event handler function. Instead the event handler has to be changed to:

public function HandleClick(param1:MouseEvent) : void
{
trace("click");
clickTAG = root.loaderInfo.parameters.clickTag;
ExternalInterface.call("window.open", clickTAG, "arplayer", "height=372,width=820");
return;
}// end function

Now I'm not a flash designer/programmer - but I am a coder and so this was blatantly obvious to me using sothink decompiler, but not so much for the creative agency.

Randomize an array

First of all the why:

you want to serve back several text ads in a random order through one adserver call. You would do this to save on calls - for instance a block of 8 text ads which are changed on a weekly basis could instead be changed in to one block that gets updated, randomized with javascript, and printed on the screen. This saves 7 extra calls a pop - which means that you save on adserving fees.

The current methods:

There are two methods suggested. The first is faulty, and if you're using it - stop now:

1)

Array.sort(function(){return (Math.round(Math.random())-0.5);});

now the idea behind this is that you use the sort method for an array, but randomly decide which of the two elements is higher. This does not work because the implementation of the sort method can be different between javascript engine implementations. In fact, give it a go. Make a new array of 10 digits and print out the results a hundred times. The answers are definitely not random.

2)

The second method is the Fisher-Yates which has been used to shuffle cards completely randomly in the past. Although a good algorithm, it requires a second array and too many swapping routines:

function fisherYates ( myArray ) {

  var i = myArray.length;

  if ( i == 0 ) return false;

  while ( --i ) {

     var j = Math.floor( Math.random() * ( i + 1 ) );

     var tempi = myArray[i];

     var tempj = myArray[j];

     myArray[i] = tempj;

     myArray[j] = tempi;

   }

}

So how should it be done?

THE ANSWER:

for(i=0;i<array.length;i++)

array.splice(Math.round(Math.random()*i),0,array.pop());

using the power of javascript array functions we can make this work with most of the calculations being done by the libraries - which are always nicely optimised. 

So how does it work?

we only use the one array, but we divide it in to two separate logical parts. The randomized array, and the elements yet to be put in to the array.

What we do is grab the last element of the right side of the array (elements to be randomized) then place it randomly in the left side of the array (elements already randomized). This is much quicker with smaller data-sets than fisher-yates, but I will give no guarantees for data sets over 100.

enjoy!

The free ad serving solution

So what do you get when you have a small blog and want to make a little money out of it?

a) Google AdManager

b) Adwords Account

c) Google Analytics

Google have come a long way from their original stance of "Don't  be evil" - and let's face it. More of us have begun to hate them (even though we just can't ween ourselves off their search engine)

So what are the alternatives?

Well there are a few nice adservers going around

1) Open X - http://www.openx.org/

This is good for those casual coders out there that want to do it all themselves and like to own their own data on their own servers. Anyone with a webhost has probably seen OpenX as an option - although it is probably listed under one of it's former names in plugin libraries - OpenAds or PHPAdsNew.

2) Rubicon - http://rubiconproject.com/

A new adserver to the fray. It offers a nice sleek front end which, really, we all love to see. It is still a hosted solution, but would be good for those who are used to a google admanager

3) APT! from Yahoo - http://apt.yahoo.com/

What can I say? It's Yahoo's answer back to google admanager. It is new and hasn't really caused the stir it should. I am looking forward to seeing what the take up will be.

Adwords Account alternatives:

Instead of going over this I did a search on google (ironic right?). Just go here http://www.rosswalker.co.uk/adwords_top10/

Google Analytics alternatives:

1) Piwik - http://piwik.org/

This is closely affiliated with OpenX, so if you're happy with OpenX - go Piwik!

2) Woopra - http://www.woopra.com/

This has another sleek interface, so if you liked the look of rubicon - you'll love this. But perhaps the best thing about Woopra is it's real time stats that you can watch change in front of your eyes. Great if you've got a popular blog and want to show off the digg effect to your mates.

3) Engine Ready - http://www.engineready.com/products/index.html

Possibly the most google analytics like of the bunch. If you want a smooth transition over - then give this one a go

Flash 10 fires popup blocker on flash click

This is a serious bug for everyone in the business to know. What exactly makes it occur is unknown, but you can spend hours looking for a solution that isn't there

Basically flash compiled in a version less than 8 may trigger ie6 and ie7s in built popup blocker when a user clicks on the flash button and the common getURL method is used.

for more information see:

http://bugs.adobe.com/jira/browse/FP-1065

Introduction

There are a few good blogs and resources for workers in the online advertising space, and below is just a few:

http://www.liesdamnedlies.com/

http://www.adoperationsonline.com/

http://www.imediaconnection.com/ 

http://www.adopstools.net/

http://interactiveadops.com

not to mention the groups on linkedIn of which the below is probably the best:

http://www.linkedin.com/groups?gid=154454

and if you leave a discussion there I will probably answer.

But this blog aims at the more technical in the field - those people who actually know what a DOM is, rather than throwing it in there as a buzzword, or at least for those that want to learn.

I will be providing useful bits of code, answering questions about tags, discussing conventions, handing out free advice and even dissecting the current state of the market.

So sit back, enjoy, and feel free to ask me any questions.

cheers,

Rhys