Interview with The Judges at the EU Search Awards in Berlin
In the lead up to the EU Search Awards, I interview some of the judges who are attending on the night. I wanted to find out what it is like to be a judge, some of the criteria and how to submit and win an EU Search award.
1) What do the judges look for in an award winning campaign?
“I think one of the key things we look for is ROI. Since the very first year of the awards we have always worked to try and find the campaigns that delivered the best ROI for a company, including in some years not choosing multi-award-winning campaigns because a smaller, lesser known submitted campaign had a better ROI.
We also look for something that is different than the usual things and also innovation. We’ve had weather parting since the first year of the UK Search Awards so something using that is less likely to win an award than something that uses existing technology in a new way”.
“We look at each application in detail. We have to judge dozens of entries therefore it helps if it the application is easy to read such as in a powerpoint presentation format. We also like to see numbers to see if there is an ROI. A picture or screenshot that clearly shows the results helps to present the changes in a easy to view way.”
“The judges are thoroughly briefed and a common scoring system is applied to all the entries. We apply scores on the objectives, the strategy, the execution and the results to each of the entries we have been allotted and during this phase, we are not really looking for a winner as we are concentrating on working our way through all the entries we have been assigned. For this years judging I had around 65 case studies to “pre-score” in this way.
In the second phase of the process, the judges meet up to discuss the entries. At this stage we are on the look-out for a winner. We discuss the case studies which have received the highest average pre-scores from the 5-6 judges who have scored them. We will be talking in detail about 3-4 case studies for each category and input is available from judges who didn’t directly score this entry. The highest average pre-score doesn’t mean the entry will win the award. A very important criteria will be the results the end-client had with the campaign but a 500% ROI in itself is not enough to win an award as we will also be looking for fresh ideas, surprising approaches, risks taken and the context of the campaign.”
“Well defined budget, clear and well described objectives, key areas of the strategy & target audience, well described execution, and tangible results.
We, as judges, must be rigorous in the evaluation and we must consider any project on the same level.
To make it we need to understand exactly any single step and consider all the variables to weight properly different projects in different contexts and budgets.
It is all about giving the judges the possibility to judge. If I can’t understand or evaluate every single aspect of the campaign it is very difficult to score it high.”
We are focusing on detail-oriented and result-driven campaigns which have many creative ideas and are executed well. It is important to include every detail when submitting the case study.
2) How can you make sure your submission passes through the initial EU Search Awards Entry?
“We see all entries as judges as a group but not as an individual (if that makes sense).
Let me expand: We individually don’t judge all entries as with almost 400 entries for the EU Search Awards it is impractical. We are separated into groups. Those groups see *all* the entries in their categories. For example, if I am judging “Best SEO Campaign”, I will see all 80 entries (or however many there are) as well as all the entries for the other categories I am judging. However someone not judging that category will only really judge the shortlist. I therefore won’t see “Best PPC Campaign” in full but I will see all the entries that make it to the shortlist.
So the “initial filtering” is a method of creating a final shortlist by a group of judges who use the specific criteria and judging guidance in order to judge which entries should be shortlisted and therefore judged by all judges. Any judge could judge more if they wished but generally our groupings give us enough work for 2 or so days of judging and so we then focus on ensuring we read and evaluate all shortlisted entries carefully.”
“It is important to ?show that you’ve taken this serious and put some time in your application. “
“We have seen varying levels of entries this year and in some cases the applicants should have done better work on the application. It is really pretty straight forward, actually. When the form says “does not apply”, the judges won’t read that section and when it says “please fill in this field”, that criteria will count zero if the field is empty. Also, simple things like respecting the limit of 1000 words and providing all the information in just 1 file and 1 format rather than multiple documents will help the judges do their scoring correctly. There isn’t really room for any creativity in the form – it is not a CV contest where you can stand out. The judges will read every line of every single entry that is in conformity with the requirements. So if you apply, your application will be read. However, only the content of the application counts – there is no bonus points for a flashy lay-out and it could even be harmful.”
For the EU Search Awards, each judge is experienced in different areas and we judge the case studies submitted in these categories. For example, my background is SEO & inbound marketing, so I mark SEO and Content Marketing campaigns. We select 5 submissions using this score and we then we discuss the case studies. As the judges have experience from different backgrounds, then they can view the case studies from a different and unique point of view.
3) What is the criteria you have to fill to submit your case study for an award?
“It’s all there in the form and anyone can enter anything from small to large 🙂 Each different category requires different information be gathered and filled in. We will also use a multitude of tools to check the claims. However – and I cannot stress this enough – you ***MUST*** fill in all the required sections and you ***MUST*** match objectives to outcomes. Failing to do that can cost you an award.
I’ve got a bit of a thing about this. Every year I see people write 2,000 words (double the allowed amount) or not fill in things like budget or similar which makes it extremely difficult to judge, so we give guidance on how to fill out the form – but the information on the form is what to fill out. We even highlight that objectives should match results on the form and year after year they don’t.
The stated guidance says for example “Entries should clearly identify the objectives and budget of the campaign or project including a breakdown of implementation & staffing costs.” There is no need to get too granular such as x level at £x per hour but if you don’t tell us at least the basics of budget, you may not be shortlisted, let alone win because you will get marked down in that section. Our guidance on scoring is not strict but when an entry is missing detail, it is considered incomplete by me.
We look for a lot of things as judges including ROI for your client. If a company spends millions or hundreds of thousands and creates no tangible or measurable benefit (you can measure sentiment and coverage for the more PR-led campaigns) for their client, they will score lower than a company that maybe spends £8,000 but returns a measurable £50,000 for their client (as a fake example). While difficult to check conversions using 3rd party tools we can check other things. If you say you achieved a position 2 ranking for a short-tailed keyword we will go out and double check it.
We are all professionals as well and we talk to others and as such we are better informed about what has actually taken place on campaigns. We may be in a pitch, chat at the bar or be targeted as bloggers or professionals for campaigns that end up in the awards.
You can be a sole trader/freelancer and submit a case study, or you can be a multi-national and submit. It doesn’t matter how big you are but rather how well you performed for your client.”
All the entries are submitted by agencies who have some big clients in their portfolio. There are some exceptions. For example there is the Best Small Agency & Best In-house Team Awards which is a great opportunity for newcomers and small teams. Being nominated for Young Search Professional Award, the nominee does not need to work in a specific agency/company. They just need to be successful in their field.
4) Have you won an award or been part of a team that won? If so what were the key areas that stood out which helped you to win?
“I’ve been judging the UK Search Awards since they started and since the rule is you cannot judge if you are entered, I’ve never entered 🙂
However winners have stood out for me over the years because they bucked the trend like Yale locks and the boating company the first year who worked on small budgets and did excellent work, the Papa Johns Pizza example who really worked to go beyond the normal approach of PPC and do something spectacular. Those are the ones that stand out to me.”
“No. I have never submitted an entry because we have been part of the Judging team since 2014, and before we were not aware of the existence of the Search Awards.
In any case I suggest some key areas to pay close attention to when submitting an award entry:
– Follow the rules (again)
– Fill the default project description document, but also attach more files if you need to.
– Complete all the sections
– Don’t be afraid of describing the simple process of how to execute the campaign that gives clear results based on the objectives.
– In case of software/innovation don’t be afraid of putting forward a single great feature of the software and what the software can do. The judges cannot try the all, so be as descriptive as possible.”
My agency SEOzeo has not entered for any categories for 2015 but for the next years we will show our best work and submit a case study.
5) Many people may think their case study is not worthy of being submitted and may be fearful of it being rejected. What three reasons would you give someone to submit their case study for an award.
“No case study is ever rejected – it may not make the shortlist but it is *never* rejected.
I would suggest that judges are more impressed at small budget campaigns that demonstrate ingenuity than big budget campaigns that are just the same old same old over and over.
Judges are looking for ROI as well as how the campaign was executed. If you believe things were particularly innovative even if it was a small budget you should enter it.
No one knows who or why something didn’t get shortlisted but other judges who judged that category. We don’t discuss those as we have no time so if you don’t get shortlisted there is no shame, no recriminations and no penalty. No one but a small subset of judges will know an entry didn’t get shortlisted and only an individual judge will know how they scored you.
One extra but important point – we are not slaves to the points system which is why live judging takes so long. We look at the whole shortlist and we re-judge as a group and we award on what we are able to see in the entry and for ourselves. Just because pre-scoring places someone at top does not mean they will win and if someone does not make the shortlist as a judge we may as for it to be included. As judges we genuinely want the best campaign to win – not just the biggest budget or the fanciest execution.”
“The three reasons I would recommend someone to submit their case study for an award are:
1) It helps you to rethink what & how you’re working
2) You may use it as marketing (tool). It is certainly true if you are nominated and win.
3) It is fun”
“We receive a lot of case studies in the European Search Awards and we do not provide any individual feedback. If you don’t believe in your case study, then I wouldn’t advise you to submit it as there is a lot of good quality submissions in almost every category. However, from a more general standpoint, case studies are a great way of improving your business. They can provide a brilliant exercise to a company seeking to improve what it does and to federate the team around a common goal of improving quality. Case studies are also an excellent way of show-casing what your company can do and to use in promotional material.
My advice with regards to submitting case studies would be this: build case studies internally on your best campaigns – recall the objectives of campaigns, formulate the strategy, do excellent execution and if the results are excellent too… then you have an entry for the European Search Awards.”
“The problem is not if is worthy…is how you tell your story and how well you describe what you did.
I think also here we should divide based on the category of appliance, but my 3 reasons (considering case study) are:
– classic and simple projects that work are great.
– reaching and exceeding objectives means you make it happens, and projects are not intended to satisfy your ego & taste, but to make your client or projects standout (hopefully enrich).
– you can make truly great things even if in a low budget.
Tell your story, but tell it well with clear evidence of facts! Don’t try to persuade the judges. Do not try to persuade the judges with dressed up rubbish.”
First reason: It is the only game-changer award in our industry which has a great impact on a company’s brand and how others in the market see them.
Second reason: Submitting a case study to this award will help you to evaluate your results from industry experts.
Third reason: You can understand your company’s position in the competitive search world.
6) How can you reassure people that they should trust the judging. There often is a trusting debated with these awards.
“It is difficult to express in words the hours of work that goes in to judging for each of the judges. For my part, I do take the judging seriously and I do read every entry and I check things carefully. I write notes on my decisions and I do my best to do the best for those who enter. My scoring isn’t wildly divergent to others and while I am hard, I feel that I am fair.
I am possibly harder on people I know but when I see something I like it doesn’t matter if they are friend or stranger, I will talk with fellow judges in the session about my reasons for wanting a particular campaign to win. I will also listen to other judges and their reasons and weigh them up for myself and either agree or disagree, just as they do.
We work hard and it doesn’t matter if it is entry category 1 or 21 – we will pull out tools and check claims in entries before awarding an award.
We aren’t perfect and 12 of us in a room with 4 or 5 more by remote still doesn’t mean we get everything perfectly right but we try our best to check everything, double check and then weigh as a group.
“Judges cannot submit their own case studies so there can be no direct incentive to be part of it. No one person alone can decide on a winner. All the judges are from different countries and we all have to vote for the award. For my part, I have not discussed any of the case studies with the other judges or indeed with any participants in the awards so I find it a rather solid judging process.”
“The judging team improves year over year and they are from various cities all over the world. One judge cannot influence another judge’s opinion. It’s impossible.
For example this year is my first year as a judge and from my experience, the judging session was very intense.”
This year, we debated on who should be the winner for up to 60 minutes for some categories. The judges are not part of any campaign and therefore are neutral so can give an unbiased score on the campaigns.
7) How many times have you been a judge and what do you like the most about it.
“ I have judged every UK Search Award, all but the first EU Search Awards and I’m also involved in the US Search Awards as you might have seen. I don’t know how many times I have been a judge as I haven’t been keeping track 🙂
I guess I like being able to help be a part of something that recognises excellence and helps elevate the overall impression of our industry. I feel strongly about improving the image of the search industry – especially SEO.”
“This is my 4th time. I really enjoy it as it gives me loads of information and ideas. It is hard work but it is also fun.”
“Only one. What I liked the most was the intense discussion about the award entries. The other exciting thing was to read and review all the different case studies which involved small budgets but also those with larger budgets. Small budgets can also generate some excellent and clever ideas.”
To be honest, it was my first year. It is a truly amazing experience and I saw a lot of creative ideas and brilliant results. As an agency owner, understanding these experiences helped me to broaden my vision at SEOzeo. I will be judge for the US Search Awards and can’t wait to be in there in October!
The EU Search Awards takes place this Wednesday on April 22nd in Berlin, Germany. Tune in on the night for live coverage on State of Digital.