Search-Powered Web Apps

The most common question I get (well, after “Have you ever heard the MC Hammer song? Really? You have?”) is “Isnt Lijit just a search tool?”

I always reply the same, “No, we are a publisher advocate.”

Which is always greeted with one of three responses: 1) a look of bewilderment; 2) a look of amusement; or 3) a look of agreement.

Perhaps the strangest response I get is: “Why?”

For Lijit, the answer is simple. Because our entire existence is predicated on publishers. Not our business model mind you (although thats part of it) but our core value.

Our belief about publisher widgets is that there are two types: Widgets that exist to make publishers better publishers and seek to develop a true partnership and widgets that provide some value extension to the publisher.

The first type are publisher advocates, they have to improve the entire experience, both for the publisher and the reader.

The second type either is successful only on a high traffic publisher, or only for one consistuency, the publisher or the reader.

Our guiding principle when we add features to Lijit is simple: “Are We Being Publisher Advocates?”

In other words, does this feature make a publisher a better publisher by providing better service or increased engagement to their readers?

This also limits our focus to three areas:

1. Content Discovery / Reader Engagement

By indexing all of a publisher’s social content and trusted sources, Lijit allows content that may have been buried in a general search engine search to bubble to the top. Why? Well, we only index the things that are important to you; general search engines index everything. So, our base value proposition is that a publisher’s readers should find everything that a publisher trusts and wishes to expose.

In addition, when a reader comes from a general search engine, our “Re-Search” box proves additional implicit white-labeled results that tend to have a relatively high click through rate, effectively keeping a reader on the publisher’s site versus clicking the back button to the search engine.

Our stats also provide a variety of information for a publisher including results that returned zero results, providing a clue as to what readers are looking for from the publisher, potentially helping to inspire future posts or articles.

2. Optimization of Monetization

Publisher monetization is a noisy, competitive field, and currently we are loathe to produce a sub-standard ad experience for publishers. We cannot just be Yet Another Google Adsense Clone. We have to be better.

Lijit has to create an experience where publishers are optimizing revenue from an under-monetized section of a publication, namely the search results.

Everyone knows that search can be monetized effectively, but we believe because the results driven through Lijit are more contextual and relevant, the resulting revenue should be higher for the publisher. So, we are spending a lot of time developing an effective user interface and experience. Its hard and takes a long time, and we are close.

Besides search results, there are two immediate things that occur when using Lijit search. Your current social content gets better promotion increasing your overall pageviews, driving additional revenue now.

3. Cross Promotional Traffic

This is really effective if a publisher has multiple blogs or a blog network. With Lijit a publisher can use a high volume publication to help drive traffic horizontally to lower traffic blogs through cross-promotion in the search results. On average, our blog networks find that almost 30% of the results clicked in a search result are to another network blog, rather than the originating publication.

Each of these three functions: Content Discovery/Reader Engagement, Optimization of Monetization and Cross Promotional Traffic are all examples of how we feel that we are being publisher advocates, helping publishers be better publishers and helping them serve their readers.

After all, at Lijit we know one thing to be an absolute truth:

If publishers didnt provide social content or trusted sources, our results pages would be empty.

I’m proud to share another in a series of guest posts written by Lijit employees. This week we present an installment from Mike, who seemed overly excited about writing and sharing this post.

Hi, I’m Mike Merideth, the Director of IT here at Lijit, and I’m going to talk a little bit about the nuts and bolts of how we do what we do. Over the past year I’ve had the opportunity to design and implement the production network and server infrastructure on which Lijit runs. It’s been a great year of challenges and breakthroughs, but if there’s one key architectural concept that has gotten Lijit to where it is today, it is virtualization. We use Xen for our virtualization technology, which has the advantage of being free Software (both in the “free beer” sense and the “free speech” sense). CentOS 5.1 (a Linux distribution which is based on the market leader RedHat) includes this functionality out of the box, and has performed very well for us.

So why does Lijit use virtualization? There are a number of good reasons:

Flexibility: When you’re launching a new web product, it can be hard to predict what pieces of the application will need more resources than you originally gave them, and which will need less. We’re able to change the amount of memory, the number of CPUs and the amount of disk space a server has quickly, easily and remotely.

Availability: Because we use an iSCSI SAN for most all of our storage, we can move virtual servers between pieces of physical hardware. So if we lose one of our physical servers, we can quickly bring up the virtual servers it hosted somewhere else.

Resource utilization: CPUs today are incredibly fast and powerful; far more so than most applications need. Similarly, RAM has become cheap enough that a server with 16 or even 32 gigabytes of RAM is not particularly unusual, or particularly expensive. Running a simple web server on such a system would be a waste of CPU and memory, and therefore a waste of electricity. If you can run several virtual servers on such a system, however, you can get the maximum return on your investment by making sure you’re fully utilizing all of the CPUs and all of the RAM. Which is all tied to…

Cost savings: Colocation is expensive, and electricity certainly isn’t getting any cheaper. Using virtualization means we can get the absolute greatest value out of the rack space and electricity we’re paying for.

As of right now, we’re running about 200 virtual servers on about 25 physical servers. Just a few years ago we would have needed scores of physical servers consuming thousands and thousands of watts of power to do the work we’re able to do in this relatively modest environment. For a startup that would mean a higher burn rate with a shorter runway, and greater stock dilution for the founding stakeholders because of the amount of capital needed to get the work done. If you’re trying to get a tech startup off the ground, you owe it to yourself to see if you can leverage virtualization in your IT architecture. You’d really be crazy not too.

If you managed to read this post without your eyes glazing over, you may be interested in my new Linux infrastructure blog at http://linfrastructure.blogspot.com. I’m keeping notes on my experiences there, in the hopes that what I’ve learned over the past year can benefit others who find themselves in the same boat.

Photo credit: Leonard John Matthews

The following is a guest post by Dave Ferro. He’s a director on our product team, an avid rugby player and blogs about his fly-fishing adventures here.

When I interviewed for the job at Lijit, I was asked if I read any blogs, and if I had my own blog. There were a handful of blogs that I read at the time, a couple centered around fly fishing, a weekly column from a sports writer from the bay area, and a few others from the business community that I would check into periodically. As far as writing a blog was concerned, I didn’t figure I had the time, nor would I have assumed that anyone would ever read it. I was told that writing my own blog was going to be a condition of employment; the logic being that to work for a company that provides utility for blog publishers, I should first walk a mile in their shoes. That made sense. Upon receiving the job offer, I went about creating my own blog.

As with many undertakings where one walks in blind, I learned things that I never would have thought of before. The first thing that I learned is that people who you don’t even know will read your blog from places as far away as India, Hong Kong, and Santiago. Even domestic locations like San Antonio and Topeka, where I have never been, and will likely never go.

The second thing I learned about blogging is that there is a veritable plethora of gadgets, widgets, feeds, and other utilities that a publisher can add to his site…the best of which being the Lijit Search Wijit. There are also about a hundred ad networks who will give you a little snippet of code that will place an ad on your site.

Sifting through these ad networks is somewhat of a daunting task. Each of these networks has more or less the same message, “Put our line of code in your site, and we’ll pay you. Not only that, but our black box is the best black box, so we will pay you more than anyone else will.” It all sounded good, so I applied to a few of the bigger names and even a couple of really obscure ones that sound like they are run out of someone’s garden shed.

My site was accepted immediately by one of the household names, and the ones whose websites would have looked lame in 1997. Strangely, my site was rejected by a few upper and middle tier networks, via an impersonal email. Others said that my site was under consideration, and that they would get back to me when they decided that they wanted to work with my site. I love open ended exchanges like this.

I found that once you are accepted by an ad network, you have entered a one-sided partnership. Good luck getting a real answer to a real question. Good luck figuring out why you made 25 cents per click on Tuesday and then 15 cents per click on Wednesday. Some will allow you to block any domains of advertisers that you don’t want to appear on your site, where others leave you with the option of displaying the ads they send you.

At the end of the day, it is your blog, and you have the opportunity to change anything you want. It seems that there is a lot of trial and error you will go through in order to find out what works best for you.

I’ve been at Lijit a few months now, and everything is starting to make more sense. Every product and messaging conversation we have revolves around what we can do to make it easier for publishers, and what we can do to create a better experience for our publishers and their readers. As a product guy, I get to work on features and products that provide utility and real value for our publisher partners. Walking through the trials and tribulations that publishers everywhere will face has given me a better perspective on how to do that.

Recently, one of our publishers reached out to us to get our take on third party cookies. They were considering removing our widget due to the fact that we set a cookie when a browser views our widget. This brings up the debate over security and safety when it comes to third party cookies, and cookies in general. We responded to the post in the comments, but I wanted to elaborate on that a bit more.

Why Lijit uses cookies:

We use cookies to not only track whether someone is a Lijit user (allowing them to login etc), but also to allow us to match up a blog/widget visitor to any searches they may perform through our widget. This helps us to provide valuable metrics to our publishers in the form of stats, which in turn allows publishers to give their readers better content.
Generally,cookies allow us to gather better data about our users. Due to the ever rising pattern of “cookie blocking”, either by browsers, firewalls, security software,or explicitly by users, we have had to find other ways to continue gathering statistics. This means we have to use traffic pattern matching techniques, and logging analysis to get all of the data we need.

Historically, advertisers were the primary “pushers” of third-party cookies. This would allow them to track your viewing behaviors across any properties where their ads appeared. Some people disliked this since there was no real value to the web user, and the advertisers got free data. This was perceived as not only a security and privacy issue , but also pushed the perception of third party cookies into a grey area. These conditions, and the rise of spyware and malware pushed OS, and browser companies to institute tighter control.

In the current world of social media , distributed web services, and widespread widget adoption, the value to the user has changed. There are many services, Lijit included, that offer value to the user during their browsing session, versus just “tracking” them. The key, is that the web user is informed about what sites they visit, and the kind of content they allow in their browser. It is important for companies to disclose how they use the information they collect, and Lijit does this in our privacy policy.

Overall, the message should be about awareness and consumer education. The value of enabling third party cookies can actually be additive to the consumer vs. being a security or privacy concern. Modern browsers allow you to whitelist services you trust, and there are many services on the web (such as Lijit) that deserve that trust.

Over the weekend, one of our favorite Lijit users, Fred Wilson, posted his desire to do a blog spring cleaning.

Of course, the first thing I did was rush over to Fred’s blog and see if Lijit had made the cut. Luckily, it appears that we are one of the chosen few.

Thinking that Fred made a good point about keeping a blog clean, I headed over to my blog at Learn To Duck and started whacking widgets.

Last.fm? Gone. Instant Message widget? Outta there.

After the third whacked widget, I began to think about 1) why people should keep Lijit on their blogs (other than its the coolest widget ever!) and 2) what value do widgets bring to a blog.

Widgets, by design, should provide three things:

1. No apparent slow down of the blog page itself;
2. Real utility to the readers of the blog;
3. Real utility to the publishers themselves.

So, how does Lijit stack up?

Page Load Speed:

I asked one of our developers what specifically do we do that ensures that Lijit doesn’t slow down a blog’s load. Andy explained that we cache the javascript for a two hour period, making the javascript call (which is the main culprit in page load slow down - watch the lower left of your browser, you can watch how long it takes widgets to load) much shorter. In addition, we will always display the latest widget to a user, and then cache the javascript in the background for the next user.

Not being technical, I am guessing that means we load more quickly because the site doesn’t have to go get the widget every single time someone comes to the page.

Reader Utility:

Why is Lijit better than standard site search or Google custom search? Two specific reasons:

1. We search the publisher’s entire social graph and trusted sources.
2. While Google tries to index everything, Lijit just indexes everything that is important to you.

So the likelihood a relevant result is returned through Lijit search should be higher than if the same search was done on any standard search engine.

Also, unlike most widgets, Lijit is built to keep users on your site. Most widgets are connected to a destination site and their sole purpose is to drive traffic to the destination site. Lijit just wants to search your stuff.

Publisher Value:

For a widget to earn the right to live on my blog, it must make me a better blogger. The stats that Lijit provides is a direct example of this. One stat that I use a lot is “Results that returned zero results.” If people are searching for topics, but not finding them, then I need to write about them.

Also, because Lijit is trusted network-based search, relevant content from my blog is surfaced on other blogs that include me in their network. Basically, I get more traffic the more people trust my content, because it appears more often in the network’s search results.

Finally, because more of my social content is exposed through Lijit search, readers are more engaged and consume more content across my site and my social graph.

I certainly can understand Fred’s desire to clean up his blog, and have echoed his efforts on my own blog. Frankly, I wish more people would periodically do a widget clean up and when the time comes to determine which widgets stay and which are round filed, ask yourself three questions:

1. Does this widget slow down my blog page load?
2. Does this widget provide real value to my readers?
3. Does the widget provide real value to me as a publisher?

Hopefully, the answer in Lijit’s case is a clear yes to all three.

The Lijit team spent five days in Austin recently at the South by Southwest Interactive Conference. Most people are familiar with the music and film portions of SXSW, but if you’re a tech geek, then the Interactive portion of the conference is like Disneyland. But better because there’s free booze everywhere. Between Todd, Micah and myself, we were able to cover a lot of ground in Austin. After some reflection, I present my favorite moments from South by (which is what all the cool kids call it):

• Seeing Lijit publishers in action

Everywhere I looked at SXSW, I saw familiar names and URLs. Having the chance to thank users and speak with them firsthand about what they want from Lijit was priceless. I went to a session on building community and one of the speakers, Jake McKee, the Community Guy, was proudly wearing a Lijit t-shirt. Bestill my beating evangelistic heart!

• Blog Network Camp

b5media organized this event and Lijit was proud to be one of the sponsors. There was a great turnout and we enjoyed meeting Darren, Aaron, and Christina, among others. Todd gave an impressive demo of Lijit and the response from the bloggers at the event was positive. We’re looking forward to seeing how we can make network search even better!

• How to piss off a blogger

I attended this discussion on blogger relations and was pleased to find out that I don’t do any of the things we listed as ways to piss off bloggers. I’m self-taught and it’s taken me a year, but I feel that I’m in a position to contribute a lot to these panels. As I’ve always said, if you treat a blogger like you’d like to be treated, you’re probably doing the right thing. To read more about what we came up with, check out the post written by the session moderator.

• Hanging in da Bloghaus

This was the place to be to meet anyone who is anyone in the blogging world. Interviews were going on left and right, blog posts were frantically typed, free beer was consumed…and all with the energy and enthusiasm of a frat house. The Bloghaus is where I made the most connections, met the most Twitter friends and rubbed shoulders with the most A-listers. A very dangerous place for someone who freaks out around (very) minor celebrities!

• Startup Metrics for Pirates

Todd spoke on a panel about different ways of measuring the growth of your startup. While the panel spoke with knowledge and experience, it was also a great panel because they spoke like pirates. There were costumes, interactive pirate crowd comments and lots of “Yarrr”ing. Who knew seeing Todd with an eyepatch and earring would be one of the highlights of SXSW?

• The people
  

Duh. Everywhere you went, there was potential for inspiration and connection. Something about the mellow Austin mood must have rubbed off, because I had no problem walking up to anyone, regardless of their page views, and starting a conversation. It was such an electric energy, matching avatars to faces and meeting new friends. (Micah went as far as getting a permanent souvenir of the time in Austin.) You hear people say it all the time in the online world, but honestly, it really is all about the people.

You can read more about Todd’s SXSW experience here and here. Micah wrote about some lessons he learned and posted a round-up of how Colorado’s startups were represented at the event.

At Lijit we collect statistics on thousands of individual blogs, blog networks, and conventional online publications. For some time we have noticed that blogs that use the Lijit Search Wijit, specifically those that use the Re-Search feature, enjoy a page view lift as a result. Recently, we started digging into some of the usage statistics from our database and found some interesting trends.

In an earlier post, I spoke about the concept of “second click“. When a user enters search terms into Google or Yahoo and selects “search”, that is the first click. The large horizontal search engines own the first click.

When the reader selects a result from the list of items that Google or Yahoo has returned, that constitutes the second click. It’s the second click that leads readers to a publisher’s site. The average blog within in our network sees approximately 25% of its page views coming as a result of a referring horizontal search engine (or second click). The big search engines are a great source of new readers and page views if you can successfully hold on to that reader. The Lijit Search widget takes advantage of this situation to promote publishers’ content to readers when a blog page displays as a result of a referring search.

In our study, on a daily basis, the Lijit Search Wijit promoted a publisher’s content an average of 874 times and the content items in the re-search box were clicked on average 4% of the time. This converted to real-world page view increases, which ranged between 0% and 10% with an average of a little over a 1.5% increase.

These are all meaningful numbers in the world of publishers and subscribers. More page views results in more ad revenue and more engaged readers.

Some other interesting observations from the study… Publishers with the Lijit Search Widget above-the-fold (i.e. higher on the site) showed markedly higher second click conversion than those publishers with the Wijit installed below the fold. Publishers with information-centric sites optimized for SEO showed much greater second click impressions. Those publications that were more focused on Q&A or providing concrete information showed significantly higher second click conversions. Interestingly, higher page view publications were more likely to have a larger percent of users arrive from search engines, thereby setting up the second click scenario. Presumably, this is due to the higher page rank of these sites and the positive feedback loop that occurs as a result of this – basically, the big get bigger.

In the future I intend to dig into some of the interesting dynamics of blog networks. Blog Networks have specialty dynamics that make some of these results even more interesting and more pronounced. Stay tuned.

For now, make sure you have Re-Search turned on and your Lijit Search Wijit displayed above the fold to maximize your page views!

Venture Beat published an interview with Google’s Marissa Mayer that ran today written by Doug Sherrets. Andy@Lijit was nice enough to forward it on to me. I’m always interested in hearing what others think about socially influenced search (careful choice of words). Having lived in this world I know some of the more esoteric opportunities and problems of the space.

First, I have to clear the decks with something right out of the gate. The term “Social” has to be the most abused tech buzz word of the last couple years.

Todd’s list of what social isn’t…

1. A person, doing something.
2. A group of people, who don’t know each other, doing something.
3. A group of people, who don’t know each other but behave the same, doing something.
4. A group of people, who do know each other, but don’t interact, doing something.
5. A group of people, paid by someone, doing something. (test: What search startup is that?)

Quotes from the article with Marissa about Social initiatives at Google:

“One thing we tried…is labeling – have users annotate the search results they see and have those annotations be shared with people on their social network or with people of like mind and interest“

“Another classic thing to try is “other users like you”, where you build implicit social connections between users who are like each other“

“Other users that did that search, also searched for”

“You could take annotations that people have entered in something like Google Coop and broadcast the annotations.”

You see what is happening here don’t you? What derails ideas about social search are very simple. The Internet world grew up with a one box, take a pill, mentality. If we can’t search the entire world’s data in one box and have that box know what we mean, it’s a non-starter because of what we have come to expect.

Yet, social behavior is just the opposite. Social behavior is about the people we “know”. It’s important to “know” people in order to validate the result set they help deliver to you. We don’t “know” everyone and that contradicts the “one box” expectation of social based search. It also contradicts the “best answer” expectation. No matter how grand the plan is for social search you need to “cross the chasm” of who you and your network know, and who you and your network does not know.

For social search to work in a global, one box world, it cannot be completely a social model. What does work is using a social network of people you “know and interact with” to find a “local expert” within a known network. That “local expert” will likely have pointers to high quality data produced by “global experts”.

The system is not perfect. But it does work quite nicely in practice within the world of general knowledge and day-to-day problem solving, around YOU. Where the system is does not work as well are in situations where the local network, as whole, has little basic knowledge of the item or concept being searched. Interestingly, these are the situations where more traditional information stores like Wikipedia work exceedingly well. This is why I feel quite satisfied when I look up Nuclear Reactor in Wikipedia, but a person in the Nuclear Power space would look to his network to find more detailed information.

The graphic above is really what we want to achieve at Lijit. Though our Search Wijit we facilitate and explore the connections, discussions, and searches of connected “real” social networks. These people interact, get to know each other, have a proxy (the blog) for the metadata that exists in real world physical relationships. In other words they do more than simply throw food at each other. When the network – of networks – hits a critical mass we should have global pointers needed to cross the social chasm – in one box.

Micah shot me an interesting article the other day. It is titled “The Fight for the Second Click“. It’s pretty interesting and predicts Google’s move into (more) media services in order to monetize the click that occurs after “the search”. Google clearly owns the first click and has managed to monetize it rather effectively. In order to continue the revenue ramp, they will be increasingly forced to own the second click.

I found this concept fascinating. At Lijit we know from watching reader behavior on our publishers’ sites that a huge percentage (33%-50%) of readers come from horizontal search (The Second Click).

We also know that the normal behavior of one of these readers is to read the article that Google referenced and then hit the back button. Reader gone, moment lost, second click wasted.

This is precisely why the Lijit Re-Search feature was added to the Lijit Search Wijit. When you have this feature turned on, Lijit hooks the reader into staying for a third click and beyond. Bottom line, you only get one click to keep to your readers around – do the most you can to mine that opportunity.

Re-Search activates when a reader enters your site from a horizontal search. When a reader types “Comcast tivo cablecard” into the Google box, many results are returned, perhaps thousands. If one of those results is a link to your publication (in this example, my personal blog) and that reader clicks it, they end up on your site and The Second Click has been executed.

On a normal site without the Lijit Wijit installed it’s now a crap shoot as to what happens to that reader next. Statistically speaking the back button is most often the next click. By contrast, as the reader reads the reference article/post, Re-Search goes out and finds other articles/posts written by you on that same subject and offers them up in the Search Wijit. When the reader is done reading the reference article, there now exists the opportunity for you, the publisher, to own the third click – one of the other articles/posts on that subject.

Publishers! Maximize your second click, the one you own!

Scoble has a great couple of podcasts today on Social Graph based search. He talks about how new social based search plays are resistant to SEO (Search Engine Optimization) and are eventually going to kick Googles ass.

This is exactly the basis of what Lijit was founded on. The idea of social trust being the major driver in judging the value of content is spot on.

Scoble talks about how Mahalo is a social graph based search because they have content editors that are part of their social graph. I think the example is a little broken in the Mahalo case in that the graph is not really naturally occurring. In other words the graph as selected based on social popularity, who wants the job, etc. perhaps not actual expertise in a specific domain. He also suggests Facebook is in a great place to capitalize on their large social graph to revolutionize search. I agree, however Facebook is still a walled garden of content. If Facebook would extend their social reach (and content reach) beyond their walls they would be in a great position.

Lijit is using the naturally occurring networks that already exist out on the web to create powerful search from the ground up. We don’t hire content editors and we don’t force editors (publishers) into any specific social network. Lijit uses the networks you have to build the world’s most powerful foundation for search.

Great post Robert, we agree.