Search-Powered Web Apps

This post is by our very own Derek Greentree, who visited the Yahoo! campus for a field trip. In addition to using his business cards for the first time while on this trip, he was also very excited about being the recipient of developer swag. And he was nice enough to share it with the rest of us in the office.

Two weeks ago, I flew out to sunny California to attend Yahoo! Open Hack Day, an interesting developer event periodically held by Yahoo!. Attendees get access during the event to up-and-coming APIs (and other technology) in development at Yahoo!, and are given a challenge to create a mashup or other interesting hack and demo it at the end of the event. You can find information about the hacks that were demoed at the hack day blog.

If you’ve never been to the Yahoo! Sunnyvale campus, you probably won’t know quite what to think, as it’s filled with purple carpet, exclamation points on the walls, and emoticons everywhere. It’s interesting to see a very large corporation try to promote a youthful and exuberant appearance at a corporate headquarters with security guards, a cafeteria, fountains, volleyball courts, outside dining, and multiple buildings. I arrived Friday morning and was also surprised at how organized the event was; after entering, I was always greeted with friendly faces willing and able to help me find what I needed.

The theme of the weekend was APIs and openness. First, I’m happy to see that Yahoo! is getting behind OAuth, an open standard for API authentication. I wish everyone supported this simple mechanism (*cough* Facebook *cough*), as the various APIs offered by services out there desperately need to settle on a single, well-understood mechanism for allowing users to grant access to private data.

The second most exciting thing demoed (for me) that weekend was YQL, which will be a single URL that takes a SQL-like query and returns data from many Yahoo! services–like Flickr, mybloglog, Yahoo! Messenger, Yahoo! Mail, and others. Currently, each of these services has a different API and a different authentication mechanism, which means that supporting them is a pain. Having one mechanism for authenticating to the Yahoo! API (OAuth) and for querying any data within it will greatly simplify the code base we use here to interface with external services.

Next was Yahoo! BOSS, a search API. Many services (like Google and even Yahoo!, using yet another API) expose search services to external users, but BOSS is special. The problems with all the major existing APIs out there is draconian usage restrictions. Many of them, for example, don’t allow you to:

• Reorder search results
• Change the display of search results
• Use the API more than a certain (low) number of times per day
• Inject advertising of your own into search results

This is a mistake: allowing services to use your search data in unique and interesting ways is a win-win for the service and the search provider. With BOSS, on the other hand:

“BOSS (Build your Own Search Service) is different–it’s a truly open API with as few rules and limitations as possible. With BOSS, developers and startups now have the technology and infrastructure to build next generation search solutions that can compete head-to-head with the principals in the search industry. BOSS will grow and evolve with a focus on providing additional functionality, tools, and data for developers.”

I attended an excellent presentation by Vik Singh, a member of the BOSS team, on some example usage. He’s also developed a very cool Python library called the BOSS Mashup Framework that lets you whip up interesting mashups using Yahoo! Search with simple, elegant code. If you’re a developer and into this stuff, you definitely should check things out.

Perhaps most importantly, the weekend made me think about APIs that Lijit wants to offer (or could offer) to the outside world. We have access to a lot of interesting and unique data here, and I’ve seen more than a few startups that would benefit by having access to the work we’ve done. Look for more on this in the future, but I think providing access to Lijit’s data via a set of APIs could produce some very interesting mashups, and help Lijit grow in cool and unique ways.

[Photos found on Flickr and used via CC license: freshelectrons, Jinho.Jung, and bluesmoon]

Once the gauntlet was thrown, there was no looking back. Micah pulled off a friendly competition that brought together all those with big appetites in the local tech community. 12 guys showed up to compete and after deciding on the rules together, the contest began. But not before the spectators put their predictions in for who they thought would win. Jenny from Boston thought that Danny Newman was the one to pick, mainly based on the fact that he came dressed to play…

Brad Feld was honest when he said that he would stop eating when he was full…because Amy, his wife, told him to and we all know what a good husband he is. (Additionally, Brad kept making the competition harder by adding in his own rules, like drinking a beer between each round and attempting push-ups as well.)

Spectators took turn ordering the rounds of sushi so that the competitors didn’t have any unfair advantages. The first few rounds were easy, but the sushi got progressively worse as the night wore on. Plates of the hard stuff (quail eggs, smelt roe, fermented soybean, and the like) began appearing in the later rounds, knocking out weak-stomached competitors along the way.

After 14 rounds of sushi, one competitor puking, and many others looking like they were going to puke, the first Sushi Regurge champion was crowned. To the surprise of no one, the same guy who organized the entire event (and talked the most trash) was the one who was left standing at the end of it all.

Many thanks to the employees of Hapa, who put up with our screaming and cheering, and to everyone who showed up to support the competitors. And of course, thanks to Micah…

without whom, none of this disgusting gorging would be possible. Thanks for pulling it together, calling everyone out, and putting your best stomach forward! We look forward to another equally unappetizing gathering like this again next year.

The most common question I get (well, after “Have you ever heard the MC Hammer song? Really? You have?”) is “Isnt Lijit just a search tool?”

I always reply the same, “No, we are a publisher advocate.”

Which is always greeted with one of three responses: 1) a look of bewilderment; 2) a look of amusement; or 3) a look of agreement.

Perhaps the strangest response I get is: “Why?”

For Lijit, the answer is simple. Because our entire existence is predicated on publishers. Not our business model mind you (although thats part of it) but our core value.

Our belief about publisher widgets is that there are two types: Widgets that exist to make publishers better publishers and seek to develop a true partnership and widgets that provide some value extension to the publisher.

The first type are publisher advocates, they have to improve the entire experience, both for the publisher and the reader.

The second type either is successful only on a high traffic publisher, or only for one consistuency, the publisher or the reader.

Our guiding principle when we add features to Lijit is simple: “Are We Being Publisher Advocates?”

In other words, does this feature make a publisher a better publisher by providing better service or increased engagement to their readers?

This also limits our focus to three areas:

1. Content Discovery / Reader Engagement

By indexing all of a publisher’s social content and trusted sources, Lijit allows content that may have been buried in a general search engine search to bubble to the top. Why? Well, we only index the things that are important to you; general search engines index everything. So, our base value proposition is that a publisher’s readers should find everything that a publisher trusts and wishes to expose.

In addition, when a reader comes from a general search engine, our “Re-Search” box proves additional implicit white-labeled results that tend to have a relatively high click through rate, effectively keeping a reader on the publisher’s site versus clicking the back button to the search engine.

Our stats also provide a variety of information for a publisher including results that returned zero results, providing a clue as to what readers are looking for from the publisher, potentially helping to inspire future posts or articles.

2. Optimization of Monetization

Publisher monetization is a noisy, competitive field, and currently we are loathe to produce a sub-standard ad experience for publishers. We cannot just be Yet Another Google Adsense Clone. We have to be better.

Lijit has to create an experience where publishers are optimizing revenue from an under-monetized section of a publication, namely the search results.

Everyone knows that search can be monetized effectively, but we believe because the results driven through Lijit are more contextual and relevant, the resulting revenue should be higher for the publisher. So, we are spending a lot of time developing an effective user interface and experience. Its hard and takes a long time, and we are close.

Besides search results, there are two immediate things that occur when using Lijit search. Your current social content gets better promotion increasing your overall pageviews, driving additional revenue now.

3. Cross Promotional Traffic

This is really effective if a publisher has multiple blogs or a blog network. With Lijit a publisher can use a high volume publication to help drive traffic horizontally to lower traffic blogs through cross-promotion in the search results. On average, our blog networks find that almost 30% of the results clicked in a search result are to another network blog, rather than the originating publication.

Each of these three functions: Content Discovery/Reader Engagement, Optimization of Monetization and Cross Promotional Traffic are all examples of how we feel that we are being publisher advocates, helping publishers be better publishers and helping them serve their readers.

After all, at Lijit we know one thing to be an absolute truth:

If publishers didnt provide social content or trusted sources, our results pages would be empty.

Over the past month or so, we have been working with several companies to better integrate their content into our results.

We recently improved the integration of Flickr and YouTube and have continued down this path, by adding six additional content sources.

Initally, most of the integrations make it easier for us to index that content, but over the next few months, we hope to bring some interesting innovations to the results themselves (see this post for a sneak peak.)

So what are the content sources? Glad you asked (in alphabetically order, and in their own words):

Disqus - “Disqus, pronounced discuss, is a service for blog comments. Disqus enables bloggers to make the conversations on their blogs more interactive and manageable. Our distributed comment system connects readers across blog communities, while empowering publishers in promoting their content.”

Intense Debate - “Intense debate is a comment replacement service for blogs. Intense Debate increases the interactivity of blog comments driving more reader engagement. In addition, Intense Debate tracks your comments, and the comments of your friends, across all blogs helping publishers be discovered.”

Mixx - “At Mixx, all of the content is user-generated, which means that we don’t have an editor who decides what you should find interesting. We figure that you can find that same-old same-old pretty much anywhere else. So instead, we make it easy for you to submit stories, photos, videos—whatever you find interesting, informative, different or fun. You can also see—and vote on—the stuff that other Mixxers submit.”

Tumblr - “To make a simple analogy: If blogs are journals, tumblelogs are scrapbooks. You can also look at tumblelogs as slightly more structured blogs that make it easier, faster, and more fun to post and share stuff you find or create.” (Note: If you have a custom domain for your tumblr, enter it as a blog or URL. We are still working through some minor things.)

Viddler - “A fresh, creative web application that allows you to upload, enhance, and share digital video easily and quickly inside your web browser.”

Vimeo - “Simply put, Vimeo is the home for videos you make. You can use Vimeo to upload and share videos with anyone, watch great videos, meet new people, and explore Vimeo the way you want to.”

So, there you have it. Six new content sources. We are planning on continuing to add new sources every month or so.

Have a source you would like us to add? Please leave a comment, and we will start the process.

It’s official. If you look for us here–

you’ll be able to pick us out from this list…

and easily put our wijit on your Typepad blog.

In addition, we are featured here:

It brings us much joy to know that Lijit is being accepted into the realm of Widget Galleries. (Now if they would only make us happier by changing their spelling to wijit!)

TechStars brings aspiring technology founders to Boulder, Colorado for an intensive three month period, provides seed funding, education, and connections, and will result in the formation of ten new companies during the summer of 2007.

Techstars gives driven people a way to make it happen! And we here at Lijit are nothing but driven. Lijit is a huge idea of re-building Internet Search from the ground up, starting with the data individuals forage on the Internet and the relationships that help shape that data into information. We released our first service about a month ago and it was a natural to try out on Techstars!

Using the Lijit Search Wijit on the TechStars website, you can search the knowledge base of all the Techstars Mentors at once. Not just their blogs, but their del.icio.us bookmarks, flickr photos, and even others that influence their opinions.

Give it a try! Search on “Venture Funding”…something near and dear to an entrepreneur’s heart! See what the Techstars Mentors have said and bookmarked.

We think Techstars is awesome, and what could be better than searching for entrepreneurial subjects through entrepreneurial eyes!