Search-Powered Web Apps

The following is a guest post by Bill, a software developer here at Lijit who is leading the search engine team. I thought his post was a great follow-up to the one from our CEO on the company’s recent funding and appreciate Bill taking the time to share his thoughts on how we should spend that cash.

As you may know, Lijit recently received 7.1 million dollars in venture capital funding. This means different things to different people. To the investors, it means they see promise in the company and are willing to take a calculated risk. To the executives, it’s a significant milestone in building a successful company. To the employees, it means we still have jobs, and it keeps alive the stock-option dream. But to the engineering team, it signifies a new phase.

Prior to this, Lijit has been a true startup. Everybody has worn multiple hats: the VP of Engineering does systems administration, the senior architect does configuration management, and everybody doubles as the QA team. This early phase can be very exciting and very satisfying. You get to do a little bit of everything, and the urgency to ‘just get it done’ means that you’re rarely constrained by bureaucracy or red tape. But it’s also a difficult time. Everybody is overworked, you often have to do tasks outside of your comfort zone, priorities and direction can change daily, and progress is often constricted by a lack of resources.

I only joined Lijit a few months ago, so I missed a lot of the early pains. But I got here in time to experience some of it, and I worked with many of the Lijit staff at a previous gig where we went through all these phases.

With the funding, it all begins to change. From an engineering perspective, this can be a very exciting time. In the past couple of months we’ve hired a QA team, built a dedicated Test Environment which mirrors the production system, and instituted a bug tracking process. We’ve brought additional developers on board with specialized skillsets, and organized into teams dedicated to each of our primary products (website, search platform, adserving platform). Not only have we built a talented IT team, we’re hiring a configuration management engineer. We’re adopting agile development methods, and we’re building a product roadmap and release timeline that give us direction months into the future. Across the board, we’re transitioning from a small team with limited process to a larger, more specialized team, with greater resources, and naturally, more process.

If we do this right, we become a more productive and higher quality organization which can quickly respond to business needs. If we do it wrong, we can become mired in process and overhead.

And that, really, is the exciting part–we get to define ‘doing this right’. The trick is to integrate these processes and resources while still remaining nimble. We get to pick and choose the parts that make sense. If it doesn’t make sense, if it doesn’t make us faster and improve our quality, then we don’t do it. It’s easy to get bogged down in all this stuff. But we won’t because we’ve been here before.

Photo credit: noahwesley

This week Cuil launched. We pay attention to everyone in the search space even though we really don’t consider most companies competitors. Lijit builds search based applications that install on publishers sites. We selected that mission very carefully because publishers are first and foremost our customer. We don’t operate a destination search site but rather empower the sites we reside on and the publishers we serve. Lijit has all the parts of a search company, we have lots of computers crawling and indexing the web and our secret sauce is the way we bring a level of trust to results and an alternate methodology to ranking pages.

In the world of destination search, Google pretty much owns the show. It’s not impossible for that to change as witnessed by Google’s own rise. The latest contender in destination search is Cuil. I tried a few searches and had really good results. Others, not so much.

I had to chuckle at the Cuil results for a search for “Lijit”.

Cuil seems to lead with the Wikipedia definition of things (as do many other destination search engines – much to Google’s consternation). In our case the top result is the Wikipedia page Stan made about us a long time ago. (An aside: Now, I’m pretty sure we are the most knowledgeable people about “lijit” but Wikipedia outwardly dogs our entry as being a “conflict of interest” regarding this subject matter. It seems to me that requiring people less knowledgeable to be involved is a conflict of interest with regards to Wikipedia).

Anyway, where I was going here are the images. I kind of like the tabloid presentation of things but where did these images come from?

Apparently the best match images of “Lijit” is of a tire company logo, a boat of some type, a “buy now” button (our service is free), and my favorite a guy sniffing glue.

Search is hard..

Yesterday Yahoo! announced their BOSS platform. BOSS is an API into their search infrastructure that lets application builders build a new unique search experience on top of all the Yahoo indexed content. It makes a lot of sense, as search is not an easy thing to do, especially when your corpus of content is the entire world. For fun I lined up Google, Yahoo, Me.dium, and Hakia.. The latter two are utilizing the BOSS platform. Finally, I threw in my Lijit search operated off Lijit’s own crawling and indexing infrastructure.

It’s actually pretty interesting how similar Google and Yahoo results are for this vast sample set of 1 search for “paris catacombs”.

Obviously Lijit returns much different results because Lijit isn’t constrained by the “one right answer” problem. The Lijit results are centric to me (the publisher) and my experiences, as if you had asked me about the Paris Catacombs. The right answer is what “I know” about the paris catacombs, the corpus is my experiences – my writings, my bookmarks, my photos, my videos and stuff my friends know. Of course, Lijit is not a destination search site so we live in a different space then the others. They have the problem of delivering one right answer, we have the difficulty of returning thousands.

The takeaway for me is Yahoo! BOSS and Google CSE are both great platforms to build innovative search applications on top of. However, at the end of the day, innovation on top will rule that day or other everyone will just return the same results formatted differently. Look forward to seeing more in this space from everyone.

During the Facebook frenzy of the last year or so I have heard over and over that Facebook will kill LinkedIn, or LinkedIn is playing catch-up to Facebook, or some derivative of that theme. This is of course based on the meteoric rise of Facebook’s user base, even within the 40 something segment where I reside. While I understand the thought process of this, I simply don’t get the utility out of Facebook that I get out of LinkedIn. Now, one could counter that LinkedIn isn’t any fun, which is true – but it is useful, specially finding people with specific expertise in something I need.

But if I take a step back for a minute, LinkedIn could be so much cooler.

Let’s take a real world example. Two years ago my daughter was doing a report on 3D rendering. I thought it would be cool to find out what rendering platforms existed and what people were using out there. It dawned on me to go to LinkedIn and look for people that worked at Pixar in the off chance I may know someone. It ended up that I found a guy that worked with a friend of mine from the simulation group back at NASA. This guy was a technical director on the movie Ratatouiie that was then in production. Using LinkedIn and my friend I was able to get a conference call scheduled and my daughter and I spent a hour learning about the profession, the tools, etc. All very cool, a total LinkedIn success story.

After that call it dawned on me how cool would it be if I could search for information though that person. Surely Pixar Guy had bookmarks in del.icio.us, perhaps things he has written that appear online, maybe even a blog. We were lucky that Pixar Guy would take an hour out of their busy day to take a call with a teenager and her dad, but for everyone person like that there are probably a 100 that would not. If I could leverage that person’s knowledge without having to bother them how cool would that be. Of course, Lijit is in the search space focused on finding information through people and their connections so the leap was easy for me to make.

Yesterday, LinkedIn announced Searchable Groups Directory and this morning I checked out their blog to see what it was all about.

LinkedIn now allows self forming professional groups to create a tighter Link between each other. For instance one group in their screen image is the CIO forum. Imagine being able to search (for information) through some of the world’s top CIO’s. That would be an interesting perspective on enterprise software for sure.

So, for fun I decided to take a look what an integration of LinkedIn and Lijit may look like. The results were pretty cool.

Click on my hacked version of my LinkedIn page and then enter a search term in the new “Search My Content” field. I listed myself on LinkedIn as a startup CEO and CTO so a natural search term to enter maybe something like “Angel Financing“.

This could be a whole new layer of engagement within LinkedIn that provides even more use to me. In fact, based on the information that I can glean it would seem I would be a lot more likely to use the LinkedIn Answers because I drilled down on useful information..

Now image this capability tied to the LinkedIn Searchable Groups feature they just released. Essentially this would become the world’s largest vertical search engine generator. How cool is that?

Here at Lijit, there are quite a few things we take seriously. For example, we take customer service seriously. We monitor twitter and other services to make sure that our users are happy. We take search technology seriously, constantly looking for ways to improve our value to publishers.

But, there is something that might be taken even more seriously here at Lijit.

Sushi.

Thats right. Lucious fish on savory rice balls.

Luckily, Colorado has fantastic sushi restaurants (yes we know we are land locked. Ever hear of these things called planes? They can carry fish, thank you very much.)

But, its just not enough for us at Lijit.

We want to be known as the Greatest Sushi Eaters of All Time.

So we have decided to sponsor the first ever:

2008 Boulder Sushi Regurge Open

You read that right.

The rules are simple, and as follows:

1. You must eat sushi, not that pansy sashimi crap.
2. You may order your own sushi.
3. There is a $25 entry fee that will go to the establishment (that still doesnt know what they are hosting)
4. You will pay $1 for each piece of sushi you eat into the pot.
5. Top three finishers split the pot. (60% to the winner; 30% to second place; 10% to third)
6. You may heckle, but not touch other participants.
7. No bathroom breaks.
8. The final three will order sushi for the person to their left.
9. You are out if you dont eat sushi fast enough.
10. You are out if you throw up.
11. You are out if you dont recognize the greatness that is Micah.
12. The tie breaker will be desserts.

And we are throwing the guantlet to a few folks:

Brad Feld: Perhaps you should invest in some preparation for a beat down.

EventVue: Can you register the beat down you are going to get?

David Cohen: Maybe the Techstars teams will think you are still cool.

FlingItGirl: You can Fling It, but can you Bring It? (Oh, its been broughten!)

Danny Newman: Did you invent the loss too?

StickerGiant: Make a sticker that sucks as bad as your sushi eating skills.

Andrew Hyde: Here’s an idea. Why not start Startup AndrewCantEatSushi?

Me.dium: Why are all the little dudes in my sidebar going to YouCantEatSushi.com?

Bruce Wyman: Your loss will be a piece of art.

Intense Debate: Lets discuss how bad your sushi eating skills are.

Techstars 2008: I hope you can eat sushi better than…well thats just mean.

Jeremy Tanner: You might have jokes, but you dont have sushi eating skills.

Of course, all are welcome to show up, although most will leave disappointed.

We plan to live blog, UStream, Tweet and Social Media/Web 2.0 the heck out of the event.

We may make fun of many things, but we dont joke about sushi.

Its ON!

(Time and Date to be announced. If you want to participate leave a comment. No animals were hurt in the production of this blog post, but man, there will be some fish dying!)

For the past 15 years or so, I have coached youth lacrosse. Usually eigth graders, sometimes younger, but always about the time when growth spurts occur. In fact, every year there is a kid that shuffles, limps or hops when he runs because of the pain, the growing pains, in his legs.

The nice thing is that as soon as it passes, the kid is usually a much better player, and often truly excels at the sport.

Startups are often the same way. Yesterday, we were written up in TechCrunch. The focus of the article was: Lijit was slow, they figured it out, and now they are growing quickly again, landing a great partnership. Just like the lacrosse kids. We experienced growing pains, learned from them, and have begun to grow even more quickly because of them.

We also learnedhow our product worked with larger networks. Like the TechCrunch article said, we increased b5media’s overall pageviews by 1.5%, and that was with two search boxes and a below the fold placement.

But that wasnt the only interesting statistic. Approximately 23% of all clicks on a search results page went to a site other than the original publication. Meaning that 23% of the clicks were cross-promotional in nature. This was a great discovery since one of the core values of Lijit is helping readers discover new content. In essence, 23% of the readers searching across the b5media network were finding sites that they may have never known existed, driving additional readership and pageviews.

Our learnings were simple: 1) search is hard to do well; 2) search has to be fast; and 3) search has to be relevant to be useful. As we continue to innovate our product, we are finding new and interesting successes and challenges, and we while we hope that our growth continues at the same rate, we are ultra focused on ensuring both speed and relevance.

Interestingly enough, we arent the only startup that has faced rapid growth and scaling issues because of it:

Hopefully, we all learn a few lessons along the way…

And, if you are a publisher network looking to drive some additional page views and cross promotional traffic, please drop me a line at micah [at] lijit [dot] com. I promise to hook you up!

Part of startup life is that a different part of the company gets the focus at different times. For a few months Dan has had the Monkey riding around on his back while we scale the Lijit infrastructure to deliver the service everybody is signing up for (faster then we imagined). Last week we released our new software platform on our new hardware (200 servers) in our new data center (12 racks) and everything is running Sweeeeet!

Just this morning the monkey left Dan’s office, grabbed a banana in the kitchen and started sending emails from Micah’s desk. He seems to be trying to “help” Micah fill up all that infrastructure with even more publishers !

Zach is officially known as a “Systems Architect” around the office, but unofficially known as the guy who makes sure that Mike doesn’t go crazy. Our CEO, Todd Vernon, wooed Zach for a while before he was able to talk him into jumping aboard. Rumor has it that the sweet office in downtown Boulder sealed the deal for him. There is much to learn about Zach and the following questions are only the beginning of our lesson…

• What is your Lijit contribution?

I come to Lijit with about 13 years of practical systems, network and security experience across environments large and small. I hope to make a positive impact especially in the production network, as Lijit has significant performance and security demands there. I also want to help make the production and office systems more manageable, so that over time people find it easier to get their jobs done, and developers and IT folk wake up to fewer surprises in the data center.

• What is your least favorite word and why?

Performant, because it transforms a real verb into a fake adjective in order to shorten the three syllable phrase “performs well” into a three syllable word that makes one sound like an EMC salesperson, and EMC sales people give me the willies.

• What sound or noise do you love and why?

I have a new drum set, and the kick drum sounds like a howitzer. I love that.

• What profession, other than your own, would you like to attempt and why?

Cosmology. If you’re a cosmologist, you can never fully understand your own profession, but nobody could reasonably expect you to anyway, so it’s OK. I just read an article asserting that in the future, the universe will be expanding so rapidly that light from its farthest recesses will never be able to reach observers on Earth, and Earthicans may perceive our by then bloated galaxy as the extent of the universe. So the job should only get easier with time.

• Something you’re guilty of…

I introduced my father to Gran Turismo 4 for the PS2, knowing that he would become addicted to it, potentially destabilizing his job and family life. It worked, and I smile inside every time I think about it. For so many years he mocked me for spending my allowance on video games. Who’s laughing now?

Like a few others, Zach started his blog when he was hired here. He says he wants to have a place to test the Lijit widget (likely story!) but he hasn’t actually posted anything yet. Perhaps when he sees all the traffic I’m pushing his way, he’ll be inspired to write something. I mean, since he’s already demonstrated his wit with the above answers, there are really no acceptable excuses. Zach, thanks for playing along; we can’t wait to see more from you!

At Lijit, we are constantly developing and testing new features for the product, the search results and the re-search box.

Recently, we have begun to explore the concept of including Flickr thumbnails within the search results themselves. Pretty cool idea, right?

What we cant come to a final decision on is exactly where the thumbnails should be placed. So, we are asking our users for feedback.

Here are the choices:

Here are the thumbnails on the LEFT (click the picture to expand):

Here are the thumbnails on the RIGHT (click the picture to expand):

And, here is a quick poll to help us figure this out. The cool thing about this quiz, is you can cut and paste it onto your blog, and share it with your friends. Check it out and thanks for the help!

Last month, Eric Olson and Frank Gruber brought TECH cocktail to Boulder. Lijit was more than happy to help sponsor the event. While they were in town, the guys stopped by our offices to say hello and Frank wanted an official tour. Watch to see what happens as we cruise around the Lijit offices…