Search-Powered Web Apps

The Lijit team spent last weekend at the Blog World Expo in Vegas. In addition to speaking with bloggers, answering questions, and thanking our current users, we had the opportunity to debut our new trade show booth. While this may not sound exciting, it represented a big deal because it’s the first real booth-like thing that Lijit has ever had. We put a lot of heart into our trade show presence last year, but we put a lot of money into it this year and it made a difference.

Armed with the popular Lijit swag of small laptop stickers and uber comfortable t-shirts, we decided that we needed something more. We wanted to provide even more value and functionality to the bloggers attending the conference, so we decided to help people clean up their search. And their hands. (Don’t forget what a dirty city Vegas is…)

Besides spreading the joy of hand-sanitizing, we also helped to spread the word of our search. The beauty of Blog World is that everyone there is passionate about what they’re doing, so from wine bloggers to mommy bloggers, we had the chance to talk to people about what they love doing and to show them how Lijit can make them even better at it.

In addition to hearing feedback from blog publishers already using Lijit (big thank-you!), we also had the chance to convert some new publishers to our search goodness. Welcome…

Quickrelease.tv
Volleyball Voices
Vino Las Vegas
MoneyNing
Frugal Upstate
John Hawkins Unrated
Don Nozzle
The Wiz of Odds
Money Making Scoop

While we didn’t have much time to walk around the trade show floor, there were a few companies that caught our eye, including MindTouch, Zemanta, Woopra, JS-Kit, Photrade, ContentRobot, and BlogCatalog. This represents just a small sampling of the many Blog World exhibitors present and we only wish there had been more time to find out about all the companies in attendance.

We loved having the chance to co-sponsor some giveaways with b5media, because they are always such a fun group to be around. In addition to the poker chips they were handing out for the game in their booth, we got to see some of our favorite b5 bloggers, like Mary Jo…

Lijit was mentioned in a few other Blog World recaps. John Andrews, Senior Marketing Manager for Walmart, was impressed with us (as you can see in this video clip) and we sold Ben Koo on the up-and-coming Boulder tech scene. David called us some of his favorite naggers, which he meant as a compliment (I believe) and Carlton liked our Purell swag so much, he blogged about it. Additionally, Lijit captured Greta’s interest and managed to make an impression on Karen again this year.

The entire Lijit team worked hard while at Blog World and must thank all those who stopped by our booth to say hello! We appreciate your interest and support in making this year’s Blog World an even bigger success than last year. See you in Vegas next year!

[photo credit: carltonreid]

This post is by our very own Derek Greentree, who visited the Yahoo! campus for a field trip. In addition to using his business cards for the first time while on this trip, he was also very excited about being the recipient of developer swag. And he was nice enough to share it with the rest of us in the office.

Two weeks ago, I flew out to sunny California to attend Yahoo! Open Hack Day, an interesting developer event periodically held by Yahoo!. Attendees get access during the event to up-and-coming APIs (and other technology) in development at Yahoo!, and are given a challenge to create a mashup or other interesting hack and demo it at the end of the event. You can find information about the hacks that were demoed at the hack day blog.

If you’ve never been to the Yahoo! Sunnyvale campus, you probably won’t know quite what to think, as it’s filled with purple carpet, exclamation points on the walls, and emoticons everywhere. It’s interesting to see a very large corporation try to promote a youthful and exuberant appearance at a corporate headquarters with security guards, a cafeteria, fountains, volleyball courts, outside dining, and multiple buildings. I arrived Friday morning and was also surprised at how organized the event was; after entering, I was always greeted with friendly faces willing and able to help me find what I needed.

The theme of the weekend was APIs and openness. First, I’m happy to see that Yahoo! is getting behind OAuth, an open standard for API authentication. I wish everyone supported this simple mechanism (*cough* Facebook *cough*), as the various APIs offered by services out there desperately need to settle on a single, well-understood mechanism for allowing users to grant access to private data.

The second most exciting thing demoed (for me) that weekend was YQL, which will be a single URL that takes a SQL-like query and returns data from many Yahoo! services–like Flickr, mybloglog, Yahoo! Messenger, Yahoo! Mail, and others. Currently, each of these services has a different API and a different authentication mechanism, which means that supporting them is a pain. Having one mechanism for authenticating to the Yahoo! API (OAuth) and for querying any data within it will greatly simplify the code base we use here to interface with external services.

Next was Yahoo! BOSS, a search API. Many services (like Google and even Yahoo!, using yet another API) expose search services to external users, but BOSS is special. The problems with all the major existing APIs out there is draconian usage restrictions. Many of them, for example, don’t allow you to:

• Reorder search results
• Change the display of search results
• Use the API more than a certain (low) number of times per day
• Inject advertising of your own into search results

This is a mistake: allowing services to use your search data in unique and interesting ways is a win-win for the service and the search provider. With BOSS, on the other hand:

“BOSS (Build your Own Search Service) is different–it’s a truly open API with as few rules and limitations as possible. With BOSS, developers and startups now have the technology and infrastructure to build next generation search solutions that can compete head-to-head with the principals in the search industry. BOSS will grow and evolve with a focus on providing additional functionality, tools, and data for developers.”

I attended an excellent presentation by Vik Singh, a member of the BOSS team, on some example usage. He’s also developed a very cool Python library called the BOSS Mashup Framework that lets you whip up interesting mashups using Yahoo! Search with simple, elegant code. If you’re a developer and into this stuff, you definitely should check things out.

Perhaps most importantly, the weekend made me think about APIs that Lijit wants to offer (or could offer) to the outside world. We have access to a lot of interesting and unique data here, and I’ve seen more than a few startups that would benefit by having access to the work we’ve done. Look for more on this in the future, but I think providing access to Lijit’s data via a set of APIs could produce some very interesting mashups, and help Lijit grow in cool and unique ways.

[Photos found on Flickr and used via CC license: freshelectrons, Jinho.Jung, and bluesmoon]

This Wednesday, Frank Gruber and Eric Olson bring their traveling techie road show to Boulder for a second appearance. If you’ve never been to a TECH cocktail before, you’re in for an evening of fun. Imagine a bunch of tech enthusiasts, VCs, bloggers, and entrepreneurs drinking together, watching demos, and sending the geek quotient of the Boulder Theater through the roof. Yes, it really is that good. (And I’m not just saying that because Lijit is a sponsor…)

• Where? The esteemed Boulder Theater
• When? August 20th, from 6:30-9:00 PM (although the TC site says that it’s taking place EST, I wouldn’t bet on that, or we’re all going to be early…)
• Why? Duh…because geeks and beer are a great combination.
• How much? Totally free, but you must RSVP

Last time the boys brought the show to town, we found out that as a sponsor, Lijit was supposed to be one of the demos. We hadn’t really thought about that and, at the last moment, grabbed our homemade widget to bring with us to the party. People loved being turned into widgets, as Frank Gruber can attest…

While we’re not sure if the widget will make an appearance this time, the TechStars are coming out to play and they will be demoing their wares. TECH cocktail will be the conclusion to their day-long investor presentations, so you can imagine the relief these teams will be feeling. The TechStars have been working hard all summer and honestly…the least you can do is to come out and support your local tech community by drinking some beer. Is that too much to ask?

In conclusion…

What more could you want? (Unfortunately, due to the limited supply of Lychee liqueur, there will probably NOT be any Lijitos made at this event.) You can RSVP here and we hope to see you on Wednesday night!

After a lot of work, we are proud to introduce something new for our WordPress.org users. If you like the Lijit search and want it to be integrated on your site in a more, how shall we put it, hijacked fashion, then this plugin is for you.

As everyone knows, you can currently install the Lijit widget on your blog in the traditional, add-the-code-to-your-sidebar kind of way. But now, with this new Wordpress plugin, you can keep the current search bar you have on your WordPress blog, but have Lijit power the search results. It’s like magic.

It looks like a normal search bar, but when you do a search, BAM! Search results brought to you courtesy of Lijit.

Like any other Wordpress plugin, you can find the Lijit Search (version 1.0) in the plugin directory. Or HERE.

Once you download the widget, simply log into your WordPress dashboard, go to the plugin page and activate it. Upon activation, you get to choose whether you want to use your existing Lijit account or you can create a new one inside the WordPress admin console.

You can also choose between the classic widget install option OR hijack the existing WordPress search box and make it Lijit-enabled.

But wait! That’s not all. We’ve also included another super-cool feature…integrated statistics. You love the stats that Lijit gives and now you can see those from the comfort of your WordPress dashboard. Simply click on your Lijit stats and you’ll see all that detailed goodness, telling you exactly what you need in order to keep your readers happy.

Many of you asked for a better way to integrate Lijit search on your WordPress blog and here it is. With a cleaner and tighter feel, Lijit and Wordpress.org are playing even better together than ever.

***This plug-in is only available for those WordPress users with versions 2.3-2.6.***

We would like to thank the great team at Crowd Favorite for helping us to make this happen. And by all means, let us know what you think of the plugin. We think it rocks, but are more excited to hear your thoughts.

Happy hijacking!

The following is a guest post by Bill, a software developer here at Lijit who is leading the search engine team. I thought his post was a great follow-up to the one from our CEO on the company’s recent funding and appreciate Bill taking the time to share his thoughts on how we should spend that cash.

As you may know, Lijit recently received 7.1 million dollars in venture capital funding. This means different things to different people. To the investors, it means they see promise in the company and are willing to take a calculated risk. To the executives, it’s a significant milestone in building a successful company. To the employees, it means we still have jobs, and it keeps alive the stock-option dream. But to the engineering team, it signifies a new phase.

Prior to this, Lijit has been a true startup. Everybody has worn multiple hats: the VP of Engineering does systems administration, the senior architect does configuration management, and everybody doubles as the QA team. This early phase can be very exciting and very satisfying. You get to do a little bit of everything, and the urgency to ‘just get it done’ means that you’re rarely constrained by bureaucracy or red tape. But it’s also a difficult time. Everybody is overworked, you often have to do tasks outside of your comfort zone, priorities and direction can change daily, and progress is often constricted by a lack of resources.

I only joined Lijit a few months ago, so I missed a lot of the early pains. But I got here in time to experience some of it, and I worked with many of the Lijit staff at a previous gig where we went through all these phases.

With the funding, it all begins to change. From an engineering perspective, this can be a very exciting time. In the past couple of months we’ve hired a QA team, built a dedicated Test Environment which mirrors the production system, and instituted a bug tracking process. We’ve brought additional developers on board with specialized skillsets, and organized into teams dedicated to each of our primary products (website, search platform, adserving platform). Not only have we built a talented IT team, we’re hiring a configuration management engineer. We’re adopting agile development methods, and we’re building a product roadmap and release timeline that give us direction months into the future. Across the board, we’re transitioning from a small team with limited process to a larger, more specialized team, with greater resources, and naturally, more process.

If we do this right, we become a more productive and higher quality organization which can quickly respond to business needs. If we do it wrong, we can become mired in process and overhead.

And that, really, is the exciting part–we get to define ‘doing this right’. The trick is to integrate these processes and resources while still remaining nimble. We get to pick and choose the parts that make sense. If it doesn’t make sense, if it doesn’t make us faster and improve our quality, then we don’t do it. It’s easy to get bogged down in all this stuff. But we won’t because we’ve been here before.

Photo credit: noahwesley

I’m proud to share another in a series of guest posts written by Lijit employees. This week we present an installment from Mike, who seemed overly excited about writing and sharing this post.

Hi, I’m Mike Merideth, the Director of IT here at Lijit, and I’m going to talk a little bit about the nuts and bolts of how we do what we do. Over the past year I’ve had the opportunity to design and implement the production network and server infrastructure on which Lijit runs. It’s been a great year of challenges and breakthroughs, but if there’s one key architectural concept that has gotten Lijit to where it is today, it is virtualization. We use Xen for our virtualization technology, which has the advantage of being free Software (both in the “free beer” sense and the “free speech” sense). CentOS 5.1 (a Linux distribution which is based on the market leader RedHat) includes this functionality out of the box, and has performed very well for us.

So why does Lijit use virtualization? There are a number of good reasons:

Flexibility: When you’re launching a new web product, it can be hard to predict what pieces of the application will need more resources than you originally gave them, and which will need less. We’re able to change the amount of memory, the number of CPUs and the amount of disk space a server has quickly, easily and remotely.

Availability: Because we use an iSCSI SAN for most all of our storage, we can move virtual servers between pieces of physical hardware. So if we lose one of our physical servers, we can quickly bring up the virtual servers it hosted somewhere else.

Resource utilization: CPUs today are incredibly fast and powerful; far more so than most applications need. Similarly, RAM has become cheap enough that a server with 16 or even 32 gigabytes of RAM is not particularly unusual, or particularly expensive. Running a simple web server on such a system would be a waste of CPU and memory, and therefore a waste of electricity. If you can run several virtual servers on such a system, however, you can get the maximum return on your investment by making sure you’re fully utilizing all of the CPUs and all of the RAM. Which is all tied to…

Cost savings: Colocation is expensive, and electricity certainly isn’t getting any cheaper. Using virtualization means we can get the absolute greatest value out of the rack space and electricity we’re paying for.

As of right now, we’re running about 200 virtual servers on about 25 physical servers. Just a few years ago we would have needed scores of physical servers consuming thousands and thousands of watts of power to do the work we’re able to do in this relatively modest environment. For a startup that would mean a higher burn rate with a shorter runway, and greater stock dilution for the founding stakeholders because of the amount of capital needed to get the work done. If you’re trying to get a tech startup off the ground, you owe it to yourself to see if you can leverage virtualization in your IT architecture. You’d really be crazy not too.

If you managed to read this post without your eyes glazing over, you may be interested in my new Linux infrastructure blog at http://linfrastructure.blogspot.com. I’m keeping notes on my experiences there, in the hopes that what I’ve learned over the past year can benefit others who find themselves in the same boat.

Photo credit: Leonard John Matthews

Recently, one of our publishers reached out to us to get our take on third party cookies. They were considering removing our widget due to the fact that we set a cookie when a browser views our widget. This brings up the debate over security and safety when it comes to third party cookies, and cookies in general. We responded to the post in the comments, but I wanted to elaborate on that a bit more.

Why Lijit uses cookies:

We use cookies to not only track whether someone is a Lijit user (allowing them to login etc), but also to allow us to match up a blog/widget visitor to any searches they may perform through our widget. This helps us to provide valuable metrics to our publishers in the form of stats, which in turn allows publishers to give their readers better content.
Generally,cookies allow us to gather better data about our users. Due to the ever rising pattern of “cookie blocking”, either by browsers, firewalls, security software,or explicitly by users, we have had to find other ways to continue gathering statistics. This means we have to use traffic pattern matching techniques, and logging analysis to get all of the data we need.

Historically, advertisers were the primary “pushers” of third-party cookies. This would allow them to track your viewing behaviors across any properties where their ads appeared. Some people disliked this since there was no real value to the web user, and the advertisers got free data. This was perceived as not only a security and privacy issue , but also pushed the perception of third party cookies into a grey area. These conditions, and the rise of spyware and malware pushed OS, and browser companies to institute tighter control.

In the current world of social media , distributed web services, and widespread widget adoption, the value to the user has changed. There are many services, Lijit included, that offer value to the user during their browsing session, versus just “tracking” them. The key, is that the web user is informed about what sites they visit, and the kind of content they allow in their browser. It is important for companies to disclose how they use the information they collect, and Lijit does this in our privacy policy.

Overall, the message should be about awareness and consumer education. The value of enabling third party cookies can actually be additive to the consumer vs. being a security or privacy concern. Modern browsers allow you to whitelist services you trust, and there are many services on the web (such as Lijit) that deserve that trust.

Derek Greentree, one of our Senior Software Engineers, has agreed to share his thoughts on what he does here at Lijit. You can read the first part of this series here.

When you first start optimizing a website, there are many questions to answer. What are you optimizing for? Raw speed? Maximum concurrency? How will you determine the bottlenecks in your way? How will you test the optimizations you make to be sure they were real and not placebos? How will your optimizations scale? And finally, how much time are you willing to put in for what amount of return? A famous quote among software development people, coined by Donald Knuth, states that premature optimization is the root of all evil. Web applications are no exception.

First, you’ll need an environment where you can test your changes. You should have a test environment anyway, but this environment will exist mostly for benchmarking, which means you’ll be putting load on it, so you may want to have a separate environment just for load testing. Since you’re going to be deploying your optimizations eventually to your production servers, this environment needs to mimic your production architecture as closely as possible. Don’t load test on a machine that serves its content from a network share, when your production servers serve that content from local disk.

When you have an environment ready, your next step is to benchmark what you currently have. Many tools exist for this, but at Lijit, for website benchmarking, we use Siege. After a siege run, you’ll have data such as how many requests completed, the distribution of HTTP status codes, and average number of transactions per second. Be statistically smart about this - run several tests and average them together to get a baseline reading. Try to max out your environment; knowing when your overhead runs out in your current architecture is powerful knowledge. And, if you can max out your environment during the test, you can be sure the limits you’re hitting aren’t limits of the load testers, but the thing being tested. Try to hit your site in a browser during the load test so you can see what the user experience will be like in a high load situation. Anticipate problems before they occur.

Once you have these baseline numbers, you can test optimizations by running the same sequence of load tests and comparing the results. You can see if the various status codes coming back from the test changed. For example, if you deploy an optimization and start getting a lot of HTTP 500 instead of HTTP 200, you broke something.

Getting a good benchmarking environment setup is the essential first step to trying to optimize your site. In the next article, we’ll talk about some basic things to look at when approaching optimization, including whether you need to do it at all, and some specific examples of problems and solutions we’ve encountered at Lijit.

Modern browsers have added the ability to keep a list of frequently used search engines available for quick access at the top of your browser. If you’re using Firefox or Internet Explorer 7 you can add any Lijit search engine to that list. Simply go to a blog that has a Lijit Search Wijit on it, that user’s profile page or their search result page. You’ll notice an extra item in the list of search engines allowing you to add it to your list. Now anytime you need to do a search on that person, you can easily select their name and do that search. Below are screenshots for both IE7 and Firefox:

Internet Explorer 7:

Firefox:

Last week we released a new set of enhancements to Lijit, primarily focused on the signup process.

But there’s one feature that I’m particularly excited about but that isn’t immediately obvious: automatic blogroll crawling. What’s that you ask?

If you have a blog, it probably has a section where the you list other noteworthy blogs. These are blogs you read and basically find attention-worthy. Lijit can now automatically find this part of your blog and add all of those blogs to your Lijit search. And the Lijit server will check on your blog every day or so to see if you’ve made any changes–no need to configure things here every time you make a change.

Blogrolls form a huge implicit trust network, and no other service has really exploited them until now. Adding them to your network is just the first step, I’m also working on some blog authority algorithms that rely heavily on blogrolls. I’ll post some preliminary results soon.

So get those blogs rolling!